I use Xero to manage the accounting for my businesses, and Xero has a great feature called bank feeds which automatically imports transactions from your bank accounts for reconciliation with your accounting entries. I have bank feeds set up for my main business bank accounts and they’re working great, so when I recently set up a Vanguard investment account I wanted to get its bank feed connected to Xero as well. I wasn’t sure if that is possible, so I queried JAX (which is Xero’s AI chatbot).
JAX failed, but I think it is the kind of question AI should be able to answer so I have written it up below.
This writeup is part of my series on failed conversations with AI chatbots. The writeup is long and full of boring details because it is not intended to be a succinct and enjoyable read; instead I want it to be a straightforward and perhaps even dry incident report. You may read more about why I am doing these writeups in the blog post that I wrote to announce this series.
These are screenshots of the Xero JAX AI chatbot in the Xero web app.
My questions are on the blue backgrounds, AI responses are on the white background.
JAX starts by giving a step-by-step guide, which is a pretty reasonable way to answer a question like this so I don’t fault it for giving this answer.
However it is not useful to me because I had already tried these steps before turning to JAX. That is a common problem I have with AI chatbots: they inevitably start by telling me what I already know and have already tried. I can’t fault it… yet it feels patronising.
JAX says “If Vanguard appears in the list”.
Why doesn’t JAX know if Vanguard will or will not appear in the list? It should know that because whether Xero supports Vanguard bank feeds is factual information which is not dependant on my situation.
That JAX doesn’t know may explain why the rest of this chat is a failure, because everything in the chat relates to how Vanguard is handled by Xero.
Incorrect; these are not helpful, they are confusing.
These pages are for five unrelated banks and none of them mention Vanguard at all. This whole section should have been dropped because my question was about Vanguard, not these banks.
If JAX can provide guidance for my specific situation why did it not give that guidance to me already? Why would I ever want generic guidance when specific guidance is available?
I am irritated by this statement.
I followed the instructions above and Vanguard did appear in the list, but I couldn’t use it because it is only listed under the “United States of America” heading. That situation is not covered by the instructions above so I ask a followup question.
Jax says “If your Australian Vanguard Cash Account is operated through ANZ and is eligible for ANZ direct feeds” which is exactly my question: is it eligible? I would expect an AI chatbot to be able to answer that, but JAX is turning the question on me.
The good part of this answer is that it list some accounts which are eligible and some accounts which are not eligible, but my account is in neither of the lists so the answer does not help me.
The first link is relevant because it is about ANZ, though unfortunately I found it to be unhelpful for my situation. It is appropriate to include it here because it does relate to the bank I asked about.
However the rest of the links relate to banks that I have not mentioned, and so shouldn’t have been shown here.
The previous answer suggests that access to ANZ online banking required to activate an ANZ bank feed, so now my question is whether that is available for Vanguard accounts.
This does not answer the question. JAX has pulled together a variety of facts about ANZ and Vanguard which are irrelevant to my question. The similarity to my topic makes this particularly confusing: at a surface level it seems relevant, so I find myself reading over it several times to make sure I’m not missing something important.
This information should not have been shown to me at all. If JAX is unable to answer this question it should write that it cannot answer the question.
The list of resources at the end of each answer continues to be unhelpful.
Aside from a practical ineffectiveness of these links, the repeated nature of them makes JAX seem rather simplistic.
I ask about the Vanguard BSB 013936 because I figure the BSB might be a keyword that can uncover any Xero support documents that specifically relate to Vanguard bank accounts.
OK, we get a clear answer now!
No, Xero does not support bank feeds from Vanguard.
But why didn’t JAX tell me this up front? When I asked my first question “How do I connect up automatic bank feeds for my Vanguard Cash Account” JAX should have immediately told me bank feeds are not supported for Vanguard Cash Accounts.
However I am suspicious about the phrasing “it is explicitly listed as unsupported”. What lists it as unsupported? Did that come from an official Xero document or from some unverified source it found on the internet?
Perhaps it has come from this second link “BSBs not supported – Class Support”?
That page lists BSB 013936 as unsupported, but it is not a Xero website and does not mention Xero so I am unsure if this is a trustworthy source.
I ask for more information because I do not trust the previous answer.
This answer confirms that the Class Support article mentioned in the previous answer is the source for the statement that Xero does not support bank feeds from Vanguard.
JAX claims that Class “is referenced by many accounting an financial software providers, including Xero, to determine feed eligibility”, so it seems reasonable that Class can provide authoritative answers for Xero.
However I suspect that the statement may actually be false because the Class page does not mention Xero, and I can’t find a connection between Xero and Class on their website.
The “Bank feeds available in Xero – Xero Central” link gives me a list of bank feeds available in the UK, which it should not have done.
JAX should know from the context (I am logged in to Xero Australia) and from the chat (I wrote that I have an Australian account) that UK bank feeds are not relevant here. I find it especially concerning that JAX would mix information about Australian banks (first line) and UK banks (second line) in the same response, they way they are presented suggests that they both relate to Australia.
I test my suspicion that JAX is giving me false information by asking for more detail.
JAX is unable to answer. If it is true that Xero uses Class to determine feed eligibility then JAX should be able to answer this question; that it refuses to answer makes me suspect the claim is false.
However I am not sure because I have no proof that Class either is or is not used by Xero.
I try to fulfill the “can only answer queries related to Xero’s accounting products” requirement by only mentioning Xero.
JAX is still unable to answer.
JAX has repeated the same non-answer three times in a row, and so it seems JAX will be unable to help me. I abandon the chat.
JAX failed at this task. It was unable to give me instructions on how to set up my Vanguard account, but it strung me along for several minutes with plausible-sounding steps that didn’t work for my situation.
Then on the fourth question it switched to a definitive statement that what I want to do is not possible. If that was true then JAX should have told me that immediately when I asked my first question. However I’m not confident that it is actually true because the sources for that statement look suspicious to me.
So the result is that I don’t know. I suspect that Xero does not support bank feeds from Vanguard, because if it did then I probably would have found that without help from JAX. I suppose that what I really wanted was for JAX to confirm my suspicion that Vanguard is not supported, but when JAX actually made that confirmation it seemed to be based on false information. So even when I got the answer I was looking for I was not confident or satisfied by the answer.
I think that the fundamental problem here is that we cannot distinguish what is not possible from what is possible but unknown to the AI chatbot. The information available to the AI is likely to be the same in both cases (ie it has no information on that topic either way) and so the answers are indistinguishable. In this situation an AI chatbot cannot give me any useful information, neither tell me how to do my task nor confirm that my task cannot be done.