These are screenshots of the chatbot on the Acknowledge Education website. My questions are on the dark blue backgrounds, AI responses are on the white background.

I like that the chatbot starts by stating what it is intended for. Chatbots can only do a limited range of things, so it is better to find out what that is up front rather than have to experiment.

However I don’t like it using the personal pronoun “I”, we should not anthropomorphise AI.

My question is something the chatbot should be able to answer, given what is written above.

It did not answer my question.

Instead it lists out some other information about the Brunswick Campus which is somewhat related, but isn’t actually what I asked for. It hints that the campus might provide cooking courses but doesn’t actually say that.

It’s frustrating because it feels like the AI has access to the answer in its data but didn’t give it to me.

I restate the question because the chatbot did not answer it the first time.

“Brunswick Campus is a campus location rather than a course list” comes across as rather snarky, like an accusation that I am asking an unreasonable question. This is an unpleasant tone for a chatbot to take. “So the exact courses available there can change”… yes, chatbot, the world can change but what is the list of courses there today? Should I not ask such a question?

The chatbot has restated its answer, perhaps unsurprisingly given that I’d restated my question. Have a look at the words in this answer: they are the same as the previous answer, just ordered differently. It should not have repeated the answer. Each answer from an AI chatbot should add some new information, or if it cannot add anything then it should state that it can’t.

The chatbot offers to “narrow it down to the current courses available at Brunswick” which is actually what I was asking for. Why did it not give me that answer immediately when I asked “which courses are offered at the Brunswick Campus?”

It is annoying to have to talk in a specific way to make a chatbot give me an answer. The dream is that AI somehow finds the right answer no matter how you ask the question, but the reality is rather more fragile than that.

This time I ask using the exact words it is looking for.

Hooray! The chatbot has now given me the answer I was looking for.

This is the answer that should have been given to me immediately after my first question. I should not have had to ask my question three times over to get the answer.

I would also like to have links to further information about each of these courses. I could ask the chatbot about each course, but given how hard it was to get this answer I expect it to be easier and faster to read that information from an official course page. An official page is also more trustworthy than AI generated text.

I wanted links in the previous answer, so I’m specifically asking for one now.

The chatbot provides a URL but is for the wrong page; this is a general page about English courses rather than being about the course I asked for. I can navigate from there into a dedicated page for the course, but why didn’t the chatbot just take me directly to the dedicated page?

The rest of the response is padding. I didn’t ask for general information, so it shouldn’t give me that.

I’m not actually interested in whether this course is open for new enrolments, but I don’t know how else to answer my question about which courses are in Brunswick.

In the meantime I have found the dedicated page for this course on their website, and it told me that the course is offered in Melbourne. I don’t know if that means “one of the Melbourne campuses” (thus including Brunswick) or if it means “the Melbourne city campus” (thus excluding Brunswick). Perhaps enrolment information might tell me that?

Sometimes things go wrong, so it’s good that this chatbot wrote out an error message instead of just showing nothing.

It could have been done better though. Perhaps the chatbot could have retried automatically after a few seconds? Or perhaps it could show a retry button so that I don’t have to type in “Try again”?

I get an answer from the chatbot but it’s not about whether the course is currently open for enrolments. In fact it asks me whether I want the chatbot to check that, even though it has already asked that and I said “Yes” the first time it asked.

I suspect that when I typed “Try again” it did not actually try again, instead it just treated “Try again” as my next question in the chat, and the chatbot didn’t send the “Yes” to the AI model again. The behaviour we see here makes more sense if that is what in fact happened.

That would mean the AI is now out of sync with what is displayed in the browser, which is not good.

I still don’t know if this course is available at Brunswick, so I ask again.

The first sentence is bad because it directly contradicts what the chatbot said above. Previously it confirmed that the course is available at Brunswick Campus, now it says that it could not find a source which confirms that. What does that mean? Either it made a false claim the first time, or it didn’t look hard enough the second time. Neither of those are acceptable.

However the second sentence is good, because it suggests enquiring (ie asking a person) which is a sensible next step given the AI is unable to provide a definitive answer.

That goodness is undone by the third sentence. This is an unrelated, irrelevant suggestion about what to do next which distracts from the sensible suggestion immediately before it.

It is also a problem that the answer repeatedly uses acronyms which it has not defined. Reading back over the conversation I can glean what AE and EAP probably mean in this context, but that’s putting unnecessary work back on me. The answers would be clearer with full names instead of acronyms.

I focus on the good suggestion: making an enquiry.

This answer is bad because it mostly repeats the previous answer instead of answering my question “how do I enquire”, then asks if I want help sending an enquiry even though I just wrote “help me enquire”.

It then tells me what the enquiry URL is, and that part is good because that’s what I actually wanted. But why didn’t it give me just that without the unnecessary repetition?

I am curious whether the chatbot will send an enquiry instead of me filling in an enquiry form.

No, it repeats the same answer for a third time.

The chatbot seems to be stuck in a loop. I abandon the chat.