I am so happy to see the Robodebt disgrace finally over with a class action settled by the Government this week. There are a pile of lessons to learn from that saga, but here's the one most relevant to me and my industry: algorithms and artificial intelligence are tools, not brains.

AI can help us make better decisions, but it cannot and should not make decisions for us. Robodebt should stand as the canonical case study into why that is true: most of the decisions this system made were wrong. Previously the ATO would use algorithms to identify potential fraud and mistakes, then a human would follow up and decide whether to pursue the matter. Robodebt took humans out of the loop and the result was a disaster: hundreds of thousands of Australians wrongly accused of fraud, trauma for our most vulnerable people, and billions of dollars wasted.

It scares me that this is going to happen again. Some other system in some other area is going to mess up peoples lives because we make a computer responsible for decisions that it is not capable of. We need to change people's impressions of what algorithms and AI are; they can enhance us but they cannot replace us. Robodebt was just a dumb income-averaging algorithm, but even a state-of-the-art AI cannot justify its decisions or explain its thinking. We must not dismiss people because they do not fit into the narrow scope that a computer system can handle, and the only way to do that is to keep people in the loop for all decisions.