Every teacher who has marked an essay padded with invented references knows the problem by now. A student asks a chatbot for help, the chatbot answers in fluent, confident prose, and somewhere in that polished paragraph sits a fact that is simply not true. The machine has hallucinated, and unless the reader already knows better, the error sails straight through.
That everyday classroom scenario is the starting point for a pointed essay published by the Australian Association for Research in Education, whose Know stuff piece makes a deceptively simple argument. The best safeguard against AI hallucinations, it contends, is not a smarter filter or a stricter ban. It is a student who knows enough to notice when the answer is wrong.
Why chatbots make things up
The word hallucination has become shorthand for a well-documented behaviour of large language models. These systems do not retrieve facts from a database. They predict the next most plausible word based on patterns in their training data, which means they are built to sound right rather than to be right. When a model does not know something, it does not fall silent. It generates a confident guess, complete with the cadence and vocabulary of expertise.
For adults with a solid grounding in a subject, the tell is often obvious. A lawyer spots the fabricated case citation, a historian catches the wrong date, a scientist notices the impossible figure. The trouble in a school setting is that students are, by definition, still building that grounding. They are the readers least equipped to catch the errors, and they are being handed the tools most likely to produce them.
The AARE essay pushes back on a comforting assumption that has taken hold in parts of the education debate, namely that critical thinking and digital literacy alone will save us. You cannot think critically about a claim in a vacuum. To question whether a statement is plausible, you need a store of knowledge to weigh it against. In other words, the ability to detect a hallucination is downstream of actually knowing things.
Knowledge as a shield, not a relic
This is where the argument gets its bite. For years, a strand of educational thinking has treated the memorising of facts as old-fashioned, something to be outsourced now that any answer is a search away. Why clutter young minds with dates and formulas when the information is always in your pocket? The rise of generative AI, the essay suggests, turns that logic on its head. The more powerful and persuasive the machine, the more a person needs their own knowledge to check it.
The position sits neatly alongside the knowledge-rich curriculum movement that has gained ground in Australian schooling, and which underpins parts of the revised national curriculum. Advocates of that approach argue that broad background knowledge is what allows a reader to comprehend complex material, make inferences and, crucially, spot when something does not add up. Seen through that lens, an AI-saturated world does not make knowledge less valuable. It makes it the difference between using a tool well and being fooled by it.
There is a counter-view worth airing. Plenty of technologists and some educators argue that the tools themselves are improving fast, and that features such as retrieval-augmented generation, source citation and verification prompts are steadily reducing the hallucination problem. On that reading, teaching students to interrogate outputs is a transitional skill, useful now but less critical as the models get more reliable. The risk in that optimism is timing. The tools are in classrooms today, hallucinating today, while the fixes remain partial and inconsistent across the products students actually use.
A more moderate camp lands between the two. It accepts that AI is not going away and should not be banned outright, but insists that access without foundational knowledge is a recipe for confident ignorance. The goal, in that framing, is not to keep students away from chatbots but to make sure they arrive at them already knowing enough to argue back.
What it means for Australian classrooms
The debate is not academic for Australian schools, which are moving quickly on generative AI. Education ministers have signed off on a national framework for generative AI in schools, and systems from the public sector to the independents are piloting approved tools, building policies and training teachers. The federal government has also been rolling out broader AI guidance touching jobs, small business and education, and universities have spent two years rewriting assessment rules to cope with students who can produce a plausible essay in seconds.
Yet much of that activity has focused on the mechanics of access and integrity. Which tools are allowed, how to detect cheating, where the privacy lines sit. The Know stuff argument shifts the conversation upstream, towards curriculum design and what students carry in their own heads. If it is right, then the most important AI policy a school can have may not be a usage rule at all. It may be a commitment to teaching content deeply enough that students can tell when the machine is bluffing.
That framing also speaks to equity, an issue that runs through Australian education policy. Students who arrive at school with rich background knowledge, often from more advantaged homes, will be better placed to catch AI errors and use the tools to extend their learning. Students with thinner foundations risk absorbing hallucinations as fact, widening a gap that already worries educators. On this view, a knowledge-rich curriculum is not just a hedge against bad AI. It is a fairness measure.
What comes next
The practical questions are only starting to be worked through. How do you teach students to verify AI output without teaching them to distrust every source? Where does knowledge-building sit against the pressure to cover AI skills themselves, from prompting to ethical use? And how do teachers, many of whom are still finding their own footing with these tools, model good practice in real time?
None of this is settled, and reasonable people in Australian education disagree about the balance. What the AARE essay does usefully is reframe the panic. The headline anxiety about AI in schools has centred on cheating and on jobs. The quieter, arguably more important question is epistemic: in a world where a machine will always give you an answer, how do we make sure young people know enough to judge it? The answer on offer is unfashionably old. Make them know stuff.
Sources: Australian Association for Research in Education (AARE).
