There is a comfortable assumption running through much of the public conversation about artificial intelligence in government: that if a system is tested rigorously enough before it goes live, the risk has been managed. Put it through its paces, check it for bias, confirm it does what the vendor promised, and the job is done. A pointed opinion essay published this week in The Mandarin argues that this thinking is dangerously incomplete, and that the harder question is not whether an AI system works on day one but who retains control of it on day one thousand.
The framing is deliberately provocative. Testing, the argument goes, tells you how a model behaves in a controlled setting against a known set of cases. It says very little about what happens when that same model is embedded deep inside the machinery that delivers essential services, when its outputs quietly shape decisions about payments, eligibility, enforcement or safety, and when the people relying on it have long stopped questioning it. The essay’s central image, that someone has to hold the keys, is really a demand for independent oversight, for genuine off-switches, and for the maintenance of alternatives so that government is never wholly dependent on a single system or a single supplier.
Why control outlasts testing
The distinction matters because AI systems are not static. They are updated, retrained and repointed at new tasks, often by vendors operating under commercial terms that public servants did not write and cannot easily see inside. A tool approved for one narrow purpose can drift into another through nothing more sinister than a software update and a bit of scope creep. When that happens, a one-off certification completed months earlier offers little protection. What offers protection is the ability to interrogate the system continuously, to understand its reasoning well enough to challenge it, and to turn it off without the whole service collapsing.
This is where the question of dependency becomes acute. If an agency hollows out the human expertise that used to perform a function, and then hands that function to an automated system it does not fully control, it has not simply adopted a new tool. It has surrendered a capability. Rebuilding that capability in a hurry, during an outage or a scandal or a contract dispute, is far harder than it sounds. The essay’s warning about keeping alternatives alive is really a warning about institutional memory and the quiet erosion of the state’s ability to act on its own.
Two ways of seeing it
Supporters of a stronger oversight regime see this as basic prudence. Public trust in government is hard won and easily lost, and nothing corrodes it faster than the sense that decisions affecting people’s lives are being made by a black box no one can explain or appeal. Australia already has a cautionary tale seared into the public memory in the form of the Robodebt scheme, where an automated approach to welfare debt recovery caused enormous harm and was later found to be unlawful. The lesson many drew from the Royal Commission was not that automation is inherently bad, but that automation without accountability, without genuine human control and without the willingness to stop when something is clearly wrong, is a recipe for disaster.
The counter-view is not that oversight is unnecessary, but that it can be overdone. Agencies under pressure to cut costs and clear backlogs see AI as one of the few levers that can genuinely lift productivity, and they worry that layering on independent regulators, mandatory kill-switches and parallel manual systems will make adoption so slow and expensive that the benefits never arrive. Industry voices tend to argue that heavy-handed controls risk entrenching the status quo, pushing talented vendors away, and leaving Australia dependent on overseas systems built to someone else’s rules. On this reading, the answer is better-designed oversight that is proportionate to risk, not a reflexive brake on everything.
Both positions contain truth, and the sensible path probably runs between them. Not every chatbot that helps a citizen find a form needs an independent watchdog and a failover plan. But a system that influences whether someone receives a payment, keeps a licence or faces an investigation is a different animal, and the stakes justify a different standard.
The Australian stakes
For Australia, this debate is arriving at exactly the moment government is moving from cautious experimentation to real deployment. Reporting over the past year has traced generative AI tools reaching multiple Commonwealth sites, trials of AI assistants in local councils, and a steady drumbeat of agencies weighing where automation can safely take load off stretched workforces. The appetite is real, and so is the anxiety among public servants about what it means for their roles and their accountability.
The sovereignty question sharpens all of this. Much of the most capable AI on offer is built and hosted by a handful of very large overseas companies, which is part of why the definition of sovereign AI has become such a live policy argument in Canberra. If essential services come to rely on models that an Australian agency cannot inspect, cannot modify and cannot guarantee access to, then holding the keys becomes more than a metaphor. It becomes a matter of whether the state can keep operating on its own terms when a commercial relationship sours or a geopolitical wind shifts.
There is also a governance gap to reckon with. Australia does not yet have a single, binding regime for high-risk AI in the public sector, and the oversight that exists is spread across privacy law, administrative law, procurement rules and a patchwork of voluntary frameworks. The argument for someone to hold the keys is, in practice, an argument for making that oversight coherent, independent and backed by real power rather than good intentions.
What’s next
The near-term test will be whether the government’s work on mandatory guardrails for high-risk AI translates into enforceable requirements for public sector use, including clear lines of accountability and the ability to switch systems off. Expect continued pressure from integrity bodies, unions representing public servants, and civil society groups who remember Robodebt and want structural guarantees rather than assurances. Agencies, for their part, will keep pressing the productivity case, and the real negotiation will be over which uses are deemed high-risk enough to warrant the full apparatus of independent control.
The essay’s contribution is to shift the frame. The question is no longer only can we trust this system, but can we take it back when we need to. For a country that has already lived through the consequences of getting that wrong, it is a question worth holding onto.
Sources: The Mandarin
