California has asked experts to examine whether an emergency shutoff for powerful AI models would work. New York City councillors have proposed independent checks and a human override for systems offered or used in their city. I can see why.
An agent with access to the internet, company records and the ability to act can make a mistake at rather more than typing speed. The Californian order seeks advice on possible changes to the law; the New York measures are proposals, not protections already in force. [1] [2]
There is something wonderfully old-fashioned about the response. Let’s assemble a watchdog or a team of experts, send them into the works, measure what they can and order the manufacturer to rectify what they cannot. I have spent enough of my working life around projects to feel the appeal. It also sounds faintly 1984 when the thing to be inspected is a gismo that might know rather a lot about us. But if we want powerful systems in ordinary life, somebody independent needs to be able to look beneath the bonnet.
The trouble is that this bonnet does not lift onto a machine its makers fully understand. Dario Amodei, Anthropic's chief executive, has said plainly that we do not understand the detailed workings of our own AI creations. Anthropic's researchers can trace some of Claude's internal processes, but say their methods uncover only a fraction of the work, even on very simple prompts. An inspection might establish how a system behaved in a test. It cannot certify every action it might take in every situation it has yet to meet. [3] [4]
That is why I would put the law first. The watchdog’s job is to enforce the boundary set by law. I do not want an unelected body deciding where that boundary ought to be.
What is the law for?
I come to this as a Radical Liberal. For me, the law exists above all to protect the freedom of the individual. Companies have legal interests too, though I will not pretend a company's inconvenience is the same as a person's liberty.
I would prohibit mass searches of our ordinary movements or behaviour to find people an agent decides are suspicious. Public CCTV feeds should not become an invitation to investigate everybody who walks past them; it’s not a box of Quality Street. For a particular serious investigation, an application must be made to a judge for limited access, with an independent advocate to argue for the person who cannot yet know about it. I want the rights of the individual protected, including the person who is the subject of a search. There must be limits on what the state can do, however clever the software and however respectable the official who asks it.
The law should also protect people from private misuse. If an agent breaks into a system, extracts information it was not authorised to obtain or makes a consequential change without permission, there must be a clear route to investigation, redress and, where appropriate, prosecution. The mere fact that its task was vaguely well-meant cannot be a defence. Nor can a government simply call its own intrusion lawful and expect us to stop asking whether it should possess the power at all.
An inspectorate with a bite
Let’s create an inspectorate with the people, skills and statutory powers to test AI systems, examine how AI agents connect to other systems, compel incident records, and investigate what happens when one crosses a boundary. The inspector should also be able to require changes and seek a rapid halt to a dangerous deployment. Its reports should be made public as far as security and personal privacy permit.
Britain already has an AI Security Institute doing technical research and evaluating risks. That is useful expertise to build on; it is not, by itself, the independent enforcement authority I am proposing. [5]
The inspectorate must not be judge and jury. It could pass evidence of an alleged crime to the appropriate investigators and prosecutors. In England and Wales, that may include the Crown Prosecution Service. [6] Courts decide breaches, remedies and guilt. Parliament receives the inspectorate's findings about patterns of harm and decides whether the law needs changing. It does not sit in judgement on an individual case. No minister gets to instruct an inspector to avert their eyes because a favoured department or contractor is involved.
Inspection cannot be a certificate of indemnity. If a system was approved on Monday and on Tuesday its agent acts outside its permitted scope, we investigate the act and who initiated it. I would make a developer whose safeguards are not up to the job face a truly substantial fine, scaled to its size and the seriousness of the breach, along with duties to repair the damage and compensate those harmed. A symbolic penalty would simply become a line in the project budget.
Responsibility does not stop at the developer. The organisation that gave an agent excessive access must answer for that decision, and a person who deliberately instructs it to do something unlawful must answer for theirs. Several parties may have contributed to the same harm. None should be able to point at the machine and walk away. Britain already uses fines linked to worldwide turnover for serious data protection violations; the liability rule I am proposing for AI would need its own legislation. [7]
Three laws are not enough
I have had a fondness for Asimov and his three laws of robotics since my teens. It’s a nice idea: put a few clear rules inside the robot and let it work out the rest. But a sentence or two in an AI's instructions are not a safeguard. AI can misunderstand the task, be led astray by the materials it reads or even find a route its designer did not expect. Britain's National Cyber Security Centre warns against relying on prompts alone and recommends technical limits on access, monitoring and guaranteed human approval for consequential actions. [8]
So give the agent a defined job and only the credentials, sources and tools required to complete it. Make some actions technically impossible for it to perform on its own. If it cannot find an answer in recognised sources, it should ask rather than wander into a private database. If it wants to send information, alter a record, spend money, or reach beyond its approved search, it should stop and ask a named human to decide. An approval must be recorded and attributable. A human click, however, cannot authorise what the law has prohibited. And a publicly accessible record does not grant permission to assemble every stranger's life into a dossier.
Someone will, of course, build their own Jarvis in their bedroom and forget to invite the inspector round for tea. We cannot inspect every private program before it exists. We can, however, set enforceable limits on what its maker or user may do with it. I would want the law to give courts a way to stop an unlawful system and deal with the people responsible after due process.
Who guards the law?
Here’s the part I find hardest. The law can protect us; a government with a large parliamentary majority can also change it. A mass search that was unlawful yesterday might be made lawful by a vote tomorrow. Britain's courts can challenge unlawful executive action and can identify incompatibility with human rights, but they cannot simply erase an Act of Parliament because a majority has passed a bad one. [9] [10]
Churchill's familiar remark about democracy being the worst system apart from the alternatives was part of a larger point: ministers should be guided and controlled by public opinion, serving the people rather than mastering them. He even introduced the famous line with 'it has been said'. I think that qualification is worth keeping. A democratic majority is a mandate to govern, not a moral blank cheque to observe every citizen. [11]
No magic provision can be made that a future Parliament could not repeal. We can, however, refuse to let intrusive powers arrive quietly through a back door. Any proposal to expand those powers should face public scrutiny, independent examination of its effect on liberty, proper parliamentary debate and a route for those affected to challenge it. The inspectorate must be free to report when the state itself crosses the line. The government should answer to the courts too; it must not bring in “Big Brother” behind closed doors.
I want AI to help scientists find the next useful clue in solving one more of the wonders of the universe, to help workers spend less time on drudgery and to solve problems that really matter. Those are good reasons to make room for it. They are also reasons to build a framework that people can trust. The law should say which powers are off limits; the watchdog should find out when the rules are broken; the courts should hold the responsible people to account. Above all, we must keep the freedom to question the law when a government wants to move the boundary.
If an AI has been poking about in my life, I want to know who sent it. And if I’m told Parliament allowed it, I want to know when my privacy became theirs to give away. Wouldn’t you?