The row between Anthropic and the Pentagon began with a question about what Claude may be asked to do. It has left me thinking about what any government should be allowed to ask an AI to do to us.

Anthropic supplies Claude, an AI system the American military has used. It wanted to keep two limits on that use: no mass domestic surveillance and no fully autonomous weapons. The Pentagon wanted permission for “all lawful uses”. When neither side would give way, the Pentagon designated Anthropic a supply chain risk and began removing Claude from its systems. A US appeals court has now upheld that designation by two votes to one. One judge dissented, arguing that the law on supply chain risks had been stretched too far. [1][2]

I can see the military’s point about choosing a tool it can rely upon. Nobody wants an AI to refuse a legitimate task in the middle of an operation. I can also see why Anthropic would refuse to supply one for uses it considers dangerous. The company has not been ordered to remove its limits; the government does not have to keep buying Claude. The supply chain risk label has its own legal argument, but the contract is not the question that keeps me awake.

The court decided whether the Pentagon could reject Anthropic’s conditions. It did not decide whether any government ought to possess the powers those conditions were meant to restrain.

“All lawful uses” is meant to reassure us. It does not reassure me. The law can protect us; it can also become our persecutor. Change a rule, join up a few databases and flick a switch, and a government could examine lives on a scale that used to be impossible. Long before anyone stands in court, there may be no presumption of innocence in the way the system treats a person, no solicitor to challenge what it thinks it found: just bits and bytes flowing around a network until a suspicion becomes an official fact. Rights written on paper are precious, but they do little for someone who cannot even see the judgement being made about them. Anthropic warns that AI can assemble scattered information into a detailed picture of a person’s private life. [3]

Imagine an official telling an agent: “Go through every CCTV feed you can reach. Find anybody behaving suspiciously, identify them and add the suspected offence to their record.” I have no evidence that the Pentagon has given such an instruction. I am asking why we would ever build a system with the access and authority to carry it out.

What counts as suspicious? A person who changes direction on seeing a police officer? Someone who visits the same building twice in a week? A face matched to the wrong name? Researchers at the US National Institute of Standards and Technology have documented false matches in facial recognition, with error rates that vary between groups. Even a correct identification tells us nothing certain about guilt. [4]

A conviction still requires a legal process. The harm can begin well before that. A suspicion might enter an intelligence file, trigger an investigation or put someone under repeated scrutiny. An error can acquire a life of its own as other systems repeat it. An innocent person may never know why doors keep closing. A guilty person remains entitled to evidence, a fair hearing and a punishment set by law. We do not give those principles away because a computer can get through the footage before breakfast.

This is where I come at it as a radical liberal. I value the freedom of the individual more highly than the convenience of the state. I do not want us to monitor everybody because it is easier than looking for the few people we have a reason to investigate. Nor do I want a machine to identify someone, decide they are suspicious and set official action in motion. Once that machinery exists, we must trust every future minister, commander, police chief and contractor to use it wisely. That is a great deal of trust to place in a cupboard marked For Responsible People Only.

I worry about what less accountable governments might already be building. I do not know the extent of AI surveillance or autonomous military systems in China or Russia, so I will not pretend to. Nor will I give Britain or America a free pass simply because we call ourselves democracies. A government becomes a bad actor towards its own citizens when it treats their ordinary lives as material for a permanent investigation.

I think mass searches for law and order or behavioural monitoring should be banned outright. A minister’s signature or an oversight committee cannot make searching an entire population for vaguely suspicious behaviour acceptable. Some doors should simply remain closed to the state.

We already understand the principle behind a search warrant: an investigator asks a court to authorise a particular intrusion and must justify it. In England and Wales, search warrants are usually issued by a court; applications are commonly made without the person being searched in the room. [5][6] For AI access I would start there, and go further. Show a judge whom or what you are investigating, for which serious offence, which records you want and for how long. Appoint an independent advocate, a publicly funded defender for the person who cannot yet know about the application, to argue the other side. That last safeguard is my proposal, not a description of today’s ordinary warrant hearing.

The warrant should limit the search to the approved purpose. Unrelated information should be discarded; access and results should be logged and independently checked. Once secrecy is no longer necessary, the person affected should be able to discover what happened and challenge an error. The agent should have no power to alter a criminal record, impose a penalty or quietly expand its own search. Those limits belong in law and in the system’s actual permissions. Our freedom should not depend on whether the next supplier happens to share Anthropic’s conscience.

There is another danger: an agent can cause harm while pursuing an ordinary task because we specified it badly or gave it too much access. We need proper engineering boundaries for that too. But it is the second concern in this argument. A perfectly obedient machine can do terrible things when a person with authority tells it to.

AI can be an extraordinary helper. It can sift evidence, find patterns, save time and help scientists ask better questions. I want us to enjoy those benefits. I also want us to be able to say, firmly, that there are powers no government should acquire merely because a new tool has made them possible.

The court has answered the question it was asked. We still have to answer ours: which powers should no government possess, however lawful the request or respectable the person who makes it?