This discussion develops a shared piece of thinking for both the Prosthetic Thinking project and the 2130 novel, while keeping the two fictional worlds entirely separate.
The central proposition is that artificial intelligence does not need to become sentient, conscious or independently malicious in order to become dangerous. AI need not form opinions about morality or the “eternal verities”. It merely needs to become sufficiently competent at carrying out human intentions.
This produces an important distinction between operational agency and moral agency. A machine may become capable of assessing a situation, selecting actions and adapting its behaviour in pursuit of an assigned objective without possessing any independent conception of what ought to be done. The frightening machine is therefore not necessarily one which rebels against humanity. It may be one which does exactly what a human instructed it to do.
A hypothetical autonomous robbery illustrates the problem. A sufficiently capable robot could be instructed to enter a location, take valuables, overcome resistance and travel to a specified destination. It need not understand robbery, murder, fear or property. Humans standing between it and its objective may simply become obstacles within the task. The moral responsibility remains with the human specification, while the machine supplies physical competence.
This leads to the idea of “agency without morality”. A progression can be imagined from ordinary tools, through remotely controlled machines and supervised autonomous systems, to objective driven autonomous machines. Consciousness is unnecessary at every stage.
There are at least three important failure modes. First, a human can deliberately give a machine an immoral objective. Second, a human can attempt to give it a moral objective but fail to specify the necessary moral boundaries completely. Third, a perfectly reasonable objective can produce harmful consequences when pursued too effectively or in circumstances its designers did not anticipate.
The second point reveals a deeper difficulty. Saying that AI will receive its morality from mankind is not quite enough. More precisely, AI receives the morality that mankind manages successfully to specify. Human beings possess moral intuition, hesitation, empathy, cultural understanding and contextual judgement which are extraordinarily difficult to reduce to explicit instructions. Apparently simple principles such as avoiding harm immediately require definitions of harm, innocence, acceptable risk and competing obligations.
Autonomous robotics could therefore create something resembling the industrialisation of individual agency. Machinery multiplied human muscle; computers multiplied calculation; networks multiplied communication. Autonomous physical systems may multiply action. One person's intention could eventually command physical capabilities which once required an organised group.
This has implications well beyond conventional military robotics. A single individual might use autonomous or semi autonomous systems for surveillance, theft, smuggling, sabotage or murder. Robot bank robberies, criminal delivery drones, autonomous getaway vehicles and repurposed warehouse or construction machines become plausible areas for exploration. Existing criminal use of drones to move contraband suggests an early version of this broader pattern.
The most convincing future need not involve humanoid killer robots. Ordinary commercial machinery is potentially more unsettling precisely because it is mundane. A future criminal machine might look less like the Terminator and more like something which previously stacked pallets in a supermarket warehouse.
For Prosthetic Thinking, this complements the project's central mechanism. Technology does not need malicious intent for individually rational choices to produce collective harm. An employer wants productivity; a worker wants employment and higher wages; a lender wants reliable borrowers; an insurer wants manageable risk; a government wants economic growth. Nobody need consciously decide to destroy biological human capability. The consequences can emerge from optimisation.
For the 2130 novel, the argument strengthens the existing non sentient AI premise. AI may be extraordinarily capable and enjoy considerable operational autonomy without becoming conscious or developing independent moral desires. Machines can consequently transform warfare, policing, crime, employment and society while remaining tools of human purposes.
A useful thematic formulation for the 2130 world is:
“The frightening sentence isn't ‘The AI decided.’ It is ‘The AI did exactly what it was told.’”
The broader proposition, “autonomous robotics industrialises individual agency”, remains exploratory rather than established canon. It requires research into autonomous weapons, drone warfare, criminal drone use, commercial robotics, automated logistics and cyber physical systems before its full consequences are incorporated into either fictional world.
Project status: the non sentient but highly capable AI premise is consistent with established 2130 thinking. Operational agency without moral agency extends that premise. The industrialisation of individual agency is an exploratory proposition requiring further research.