The Pratidhwani

Artificial Intelligence: AI ‘kill chain’: What happens when humans trust bad data at the worst possible moment


AI 'kill chain': What happens when humans trust bad data at the worst possible moment
The modern kill chain is becoming a race to turn enormous volumes of battlefield data into decisions, often in minutes rather than hours.

Artificial intelligence is the biggest story of our time. It writes code, drafts legal notes, plans wedding menus and reads medical scans. Companies rebrand themselves around it, investors pour billions into it and governments race each other to build it first. Every boardroom, conference and family WhatsApp group seems to have a view on where it is heading.Even my friend Ayush has joined the chorus. This is a man who once needed three phone calls to split a restaurant bill on UPI. Today he lectures me about large language models, prompt engineering and why my job will disappear by 2030.It is easy to laugh at the hype. Most of it is seemingly harmless.But the same technology has moved somewhere far less funny. It now sits inside military command rooms, helping process the information used to decide what gets hit.Two incidents from 2026 show why that matters. In one, a US missile strike destroyed a girls’ school in the Iranian city of Minab, killing more than 150 people, including at least 120 children, according to UN findings and reporting on the US investigation. In another, a false intelligence report about a Chinese ship nearly triggered a US military operation after a chatbot misidentified the vessel’s cargo.Neither story is as simple as saying that “AI decided to kill”.The more unsettling story is quieter: bad information entered a system, moved quickly through it and acquired the authority of a much larger decision-making machine.That is the AI kill chain.

But first, what is an AI kill chain?

“Kill chain” is old military language. It describes the sequence of steps between finding a potential target and destroying it, taking in everything from surveillance and identification to targeting, authorisation, engagement and assessment.Think of it as a relay race in which the baton is information. Intelligence analysts, imagery specialists, targeteers, lawyers, commanders and operators each handle it at different points. If the baton is wrong at the first handover, everyone can still run perfectly. They just run in the wrong direction.AI changes the speed and scale of that race.Modern military systems can process satellite imagery, drone footage, radar, signals intelligence, open-source information and other data far faster than a team of people could manually. Systems such as Palantir’s Maven Smart System can fuse large numbers of data inputs and help analysts identify and assess potential targets.

The danger lies not in AI making every decision, but in bad information gaining momentum as it moves through an increasingly automated chain.

That does not mean a machine simply picks a person and fires a missile. Humans remain involved, and targeting continues to pass through intelligence, legal and command processes. Maven itself does not independently make lethal decisions.But that distinction can be misleading.The important question is not only who presses the button. It is what information reaches the person who presses it, how that information was assembled and how much time they have to challenge it.If the information on the screen is wrong, human involvement at the end of the chain does not magically make the decision sound.

Minab: When an old mistake moved at AI speed

On February 28, as the US and Israel began their war with Iran, two Tomahawk missiles struck the Shajarah Tayyebeh Elementary School in Minab.More than 150 people were killed, including at least 120 children, according to the UN and reporting on the US investigation. Bloomberg described it as the deadliest American military targeting error of the 21st century in terms of child casualties.The crucial fact is that the school was not newly identified as a target that morning.The site had once been part of an Islamic Revolutionary Guard Corps naval compound, and the US military’s intelligence database continued to catalogue it as a military facility even after the site had changed.The real world had moved on.The school had been separated from the neighbouring military compound by new walls and entrances. Satellite imagery showed painted walls, a football pitch and markings for school activities. The school had its own website and appeared on Google Maps. Bloomberg’s reconstruction found evidence of these changes going back years.Someone had even noticed.US intelligence analysts had detected changes at the site as early as 2019. But remarks about those changes were entered into a system that was not connected to the primary military intelligence database used for targeting. The information existed. It simply did not reach the people who needed it.That distinction matters because AI did not invent the original mistake.

The Minab school was separated from the military compound years before the strike, but its old classification survived in the targeting database.

The underlying intelligence was already stale.What AI changed was what happened next.The Minab site, still carrying its old classification, entered the Maven system alongside other potential targets. According to officials involved in the Pentagon’s investigation, Maven sat between the initial intelligence inputs and later review stages of the targeting process. Work that could previously take hours was compressed into minutes.Some officials also expected Maven to identify stale information or inconsistencies in the intelligence feeding the system. Investigators have said that expectation contributed to an over-reliance on the technology.The result was not a rogue algorithm. It was something more ordinary and potentially more dangerous: an old error travelling through a faster system.The UN’s Independent International Fact-Finding Mission on Iran later said there were reasonable grounds to believe the US was responsible for the Minab strike and concluded that the attack constituted a war crime. Washington has rejected the finding, while the Pentagon’s own investigation has focused on failures in the intelligence and targeting process.

Who failed: The machine or the people?

The answer is both less dramatic and more complicated than either.The Pentagon’s investigation has focused on outdated intelligence and failures in the targeting process, but Bloomberg’s reporting found that investigators also identified outdated imagery, an over-reliance on AI-assisted targeting and gaps in civilian-harm review.There was also a human-system problem.Civilian-harm mitigation teams across the Pentagon had been sharply reduced. Bloomberg reported that staffing across several such teams had fallen by roughly 90%, while the team at US Central Command had been cut from 10 people to one. No member of that team reviewed the Minab site before the strike.That leaves a revealing picture. The system had more information than ever, but fewer people were available to challenge it.The machine was faster, the humans were fewer and the information underneath both was wrong.That is why the familiar question, “Was it the AI or was it the human?”, misses the point. The AI does not need to make the original mistake for it to become part of an AI-enabled failure. If an old piece of information is treated as reliable, incorporated into a sophisticated system and then presented to decision-makers alongside hundreds of other data points, it can acquire a credibility it never deserved.

Why use AI at all?

It is tempting to conclude that the obvious answer is to take AI out of warfare. The military’s argument, however, is much harder to dismiss.Modern warfare produces an overwhelming amount of information. Satellites generate imagery, drones stream video, sensors track movements, intelligence agencies collect signals and analysts monitor open-source information. A modern battlefield can produce more information than humans can reasonably examine in real time.AI can search that ocean of data for patterns that would otherwise take people hours, days or even weeks to find.That is precisely why Maven has spread.

The US is betting that faster, AI-enabled decision-making can give it an edge on the battlefield.

In March, the Pentagon formally designated Maven a “program of record”, securing its place as a long-term military system and paving the way for broader adoption across the US armed forces. Reuters reported that the system analyses data from satellites, drones, sensors and intelligence sources and assists with targeting while retaining human oversight.The military therefore faces a genuine trade-off. Without AI, humans may miss important information simply because there is too much of it. With AI, humans can process vastly more information, but they also risk accepting a machine-assembled picture without adequately challenging the information beneath it.The problem is not simply that AI can be wrong. Humans have always been wrong. The problem is that AI can make a wrong answer move much faster and look much more complete.

Palantir and the machine at the centre

Maven is made by Palantir, whose software has become increasingly embedded in US military operations.The system can bring together more than 150 data inputs and help coordinate everything from intelligence and targeting to command and control. Bloomberg reported that, before the Iran war, target-list preparation that had previously taken hours could be condensed into minutes.That speed is the attraction. It is also the danger.A human looking at one outdated database entry might hesitate. A system connecting hundreds of pieces of information can make the same entry appear to be part of a much larger, coherent picture.After the Minab strike, Palantir added capabilities to Maven designed to re-review underlying intelligence, identify information that could disqualify a target and flag inconsistencies that human reviewers might have missed. According to Bloomberg, those tools have already identified anomalies.The fix for an AI blind spot, it seems, is more AI.That may work. But it raises the obvious question: if AI checks the information used by AI, where does the independent challenge come from?

The Chinese ship and the chatbot

The second case shows how quickly the chain can become dangerous even without a missile being fired.CNN reported in September that a false intelligence report circulated through the US military during the Iran war. It claimed that a Chinese vessel in the Middle East was carrying components linked to a nuclear weapons programme.The report had been prepared with the help of AI.A special operations analyst had used a chatbot that inaccurately identified the material the ship was carrying. The resulting information was turned into what appeared to be a conventional intelligence report and circulated through military channels.The military acted on it.Aircraft were in the air and armed personnel were preparing for a possible boarding. It was only shortly before the planned operation that officials examined the underlying intelligence more closely and discovered that AI had helped generate the report and that the chatbot had misidentified the cargo. The operation was halted.No shots were fired, but the incident demonstrates the same basic problem as Minab.A questionable piece of information entered the system. The system gave it a more authoritative form. Other people trusted that form, and the information travelled much further than the original mistake ever should have.One source told CNN the false report had “almost started a war”.The two cases are obviously different. Minab ended in mass civilian deaths; the Chinese ship incident ended with a last-minute correction. Yet they reveal the same vulnerability: a piece of bad information can become increasingly consequential as it moves through a chain of systems and people that assume the previous step was sound.

This problem did not begin with AI

Militaries have spent decades discovering that sophisticated systems can still make catastrophic mistakes.In 1999, NATO aircraft struck the Chinese embassy in Belgrade during the Kosovo war. Three Chinese journalists were killed. The strike was attributed to the use of outdated maps.In 1988, the USS Vincennes shot down Iran Air Flight 655 over the Strait of Hormuz, killing all 290 people aboard. The crew misidentified the civilian aircraft during a tense confrontation.In 2003, Patriot missile batteries shot down a British Tornado and a US Navy Hornet over Iraq after automated systems incorrectly classified the aircraft as hostile.These incidents were not caused by generative AI, but they belong to the same family of problems: bad information, complex technology, human pressure and excessive confidence in what the system appears to be saying.

The big question

The debate about military AI is often reduced to one dramatic question: Will a machine eventually pull the trigger by itself?That matters, but it may not be the most immediate question. The more urgent one is simpler: Who checks the information before the trigger becomes the final step?The Minab strike did not require a rogue machine. The underlying mistake existed years before the missile was launched. The Chinese ship incident did not require an autonomous weapon. A chatbot supplied a false piece of information that travelled into a military decision-making system.In both cases, the critical failure came before the final action. A human may still press the button, but by then, the hardest decision may already have been made.The real danger of the AI kill chain is not necessarily a machine deciding who lives and dies. It is one wrong fact moving fast through a system that assumes it is right, with nobody having enough time to stop it.So, in an age of AI-powered warfare, who checks the machine before the machine helps make the decision?



Source link

Exit mobile version