Unready for war, AI may already be causing deadly mistakes

Secretary of Defense Pete Hegseth has made a very heavy bet on AI in the military, and it’s doubtful that he will be entirely forthcoming about this or other incidents to come. Targeting errors certainly aren’t new, but people should demand information about whether AI is making errors worse or more frequent.

Generative AI continues to have serious problems with visual cognition, as computer scientist Anh Totti Nguyen has shown in a string of papers, the most  recent of which is called “Vision language models are blind: Failing to translate detailed visual features into words.” That paper warned that such models “consistently struggle with those tasks that require precise spatial information when geometric primitives overlap or are close.”

What Nguyen has found is particularly chilling if one looks at the satellite images that The New York Times shared, suggesting that the mistargeting could have been a problem of interpreting spatial information with respect to entities that were close together—the Shajarah Tayyebeh elementary school and an IRGC compound, both in the town of Minab located in southern Iran.

Another known issue with language models is their weakness with reasoning; and poor temporal reasoning might have been a problem here. A post on X from the foreign correspondent Louisa Loveluck argues that the incident might have been “based on intel that is a decade old. If military commanders reviewed recent publicly available satellite imagery, they would have seen a school with a sports field among the sites apparently designated for a precision strike.”

It’s easy to see how an AI system might have had access to old intel and new intel and failed to give precedence to the new information. This is especially true if the old intel was more represented in the AI training data set than the new intel. (Even if humans were in the loop and doublechecking AI’s work, they might have missed it, because people all too often defer to these systems without careful examination.)

To be clear: I do not know whether the Minab school bombing involved AI targeting, because the US military has revealed very little about the incident. But unless and until the military does actual empirical studies on collateral damage in the AI era, it won’t be clear whether AI is helping or hurting military missions. Mistargeting isn’t new, but using unreliable AI on vibes is fraught with peril.

More broadly, the military use of AI needs to be a granular question. Experts might find, for example, that AI helps with logistics and planning but makes more errors in targeting. Mileage may vary according to the task, and may be worse in unfamiliar situations, given the inherent tendencies of generative AI.

The technical problem is that current AI simply isn’t reliable; mistakes will absolutely be made. Some will cost lives; some will cost many lives. Some may lead to further escalation (a mass killing of school children could well do that); in the worst case, a series of escalations sparked by AI-triggered mistakes could even lead to a nuclear war. Given the current status in the Middle East, this concern is not merely academic.

Beyond the technical, there’s a moral problem: Militaries may well wish to use AI to cloak responsibility. One can, for example, use an AI tool to select targets that would be considered war crimes if hit, and blame AI when they are.

It is important to realize that real choices are made at the front end, by those who use the AI. How many civilian casualties are acceptable? What error rate is permissible? AI can follow a set of criteria (with varying degrees of precision depending on the quality of the algorithms and data), but humans set those criteria. In my own view the biggest problem with the algorithms targeting Gaza was not necessarily the algorithms per se (about which not much may be public) but the decision to tolerate a large number of civilian casualties as part of the targeting.

By analogy, if one rolled dice (a physical instantiation of a very simply algorithm) to pick targets, one would not blame the dice for the deaths, but those who chose to leave life or death to chance in the first place.

Humans should absolutely want any AI that is used for war to be as precise and reliable as possible, minimizing casualties, but people should also never forget that those who use them are responsible for decisions about how many casualties are acceptable. And it should be incumbent on them to understand the limitations and inaccuracies of the algorithms they choose.

Whether or not current AI algorithms are precise (they probably aren’t), and whether humans are involved in the specific selection of targets, those who use military algorithms bear responsibility for the outcomes they produce.

The race to shove AI into everything is grossly premature, because the tech fundamentally lacks reliability.

Meanwhile, the chance that people will get straight answers on what happens in this war and likely others is probably close to zero.

Many people, perhaps thousands, maybe more, will die, needlessly. And as Bloomberg columnist Parmy Olson just argued, “all of this has been happening in a regulatory vacuum and with technology that is known to make errors.”

Anyone who cares about humanity should be deeply concerned.