AI Error Nearly Triggered U.S. Operation Against Chinese Vessel
U.S. military aircraft were already airborne this spring when officials discovered that intelligence behind an armed operation against a Chinese vessel had been generated from an AI chatbot’s hallucination, CNN reported. The mission was called off at the last moment, avoiding a possible confrontation with China.
The intelligence report circulated during the war with Iran and claimed that the vessel was transporting components for a nuclear weapons program. A Special Operations Command analyst had asked a chatbot to combine open-source material with classified signals intelligence. The system incorrectly identified the ship’s cargo manifest. The analyst then used the chatbot again to turn the mistaken assessment into a formal-looking summary, which moved through command channels.
The incident highlights concerns about AI errors reaching senior decision-makers before they are challenged. The Pentagon has said AI can help accelerate the military “kill chain,” but that speed may also spread false information when human review is insufficient. Jake Steckler, a GovAI research scholar and former U.S. Army officer, said service members must understand the uncertainty of large language models, particularly when decisions could lead to the use of force. He said safeguards are needed, while warning that prioritizing rapid adoption could undermine trust in the systems.
Written by Sakhanda Wire AI from reporting by TechCrunch. Facts, quotes and full context are in the original article below.