The U.S. and China are discussing the creation of an AI notification system that would alert each country to AI miscues related to national security. The initiative comes just days after revelations that the U.S. nearly attacked a Chinese ship based on mistaken analysis by AI that the vessel was transporting components of a nuclear weapon to the Middle East. The “entirely false” intelligence report “almost started a war,” a source told CNN, which first reported the apparent AI hallucination.
The AI alert proposal, reminiscent of the nuclear “hotline” between Washington, D.C., and Moscow during the Cold War, is unusual in that it was proposed in a Sunday meeting in New York with Chinese officials and U.S. Treasury Secretary Scott Bessent rather than with the White House or the Pentagon, which normally take the lead on national security issues. President Xi Jinping of China begins a state visit to Washington, D.C., this week.
The incident raises questions about the speed of AI adoption by the Pentagon after CNN revealed that the U.S. was only moments away from boarding an unnamed Chinese vessel based on faulty AI intelligence analysis. The incident occurred last spring as the U.S. was embroiled in military strikes against Iran. It’s unclear if China was aware of the incident prior to the CNN report. Bessent’s pitch to China almost looks apologetic if that is the case.
The hero of the moment appears to be a lone special operations command analyst who dug deeper into the intelligence report created by an AI chatbot and discovered that the AI had misidentified the cargo aboard the Chinese vessel. The chatbot initially was queried about intelligence reports originating with U.S. Special Operations Command, based in Hawaii. The chatbot apparently fused together open-source data with secret signals intelligence to come to an inaccurate conclusion. Compounding the error was the use of AI to disseminate the information via a standard intelligence report that is generally trusted by the military. Planes were “already in the air” by the time deeper analysis discovered that the AI was hallucinating.
It’s unclear which of the now numerous AI systems the Pentagon utilizes was involved in creating the inaccurate intelligence report. CNN anonymously quoted a former senior U.S. official familiar with the military’s AI systems as saying: “The internal tools are mostly copies of the commercial stuff wearing lipstick.”
Defense Secretary Pete Hegseth, with vocal support from President Trump, has advocated the accelerated use of AI in a bid to speed up decision-making on the battlefield. At issue, however, may be the speed of adoption without robust human oversight. A lengthy investigation by Bloomberg journalists suggests that the air strike that killed 123 schoolchildren in Iran in the early days of the conflict may be partially attributed to an overreliance on AI analysis. United Nations investigators have labeled the air strike as a possible war crime.
Critics note that American AI systems are decentralized, with different parts of the government using a constellation of AI systems with differing operating rules and safety standards that may affect functionality and reliability. This approach differs markedly from the centralized system used by Ukraine, which is emerging as the gold standard due to its ability to effectively integrate AI under a single umbrella in military operations that links drone use, intelligence gathering from sensors, traditional ground operations and other military capabilities.
While details are scarce to the point of puzzlement, a more centralized approach to AI military use may be what’s behind President Trump’s plan to create an “AI Force” similar to the U.S. Space Force. A plan to appoint an “AI Czar” appears to be separate from that of an AI Force. But both moves appear to be in response to mounting public concern over the lack of AI guardrails that would offer protection against “misaligned” AI models even as President Trump labels such concerns a “hoax.” President Trump’s “sprawling family business empire has become increasingly intertwined with the boom in artificial intelligence,” reports The Washington Post, adding that the president “has a personal financial stake in industries benefitting from the AI build-out, creating potential conflicts of interest as his administration makes decisions that could affect the value of those investments.” Whether those potential conflicts determine if an AI Force becomes a reality or is a social media hallucination remains to be seen.
