A bipartisan coalition of state attorneys general is intensifying pressure on major tech companies over concerns that AI chatbots are producing harmful and legally questionable responses, particularly for children and vulnerable users.
In a letter sent to 13 technology companies, including Apple, Microsoft, Google, Meta, OpenAI, Anthropic, and Character.AI, attorneys general from 42 states warned that current chatbot behavior could expose companies to liability under state consumer protection and criminal laws. The letter focuses on what regulators describe as “sycophantic and delusional” AI outputs that may encourage harmful behavior or reinforce false beliefs.
The attorneys general cited media reports and documented cases in which chatbots allegedly validated users’ delusions, offered inappropriate guidance on mental health issues, or blurred the line between automated systems and human interaction. In one widely reported incident, a teenager discussed suicidal thoughts with an AI chatbot, raising alarms among state officials about the technology’s role in high-risk situations.
The Need for Safeguards
New York Attorney General Letitia James, who helped lead the coalition, said the reports point to a pattern of insufficient safeguards. “Big Tech companies must do more to stop their AI chatbots from encouraging harmful and sometimes deadly behaviors,” James said in a statement. She emphasized that children, seniors, and people with existing mental health conditions face heightened risks when interacting with generative AI systems.
The coalition’s letter argues that chatbot developers may already be violating state laws that prohibit encouraging criminal acts, providing unlicensed mental health advice, or engaging in deceptive practices. In several states, encouraging drug use, violence, or self-harm a trigger criminal liability. Attorneys general also warned that misleading users into believing they are interacting with a human could raise consumer deception concerns.
To address these risks, the group called on companies to adopt a series of safeguards. Among the requests are clearer warnings about the limitations and potential dangers of AI-generated responses, notifications to users who may have been exposed to harmful outputs, and greater transparency around training data and known weaknesses in AI models. The attorneys general also urged companies to allow independent third-party audits of chatbot systems, giving regulators and outside experts the ability to evaluate risks.
“Failing to adequately implement additional safeguards may violate our respective laws,” the letter states, signaling that enforcement actions remain a possibility if companies do not respond.
Minimal Response from Companies
Company responses have so far been limited. Microsoft and Google declined to comment on the letter, while Meta and Apple did not respond to requests for comment. OpenAI and other AI-focused firms named in the letter have previously said they are working to improve safety controls, though the attorneys general argue that voluntary measures have not kept pace with the technology’s rapid expansion.
The coalition also highlighted the growing integration of AI into everyday tools, from smartphones to social platforms, which increases the potential scale of harm. As chatbots become more conversational and emotionally responsive, regulators worry that users may place undue trust in systems that lack true understanding or accountability.
While the letter stops short of announcing formal investigations, its tone reflects rising impatience among state officials. With bipartisan backing and participation from nearly every region of the country, the effort signals that scrutiny of AI chatbot behavior is likely to intensify, regardless of how federal policy debates unfold.
