A study by Columbia University’s Tow Center for Digital Journalism conducted in March 2025 raises a troubling question: Can we trust chatbots to accurately convey the news?
The answer, according to their analysis, is — not yet.
The study found that chatbots misinterpreted information from news sources six out of ten times — often fabricating links, citing syndicated versions of articles instead of originals, and even bypassing Robot Exclusion Protocols meant to prevent automated scraping of content.
“We found that chatbots were generally bad at declining to answer questions they couldn’t answer accurately,” the researchers wrote. “Instead, they offered incorrect or speculative answers . . .”
According to the analysis, this isn’t just a ChatGPT issue; the problem “rather recur[s] across all the prominent generative search tools that we tested.”
The Center asked eight generative search tools to identify source articles, the publication, and URL from 200 excerpts from news articles randomly selected from 20 publishers. The search tools were ChatGPT, Perplexity, Perplexity Pro, DeepSeek, Copilot, Grok-2, Grok-3 and Gemini.
“Most of the tools we tested presented inaccurate answers with alarming confidence, rarely using qualifying phrases such as “it appears,” “it’s possible,” “might,” etc., or acknowledging knowledge gaps with statements like “I couldn’t locate the exact article.” ChatGPT, for instance, incorrectly identified 134 articles, but signaled a lack of confidence just fifteen times out of its two hundred responses, and never declined to provide an answer.”
“With the exception of Copilot—which declined more questions than it answered—all of the tools were consistently more likely to provide an incorrect answer than to acknowledge limitations.”
Nearly one in four Americans say they have used artificial intelligence (AI) in place of traditional search engines, according to the analysis. As search tools gain more popularity by the day, their accuracy is being scrutinized. The analysis is a follow-up to a previous analysis by the Center conducted in November 2024. Both analysis yielded similar results.
“The findings of this study align closely with those outlined in our previous ChatGPT study, published in November 2024, which revealed consistent patterns across chatbots: confident presentations of incorrect information, misleading attributions to syndicated content, and inconsistent information retrieval practices. Critics of generative search like Chirag Shah and Emily M. Bender have raised substantive concerns about using large language models for search, noting that they “take away transparency and user agency, further amplify the problems associated with bias in [information access] systems, and often provide ungrounded and/or toxic answers that may go unchecked by a typical user,” says the report.
“These issues pose potential harm to both news producers and consumers,” the report continues. “Many of the AI companies developing these tools have not publicly expressed interest in working with news publishers. Even those that have often failed to produce accurate citations or to honor preferences indicated through the Robot Exclusion Protocol. As a result, publishers have limited options for controlling whether and how their content is surfaced by chatbots—and those options appear to have limited effectiveness.”
Danielle Coffey, the president and CEO of the News/Media Alliance, which represents over 2,200 publishers in the U.S., from large outlets to small local ones, said AI has moved so fast that the laws have yet to catch up to the technology.
“Both content creators and AI companies need a clear set of rules to operate by, and users need to know what information they can trust,” she said. “For publishers and other creators to be able to protect their rights under existing law, it is essential that AI companies be required to transparently disclose the content they take and use to train and deploy their models. The Task Force (Bipartisan Artificial Intelligence Task Force) noted that transparency requirements ‘would potentially allow creators to see whether their copyrighted content was included in the training corpus of an AI model to identify infringement,’ and ‘could allow users of the AI system to assess and understand outputs’.”
