Journalists in fear of losing their jobs to generative artificial intelligence (AI) can take solace in this headline: OpenAI’s ChatGPT, Microsoft’s Copilot, Google’s Gemini and Perplexity AI did a lousy job of accurately summarizing the news, a new BBC study found.
The widely respected news organization fed the bots content from its news website and asked them basic questions about the news. More often than not, the answers contained “significant inaccuracies” and distortions, the BBC said.
Reporters and category experts were asked to evaluate the chatbot’s answers, and uniformly gave them poor marks. More than half (51%) of all AI-generated answers were flawed, with 19% of the responses introducing errors such as misstatements of facts, numbers, and dates.
Blunders ranged from patently wrong to missing nuance: ChatGPT and Copilot both incorrectly indicated politicians Rishi Sunak and Nicola Sturgeon to be still in office. Perplexity said Iran showed “restraint” while Israel’s actions were “aggressive” in a report on the Middle East that did not use either adjective.
“The price of AI’s extraordinary benefits must not be a world where people searching for answers are served distorted, defective content that presents itself as fact,” BBC News and Current Affairs CEO Deborah Turness wrote in a blog post summarizing the findings. “In what can feel like a chaotic world, it surely cannot be right that consumers seeking clarity are met with yet more confusion.”
BBC has a vested interest in the results. Like many news organizations, it is exploring a number of AI initiatives and holding talks with tech companies to develop new automated tools. But it, like most enterprises, is weighing the risks of diving into the fast-evolving AI technology without sacrificing accuracy of information, compliance, security, and privacy.
The BBC’s findings come a few weeks after Apple Inc. ceased sending news alerts via AI because several were far off base. One gaffe falsely generated from BBC News said Luigi Mangione, the man accused of fatally shooting UnitedHealthcare CEO Brian Thompson, had shot himself. (Mangione is in jail awaiting trial.)
Another informed some BBC Sport app users that tennis star Rafael Nadal had come out as gay.
“We live in troubled times, and how long will it be before an AI-distorted headline causes significant real world harm?” Turness warned.
