The controversial role of AIs as dispensers of medical advice is in focus as Pennsylvania brings a groundbreaking suit against an AI company for falsely claiming to be a licensed doctor. The state’s suit comes on the heels of a new study that indicates that AI outperforms doctors in making the correct diagnosis in emergency room settings.
The Pennsylvania lawsuit takes aim at Character.ai, which has some 20 million users. State investigators found that AI chatbot characters claimed to be licensed medical professionals, including psychiatrists who engaged in conversations with users about mental health symptoms. In addition, the Character Technologies, Inc. platform allegedly allowed users to create and deploy “characters” trained to have a specific personality that presented themselves as professionals. With millions of people regularly consulting an AI for wellness advice, fraudulent interactions can have profound consequences.
“We will not allow companies to deploy AI tools that mislead people into believing they are receiving advice from a licensed medical professional,” said Pennsylvania Gov. Josh Shapiro.
The alleged duplicity came to light when a state investigator engaged with a chatbot named “Emilie” which described itself as a psychology specialist attending medical school at the Imperial College in London. The chatbot said “it was within my remit as a doctor” to prescribe the investigator medication to treat depression symptoms. Emilie said it was licensed in the Keystone State and provided an invalid license number.
Character.ai has been in the spotlight before. Multiple families sued the company last year alleging it contributed to their children’s suicide or mental health problems. The company says its characters should be treated as fictional.
A central question is whether AI should be involved in medical diagnostics at all. The answer seems to be a qualified “yes.” In a new study published by the journal Science, Harvard Medical School researchers found that while two doctors in an emergency room situation made exact or near-exact diagnoses in 50% and 55% of the cases, an AI was close or exact 67% of the time. The study used clinical records of 76 Beth Israel patients and employed OpenAI’s o1 model. The experiment had no impact on actual patient care.
One impressive result of the Harvard study was that AI was able to correctly decide on the next best step for a patient 89% of the time. By contrast, physicians using conventional resources like search engines and medical databases were correct only 34% of the time.
“This is a big conclusion for me—it works with the messy real-world data of the emergency room,” said Dr. Adam Rodman, one of the study’s authors, in an NPR interview. “It works for making diagnoses in the real world.”
The researchers noted that the study relied on text-only inputs while real-world medicine also involves visual cues, physical cues and the patient’s voice. The results are applicable to emergency and internal medicine but not surgery.
Caution also is advisable in making AI the first layer of patient interaction. A new study published in Nature Health reveals that when patients describe their symptoms to an AI rather than medical professionals, the quality of that information markedly declines. The phenomenon is known as “uniqueness neglect” as people believe a machine can’t possibly understand the nuances of their personal situation and they clam up.
Doctors have been using AI for tasks like transcribing patient notes, reviewing health scans and early disease detection. The new Harvard study suggests AI now possesses a level of computational and reasoning ability that can support a medical diagnosis. Doctors may want to ask an AI for a second opinion, especially in stressful, time-sensitive emergency room settings. But as Pennsylvania’s lawsuit against Character.ai suggests, that second opinion may only be as good as the AI model being employed. Patients may soon be asking their doctors: “Who’s your AI?
