Clinical artificial intelligence (AI) demonstrated a remarkable diagnostic accuracy in a recent study of more than 102,000 real-world primary care clinical cases.
The study, co-authored by Dan Zeltzer, a health economist from Tel Aviv University, and Dr. Jon O. Ebbert from Mayo Clinic, was published by K Health, a clinical AI provider.
Clinicians agreed with AI-generated diagnoses 84.2% of the time, and in 57 common primary care diagnoses, AI and clinicians showed over 90% agreement. Independent adjudicators, unaware of the source, favored the AI’s differential diagnosis as often as the clinicians.
The study highlights AI’s potential to revolutionize healthcare decision-making in primary care, with implications for pre-visit medical intake and differential diagnosis.
Dr. Harvey Castro, a physician and health care consultant, explains the 84% agreement rate between clinicians and AI-generated diagnoses is remarkable.
“However, it’s worth noting that this percentage needs to increase to gain widespread acceptance among physicians,” he says. “Interestingly, some diagnostic tests we currently use in medicine have lower predictive values–for example, strep tests have a predictive value of around 70 percent.”
He notes that while some physicians may demand AI accuracy in the upper 90s, others may find the current level acceptable, given these comparisons.
Allon Bloch, co-founder and CEO of K Health says the results imply strong trustworthiness and reliability for predictive AI models that are comprehensively evaluated on a diverse set of patients in a real-world setting.
One notable result is that AI demonstrated no biases across gender, age and ethnicity in generating differential diagnoses, which Castro calls “a monumental achievement” but adds it’s one he would like to see replicated in further studies.
“The ability of AI models to generalize to a diversity of patients is an important factor when considering deployment in a health care practice and it plays a critical role in ensuring equitable outcomes,” Bloch explains. “Developing and testing AI models on diverse data sets and having awareness of data biases are measures to promote unbiased AI.”
He adds K Health’s model development and validation process is also designed to assess and reduce disparities in performance across patient groups.
“Preferences are notoriously difficult to eliminate, especially considering human biases often translate into machine-learning models,” Castro adds. “Additionally, tendencies can emerge from the specific disease populations studied, which may not represent broader demographics.”
The report also mentions the AI can improve through a unique reinforcement learning loop based on clinician feedback.
“The clinician feedback loop is perhaps the most critical aspect of this AI system,” Castro says. “It’s essential that the reinforcement learning is specialty specific. For instance, as an ER doctor, I can provide valuable feedback for an ER-focused machine learning model, but I would have no business reinforcing a surgical model.”
Bloch notes data from patient interactions on the platform are anonymized and used to retrain the AI models within K Health’s models.
“Candidate retrained models are then evaluated on multiple validation sets before release to ensure quality improvement,” he adds.
Testing and improving the company’s models based on data from rich, real cases seen on the platform allows the models to learn with finesse, which ultimately enhances quality of care and patient outcomes.
Castro says while the study’s findings are promising, it’s crucial to remember that AI should complement human expertise, not be a replacement.
“The nuanced understanding and emotional intelligence that healthcare providers bring to patient care cannot be replicated by AI,” Castro cautions.
Bloch says he imagines AI involvement in all aspects of primary care from wellness optimization, diagnosis and treatment to chronic condition management – all of which will enable better access to care.
“Patients and physicians will have access to higher quality information allowing for more preventative actions and better outcomes,” he explains. “Health care professionals will be able to spend more time with patients and operate at the top of their license.”