Study warns AI chatbot medical advice errors may go undetected

Researchers have issued a warning regarding the reliance on artificial intelligence-powered chatbots for medical guidance, stating that damages resulting from incorrect health information may pass unnoticed by physicians and regulatory bodies. The primary concern stems from the difficulty of tracking conversations conducted between users and these digital systems.

Challenges in detecting AI health risks

A study published in the journal "Nature Health," involving researchers from Binghamton University, Stanford University, Texas A&M University, and Indiana University, highlights that the issue extends beyond the potential for inaccurate medical data. The core problem lies in the inability to detect errors and assess their impact on user health, particularly when interactions remain confined within AI platforms.

Unlike traditional medical consultations, which are documented in health records, or drug side effects reported through dedicated systems, AI chatbot conversations may not be accessible to doctors, researchers, or regulators. This lack of visibility limits the capacity to identify recurring mistakes and evaluate associated risks.

Case study of severe bromide poisoning

The researchers cited a specific instance involving a 60-year-old man who sought advice from the "ChatGPT" bot on reducing chloride intake in his diet. The system suggested replacing table salt with sodium bromide. After following this recommendation, the man suffered from bromide poisoning, leading to hospitalization and symptoms including hallucinations and pathological suspicions.

This case illustrated the difficulty in pinpointing the source of the error, as doctors could not accurately determine the advice received because the conversation was stored exclusively within the AI platform. The absence of effective monitoring mechanisms makes detecting harm linked to erroneous medical advice significantly more challenging.

Policy recommendations for accountability

To mitigate these dangers, the study proposes several measures, including enabling users to access their conversation logs and developing systems capable of detecting and reporting harmful health advice. These steps would assist in identifying errors and better understanding their consequences.

Furthermore, the researchers urged policymakers to consider expanding legal liability to encompass AI companies providing medical guidance. They argue that such entities should face levels of accountability proportional to the potential risks posed by the advice they offer.

The findings emphasize the need for caution when handling medical information provided by chatbots, noting that a confident tone does not guarantee accuracy. Consulting doctors and specialists remains essential for decisions related to diagnosis, treatment, or dietary changes based on specific health conditions.