Abi, a Manchester resident, has been using ChatGPT for the past year to help manage her health, she said on Apr. 19. Abi finds that AI chatbots offer tailored advice and are more accessible than traditional general practitioners. However, she also experienced problems when the chatbot gave her incorrect medical advice after an injury.
The use of AI chatbots like ChatGPT, Gemini, and Grok for health information is growing in popularity as people seek quick answers online. This trend raises concerns among medical professionals about the reliability of such tools. England’s Chief Medical Officer Prof Sir Chris Whitty said earlier this year that “we’re at a particularly tricky point because people are using them,” but the answers were “not good enough” and often “both confident and wrong”.
Abi described how ChatGPT advised her to visit a pharmacist for urinary tract infection symptoms—a recommendation that led to proper care without feeling like she was overburdening healthcare services. But after suffering a fall while hiking in January, Abi consulted ChatGPT again: “Chat GPT told me that I’d punctured an organ and I needed to go to A&E straight away,” she said. After waiting in emergency care for hours with easing pain, Abi realized the chatbot had misjudged her situation.
Research from Oxford University’s Reasoning with Machines Laboratory found that chatbots performed well—95% accuracy—when given complete medical scenarios by doctors. However, when ordinary users interacted with chatbots by sharing information gradually or incompletely, accuracy dropped sharply to just 35%. Professor Adam Mahdi explained: “When people talk, they share information gradually, they leave things out and they get distracted.” He also noted that users who searched traditionally often ended up on reliable NHS websites and were better prepared.
A separate study by The Lundquist Institute for Biomedical Innovation found more than half of AI chatbot responses across topics like cancer or vaccines were problematic when questions invited misinformation. Lead researcher Dr Nicholas Tiller warned: “They are designed to give very confident, very authoritative responses… so the user assumes it must know what it’s talking about.” Tiller suggested avoiding chatbots for health advice unless users have enough expertise to spot errors.
OpenAI stated: “We know people turn to ChatGPT for health information… we take seriously the need to make responses as reliable and safe as possible.” The company emphasized their collaboration with clinicians but reminded users not to replace professional medical advice with chatbot guidance.
Abi continues using AI chatbots but advises caution: “I wouldn’t trust that anything that it’s saying is absolutely right.” She recommends taking all advice from these tools with skepticism.
