Is AI a Safe Doctor? The Risk and Dangers of Getting Medical Advice from A Chatbot?

With the rise of artificial intelligence, more people are turning to AI-powered tools to ask questions about their health. It is estimated that over 230 million people globally use AI Chatbots for health concerns each year. While these tools are accessible and fast, recent research suggests they have important limitations, particularly when being used to guide real-world medical decisions.

What Does the Research Actually Show?

Several recent studies have examined how individuals use AI to understand symptoms and decide on next steps, such as whether to see a GP or go to a hospital. A study conducted by researchers at the University of Oxford (Bean et al., 2025) with nearly 1,300 participants found that individuals using AI tools did not make better health decisions than those relying on traditional methods, such as online searches or their own personal judgements.

Researchers also identified a communication breakdown (human-LLM interaction failures) between users and AI systems:

• Many participants were unsure what information to provide to get accurate advice. Researchers reported that overall, participants often unsuccessfully provided the models with adequate information with 16 out of 30 initial messages only containing partial information.
• AI responses often include a mix of correct and incorrect recommendations. Researchers found that the large language models suggested an average of 2.21 possible conditions per interaction, of which only 34% possible conditions were correct. This means patients were frequently given unreliable or confusing suggestions that could easily lead to the wrong medical decision or unnecessary concern.
• The manner participants phrased information to the chatbots could lead to very different answers. AI models correctly identified relevant medical conditions in up to 94.9% of cases when assessed in isolation (control group). However, when it came to recommending the appropriate course of action, such as calling an ambulance or going to a doctor, accuracy dropped significantly, to as low as 56.3%. Conversely, when real users were involved, correct identification of conditions occurred in less than 34.5% of cases, and appropriate next steps were identified in less than 44.2%, highlighting the gap between controlled testing and real-world use.

Why This Matters Especially for Mental Health?

Mental health rarely presents in clear or standardised way. Symptoms such as anxiety, difficulty concentrating, or fatigue can overlap across multiple psychiatric and physical conditions and are often influenced by personal, social and environmental factors.

AI tools are not able to fully interpret context, personal history or nuance. This implies important details may be overlooked, and recommendations may not reflect the full picture of a person’s mental health.

For example, subtle difference in how symptoms are described, such as “worst headache ever” as opposed to “terrible headache”, has been shown to produce very different recommendations from AI systems despite potentially indicating the same condition.

A useful tool but not a diagnosis.

Artificial intelligence can still be helpful in healthcare when used appropriately. It may assist individuals in:

• Understanding medical terminology
• Learning general information about symptoms
• Preparing questions before a doctor’s appointment.

However, it should be viewed as an educational resource, not a diagnostic tool or a replacement for professional care.

Why Speaking to a Psychiatrist Still Matters?

For those seeking a Psychiatrist in Sydney or elsewhere in Australia, speaking directly with a qualified professional remains the most reliable way to receive accurate diagnosis and care. A qualified psychiatrist considers far more than a list of symptoms. Assessments involve conduct of a physical examination, understanding your current and historic medical and psychiatric history, family history, psychosocial and developmental history, forensic and drug and alcohol history, education, employment, family and personal relationships, current circumstances, presenting symptoms, and how these are affecting your life. This allows for accurate diagnosis and a personalised treatment plan, something AI tools are not currently able to provide.

Source:

• Bean, A. M., Payne, R. E., Parsons, G., Kirk, H. R., Ciro, J., Mosquera-Gómez, R., Sara, H. M., Ekanayaka, Aruna S, Tarassenko, L., Rocher, L., & Mahdi, A. (2026). Reliability of LLMs as medical assistants for the general public: a randomized preregistered study. Nature Medicine, 1–7. https://doi.org/10.1038/s41591-025-04074-y