Is ChatGPT Accurate for Medical Questions?
Updated
ChatGPT is often accurate on medical questions, but not reliably enough to use without checking. Published studies found it passing sample USMLE questions and giving physicians largely accurate answers, but also sometimes recommending treatments outside cancer guidelines and, in early versions, inventing about half its references. Below: what the studies found, where general chatbots go wrong, how to check any AI medical answer, and how AskMed, an AI medical reference for doctors, nurses, pharmacists, medical students and researchers, is built so every claim can be checked.
- Licensing examsAt or near the USMLE pass mark in 2023 (GPT-3.5); near-perfect scores by 2026
- Physicians' own questionsMedian accuracy 5.5 out of 6, lower on hard questions (2023)
- Cancer treatment advice34.3% of answers with a recommendation included one outside NCCN guidelines (2023)
- References47% fabricated and 46% inaccurate in one GPT-3.5 study (2023)
How accurate is ChatGPT on medical exams?
ChatGPT scores well on medical licensing exam questions, and its scores have risen with each new model. In the best-known early study, Kung and colleagues (PLOS Digital Health, 2023) gave ChatGPT, running GPT-3.5, 350 questions from the June 2022 public USMLE sample exam, after removing questions with images, and found it performed at or near the passing threshold of about 60% on Step 1, Step 2 CK and Step 3. Later in 2023, Brin and colleagues (Scientific Reports, 2023) tested 80 USMLE-style questions on communication, ethics, empathy and professionalism: GPT-4 answered 90% correctly and ChatGPT on GPT-3.5 answered 62.5%, against a 78% average for AMBOSS users on the same questions. By 2026, the authors of a Nature Medicine study wrote that large language models now achieve nearly perfect scores on medical licensing exams. For textbook knowledge, ChatGPT is strong.
An exam question, though, hands over every relevant fact in a clean vignette with one best answer. A real clinical question has an incomplete history, guidance that may have changed, and one patient to fit.
How accurate is ChatGPT on real clinical questions?
On open clinical questions, ChatGPT is right most of the time but not consistently. Goodman and colleagues (JAMA Network Open, 2023) had 33 physicians from 17 specialties write 284 medical questions and grade ChatGPT's answers: the median accuracy score was 5.5 on a 6-point scale, between almost completely and completely correct, but the mean was 4.8, and hard questions scored lower than easy ones. Low-scoring answers improved when the questions were asked again 8 to 17 days later, and GPT-4 scored higher than GPT-3.5. Chen and colleagues (JAMA Oncology, 2023) asked ChatGPT for breast, prostate and lung cancer treatment advice: 34.3% of the 102 answers that made a recommendation included at least one treatment not concordant with NCCN guidelines, and 12.5% of all 104 answers named a treatment that was not part of any recommended treatment. The wrong recommendations were mixed in among correct ones, which the authors called hard even for experts to spot.
Knowing the answer is also not the same as helping a person reach it. In a randomized trial by Bean and colleagues (Nature Medicine, 2026), models including GPT-4o identified the condition in 94.9% of written scenarios when tested alone, but 1,298 UK participants using them identified relevant conditions in fewer than 34.5% of cases, no better than a control group using their usual sources.
Does ChatGPT make up medical references?
ChatGPT has made up medical references at high rates when asked to write with citations from memory, although newer models do it less often. Bhattacharyya and colleagues (Cureus, 2023) asked ChatGPT on GPT-3.5 to write 30 short medical papers with at least three references each. Of the 115 references it produced, 47% were fabricated, 46% were real but inaccurate, and only 7% were both real and accurate. Walters and Wilder (Scientific Reports, 2023) had ChatGPT write short literature reviews on 42 topics and checked all 636 citations: 55% of the GPT-3.5 citations and 18% of the GPT-4 citations were fabricated, and among the real ones, 43% and 24% had substantive errors such as a wrong volume, issue or page number. A fabricated reference is formatted like a real one, with authors, a journal and a year, so it can pass a quick glance and only fails when you search for it.
An 18% rate still means roughly one invented reference in five, so search every chatbot reference in PubMed before you cite it.
Where do general chatbots go wrong?
General chatbots such as ChatGPT go wrong in medicine in four predictable ways, and none of them means the model knows little medicine. First, a reference written from memory can be invented, as the citation studies show. Second, a model answers from training data that stops at a cutoff date: in the 2023 cancer study, the authors compared answers with the 2021 NCCN guidelines because the model's knowledge ended in September 2021, so any later change in guidance was invisible to it. Third, an answer written from training data shows no source, so you cannot see where a claim came from or how strong the evidence behind it is. Fourth, the tone stays equally confident whether the answer is right or wrong: in Brin's 2023 study, GPT-4 changed none of its answers when asked whether it was sure, the wrong ones included. An error mixed in among correct statements, stated with the same confidence, is the hardest kind to catch.
To be fair to ChatGPT, its interface warns that it can make mistakes, newer models fabricate fewer references, and when ChatGPT searches the web it links the pages it used. That search isn't limited to journals, guidelines and drug labels, though, and a link only helps if you open it.
Is ChatGPT good for medical students and research?
ChatGPT is a useful study partner for medical students and researchers, as long as every fact that matters is checked. For students, ChatGPT is good at explaining a mechanism in plain words, talking through why a differential diagnosis makes sense, quizzing you on a topic and rephrasing a dense paragraph, and the exam studies show it knows most textbook medicine. The risk is a fluent explanation that is wrong in one detail, which you then learn as fact. For exam preparation, a question bank whose explanations were written and checked against sources is the safer core, with a chatbot alongside it. For researchers, ChatGPT can help with brainstorming, outlining and plain-language summaries of text you paste in, but the citation studies show why it should not be trusted to supply references. Every reference from a chatbot needs to be found and read before it goes into a manuscript or a journal club slide.
For a side-by-side look at exam tools, including where ChatGPT fits, see the best AI apps for USMLE.
How can you check an AI medical answer?
You can check any AI medical answer, from ChatGPT or any other tool, with the same five steps. Start by finding the source behind the specific claim you plan to use; if the answer gives none, treat the claim as unverified until you find it in a guideline, a drug label or a peer-reviewed paper. Open the source and confirm that it exists and says what the sentence says, for the same group of patients, since a trial in adults does not answer a question about children. Check the date, because a guideline updated after a model's training may have changed the recommendation. Check whose guidance it is: NICE, US and European guidelines can differ, so follow the one that applies where you practise. Finally, take any drug dose from the current prescribing information or your local formulary, never from a chatbot's answer. These checks take minutes, and they catch the errors the studies above describe.
- Red flag: a guideline with no year. You can't tell whether the recommendation is still current.
- Red flag: certainty on an open question. Where evidence is weak or guidelines disagree, a good answer says so.
How is AskMed built differently?
AskMed is built so that every claim in an answer can be checked, instead of relying on what a model remembers. Before any answer is written, AskMed searches a fixed list of medical sources: journals and databases such as PubMed, the Cochrane Library, NEJM, The Lancet, JAMA and The BMJ; guidelines and agencies such as NICE, WHO and CDC; FDA and DailyMed drug labels; and references such as StatPearls and BMJ Best Practice. AskMed does not search the open web, forums or social media. An AI model then writes the answer from what the search found, and each claim is followed by a link to its source at that source's exact web address, from 8 sources by default. If no usable source turns up, AskMed shows a fixed reply saying so instead of answering from memory. For management questions, AskMed looks for the latest guideline update and is instructed to give the year of each guideline recommendation it states.
AskMed never gives drug doses; exact dosing comes from the prescribing information. AskMed's answers are AI-generated too, with no human review before you see them, so AskMed can misread a source. The difference is that each sentence carries a link you can open. AskMed works on iPhone, iPad, Android and the web. Full detail: How AskMed answers.
Frequently asked questions
Is ChatGPT accurate enough for medical advice?
ChatGPT is accurate on many medical questions, but not accurate enough to act on without checking. Physicians in a 2023 JAMA Network Open study gave its answers a median accuracy score between almost completely and completely correct, yet in a 2023 JAMA Oncology study about a third of its cancer treatment answers included at least one treatment outside NCCN guidelines. A 2026 Nature Medicine trial found that members of the public using chatbots identified the relevant condition in fewer than 34.5% of scenarios, no better than people using their usual sources. For a patient, a chatbot answer is not a substitute for a doctor, and in an emergency the right step is to call the local emergency number. For a clinician or student, a ChatGPT answer is a starting point: find the guideline, label or paper behind each claim you use, check that it is current, and take any dose from the prescribing information.
Is ChatGPT accurate for medical diagnosis?
Not reliably. In the 2026 Nature Medicine trial, models tested alone named the condition in 94.9% of written scenarios, but people using them got there in fewer than 34.5% of cases, partly because they often gave the models incomplete information. Diagnosis depends on history, examination and tests, so use a chatbot to think through a differential, not to make the diagnosis. AskMed is not a symptom checker either and doesn't diagnose patients.
Does ChatGPT give real references?
Sometimes. When ChatGPT searches the web, it links the pages it used. References written from memory were often fabricated in the studies above, so search each one in PubMed before citing it.
Are newer versions of ChatGPT more accurate than the ones studied?
Yes, on every measure compared so far: GPT-4 beat GPT-3.5 on exam questions, physicians' questions and references. Most studies test older versions, but no study shows newer ones no longer need checking.
Can AskMed be wrong?
Yes. AskMed's answers are written by AI, so check the linked source behind any claim you act on, and tap the thumbs down under an answer to report an error.
Does AskMed give drug doses?
No. AskMed never gives doses, dose calculations or titration schedules, even for exam questions. It still explains a drug's uses, how dosing is chosen and adjusted, monitoring and side effects. Exact doses come from the current prescribing information.