Is OpenEvidence Accurate? What the Studies Found
Updated
OpenEvidence is accurate on most clinical questions in published tests, and its references are real, but it still makes errors, and published studies disagree on how it compares with general AI chatbots. OpenEvidence's own announcements report a perfect score on a public USMLE sample exam. Independent researchers found high-quality answers in several specialties, no fabricated references among almost 5,000 checked, and also incomplete answers, lower scores than frontier chatbots on one set of benchmarks, and weaker results on drug information questions. Below: what OpenEvidence is built on, what it claims, what the studies found, how to check an answer, and how AskMed, an AI medical reference for doctors, nurses, pharmacists, medical students and researchers, answers.
- OpenEvidence's own claim100% on the public USMLE sample exam, text-only questions, run by OpenEvidence (2025)
- Independent benchmark89.6% on 500 MedQA questions, below three frontier chatbots (Nature Medicine, 2026)
- ReferencesNo fabricated reference among 4,979 citations checked (npj Health Systems, 2026)
- Specialty testsMostly high-quality answers, with gaps in completeness, drug information and pediatric evidence (2026)
What is OpenEvidence built on?
OpenEvidence is an AI medical search engine from OpenEvidence Inc. that writes cited answers to clinical questions for verified healthcare professionals. Its US App Store listing says answers are "sourced, cited, and grounded in peer-reviewed medical literature", with every claim cited, drawing on more than 35 million peer-reviewed papers, more than 300 medical journals, and FDA and CDC sources. The listing names NEJM, JAMA and its 11 specialty journals, NCCN, Nature, the Cochrane Library and Wiley as official partners, and adds licensed full text from several of them plus content agreements with societies such as the AAFP, ACC, ADA, AAP and ACOG. Since July 2026, a feature called EvidenceGrade rates the strength of the evidence behind each answer, building on the GRADE framework. OpenEvidence's Terms of Use say the service is for educational and informational purposes only and does not substitute for clinical judgment.
The base language models behind OpenEvidence are not public, as the authors of a 2026 Nature Medicine study note. In September 2026 it released its own model family: Osler, the default for fast answers; Sackett, a deeper search model; and Snow, which writes longer reports.
What does OpenEvidence say about its own accuracy?
OpenEvidence reports its accuracy mainly through exam benchmarks that the company ran itself. On July 14, 2023, OpenEvidence announced that its AI was the first to score above 90% on the USMLE, measured on the official 2022 sample exam. On August 15, 2025, it announced a perfect 100% on the USMLE. Its methods section says the test used the sample-exam questions published by Kung and colleagues in 2023, left out every question that contains an image, fixed one recording error in that dataset, and scored one Step 3 question by OpenEvidence's answer instead of the official key, after seven psychiatrists reviewed it and agreed with OpenEvidence. On September 3, 2026, OpenEvidence said a new research model, Darwin, scored 100% on MedQA, 72.8% on MedXpertQA, 82.7% on HealthBench Professional and 87.2% on NOHARM. Darwin is in research preview, available by application only, so it is not the model clinicians use day to day.
All three results come from OpenEvidence's own announcements, not from independent tests, and a licensing-exam question gives every relevant fact in a clean vignette with one best answer, which real clinical questions rarely do.
What did independent studies find?
The largest independent test of OpenEvidence so far found it accurate, but behind three general chatbots. Vishwanath and colleagues at NYU Langone (Nature Medicine, 2026) queried OpenEvidence through its website in September 2025 and February 2026 and compared it with UpToDate Expert AI, GPT-5.2, Gemini 3.1 Pro and Claude Opus 4.6. On 500 MedQA exam questions, OpenEvidence scored 89.6%, against 97.4% for Gemini, 94.2% for GPT-5.2, 90.2% for Claude and 88.4% for UpToDate Expert AI. On 100 real questions that physicians had asked an AI during care, 12 blinded clinicians gave OpenEvidence a mean of 3.24 on a 4-point scale, against 3.62 for Gemini. OpenEvidence scored lowest on clarity, and the authors wrote that its weakness was communication, not knowledge. Harmful content and hallucinations did not differ significantly between the six systems tested. The authors call their results a snapshot of fast-changing tools, and they did not assess citation quality.
A second study reached the opposite ranking. Feng and colleagues (arXiv preprint, June 2026, not peer reviewed) had 149 US physicians compare blinded answers to 620 real questions asked on OpenEvidence, and physicians preferred OpenEvidence over Claude Opus 4.8, Gemini 3.1 Pro and GPT-5.5 on accuracy, clinical utility, source quality, verifiability and completeness. The paper states that none of its authors is affiliated with OpenEvidence, but that OpenEvidence co-developed the data collection plan, carried out the data collection and paid the physicians who rated the answers.
Does OpenEvidence make up references?
Studies so far have found that OpenEvidence's references are real, which sets it apart from chatbots that cite from memory. Lotan and colleagues (npj Health Systems, 2026) sent OpenEvidence 150 standard prompts across five specialties and checked all 4,979 citations it returned: none was confirmed as fabricated, three abstracts had attribution errors, and most references were recent and from high-impact journals. That study checked whether each reference exists, not whether it supports the sentence it is attached to. Hunter and colleagues (Hospital Pediatrics, 2026) did check support: across 17 guideline-based questions in six pediatric subspecialties, 88.2% of answers were rated as highly substantiated by their citations, and OpenEvidence cited the main guideline for 15 of the 17 questions. But 35.3% of answers cited studies that were not specific to children, so a real paper can still describe the wrong patients. Real references are the starting point for checking an answer, not proof that it is right.
The same study rated answer quality at a median of 27 out of 30, and 6 of the 17 answers carried some potential for harm, none above 3 on the study's 7-point harm scale.
Where do OpenEvidence answers fall short?
OpenEvidence answers fall short most often on completeness, drug information and nuanced decisions, according to specialty studies published in 2026. Meyer and colleagues (American Surgeon, 2026) tested 138 rectal cancer scenarios built from the NCCN guidelines: OpenEvidence's answers were rated correct in 92.8% and accurate in 81.2%, against 98.6% and 84.1% for ChatGPT-5, and the authors found both tools limited in completeness, nuance and preference-sensitive decisions. In a pharmacy study by Issa and colleagues (American Journal of Health-System Pharmacy, 2026), OpenEvidence answered 40% of 20 drug information questions accurately, against 70% for ChatGPT 5.1, though its answers had the highest overall quality score of the AI tools. Stephenson and colleagues at Mayo Clinic (Mayo Clinic Proceedings: Digital Health, 2026) found OpenEvidence's 125 fatty liver e-consults rated higher than specialists' own, yet 12 contained minor errors and one a potentially major error.
In a triage test by Jia and colleagues (International Journal of Medical Informatics, 2026), OpenEvidence was correct in 71.3% of clear cases that it answered, under-triaged 12.5% of emergencies and over-triaged 68.0% of cases that could be managed at home. Its errors leaned toward caution.
What limits do all AI answer engines share?
AI answer engines such as OpenEvidence share limits that come from how they work, whatever their sources. An answer engine first retrieves documents, then a language model writes a summary of them, and both steps can go wrong. The Nature Medicine authors point to evidence that retrieval can lower performance when irrelevant material is retrieved or poorly integrated by the model. Even with real sources, a sentence can say more than its source does: Wu and colleagues (Nature Communications, 2025) tested seven popular models on 800 medical questions and found that 50% to 90% of responses were not fully supported, and sometimes contradicted, by the sources they cited; even for GPT-4o with web search, about 30% of individual statements were unsupported. That study did not test OpenEvidence. Answer engines also change often: OpenEvidence released a new default model, Osler, in September 2026, after the Nature Medicine and pharmacy studies had collected their answers, so published figures may not describe the current version.
Finally, a fluent, well-cited answer reads with the same confidence whether it is complete or not. Checking the source behind the claim you act on is the only reliable safeguard.
How can you check an OpenEvidence answer?
You can check an OpenEvidence answer, or one from any AI tool, in a few minutes with the same five steps. First, open the citation behind the specific sentence you plan to act on, and confirm that the source says what the sentence says; a real reference can still be attached to a claim it doesn't support. Second, check that the source studied the same patients: the pediatric study above found that about a third of answers cited studies that were not specific to children. Third, check the date and the type of evidence, because a current guideline outweighs an older review and one small study is weaker than a meta-analysis; OpenEvidence's EvidenceGrade rating helps here, but read the source too. Fourth, check whose guidance it is, since NICE, US and European guidelines can differ. Fifth, take any drug dose from the current prescribing information or your local formulary, never from an AI summary.
- Open the citation. Read the sentence in the source itself.
- Match the patients. Adults, children, pregnancy and kidney function change the answer.
- Check the date and type. Newest guideline first, then systematic reviews, then single studies.
- Check whose guidance. Follow the guideline that applies where you practise.
- Take doses from the label. Use the current prescribing information.
For what studies found about general chatbots, see Is ChatGPT accurate for medical questions?
How does AskMed answer differently?
AskMed is built so that every claim can be checked, and so that anyone in medicine can use it. Before any answer is written, AskMed searches a fixed list of medical sources: journals and databases such as PubMed, the Cochrane Library, NEJM, The Lancet, JAMA and The BMJ; guidelines and agencies such as NICE, WHO, CDC, ESC and KDIGO; FDA and DailyMed drug labels, plus registries such as the EMA for local brand names; and references such as StatPearls and BMJ Best Practice. AskMed does not search the open web, forums or social media. Each claim is followed by a link to its source, from 8 sources by default, and source filters let you choose source types, study types and 3 to 12 sources. If no usable source turns up, AskMed shows a fixed reply saying so instead of answering from memory. AskMed never gives drug doses, works worldwide with no NPI or license number, and runs on iPhone, iPad, Android and the web.
AskMed's answers are written by AI too, with no human review before you see them, so AskMed can also misread a source; the point is that each sentence carries a link you can open. OpenEvidence's Terms of Use say it is not available in the UK, the EU or the EEA, and its registration asks for an NPI or other licensing number. For the side-by-side detail, see AskMed vs OpenEvidence and OpenEvidence alternatives; for every source and limit, see How AskMed answers.
Frequently asked questions
Is OpenEvidence accurate?
Mostly, but not always. Independent studies found high-quality answers in several specialties and no fabricated references among 4,979 citations checked, but also incomplete answers, 40% accuracy on a small set of drug information questions, and lower benchmark scores than three frontier chatbots in a 2026 Nature Medicine study. Use OpenEvidence as a starting point and open the source behind any claim you act on.
Did OpenEvidence really score 100% on the USMLE?
OpenEvidence reports that it did, in an August 2025 announcement. The test used the public USMLE sample-exam questions from a 2023 study, without the questions that contain images, and counted one Step 3 question by OpenEvidence's answer rather than the official key. The result was run by OpenEvidence and hasn't been published in a peer-reviewed journal.
Does OpenEvidence hallucinate?
Less than general chatbots do with references: a 2026 study found no fabricated reference among 4,979 OpenEvidence citations. In the 2026 Nature Medicine study, OpenEvidence's rate of hallucinated content did not differ significantly from that of the other five systems tested. A real reference can still be attached to a claim it doesn't fully support, so open it.
Is OpenEvidence more accurate than ChatGPT?
The studies disagree. A peer-reviewed 2026 Nature Medicine study ranked GPT-5.2, Gemini and Claude above OpenEvidence, while a 2026 preprint whose data collection OpenEvidence ran and paid for found physicians preferred OpenEvidence's answers. Single-specialty studies also split. For more, see OpenEvidence vs ChatGPT for Clinicians vs AskMed.
Can AskMed be wrong?
Yes. AskMed's answers are written by AI, so check the linked source behind any claim you act on, and tap the thumbs down under an answer to report an error.
Does AskMed give drug doses?
No. AskMed never gives doses, dose calculations or titration schedules, even for exam questions. It still explains a drug's uses, how dosing is chosen and adjusted, monitoring and side effects. Exact doses come from the current prescribing information.