2f:I[703058,["/_next/static/immutable/chunks/3bhq_5eh1d760.js","/_next/static/immutable/chunks/1855-ogyi8eh4.js","/_next/static/immutable/chunks/2pfsjpn6k6zlg.js","/_next/static/immutable/chunks/26c7v3_3uwnwu.js","/_next/static/immutable/chunks/25a3ozcrvi1ff.js","/_next/static/immutable/chunks/2r4wx50nvjh_4.js","/_next/static/immutable/chunks/23fsw06hsi5b1.js","/_next/static/immutable/chunks/06_yt270h41rg.js","/_next/static/immutable/chunks/3ezsiz-7shkna.js","/_next/static/immutable/chunks/20x2y96ldu5wx.js","/_next/static/immutable/chunks/2lxntcm3pd1_h.js","/_next/static/immutable/chunks/00dgco8irq3jw.js","/_next/static/immutable/chunks/00nwe-kq0rboc.js","/_next/static/immutable/chunks/2051ffx3xzio9.js"],"PaperListPageClient"] 30:T81d,Artificial intelligence (AI)-driven chatbots have been rapidly adopted across research, education, business, marketing and medicine. Most interactions, however, come from non-experts using chatbots like search engines, including for everyday health and medical queries. We conducted an original study to audit chatbot responses in health and medical fields prone to misinformation. Five popular chatbots were assessed: Gemini (Google), DeepSeek (High-Flyer), Meta AI (Meta), ChatGPT (OpenAI) and Grok (xAI). In February 2025, each chatbot was prompted with 10 questions from five categories: cancer, vaccines, stem cells, nutrition and athletic performance. We deployed an adversarial-like framework, using open- and closed-ended prompts designed to strain models toward misinformation or contraindicated advice. Two experts from each category rated responses as 'non-problematic', 'somewhat problematic' or Nearly half (49.6%) of responses were problematic: 30% somewhat problematic and 19.6% highly problematic. Response quality did not differ significantly among chatbots (p=0.566) but Grok generated significantly more highly problematic responses than would be expected under a random distribution (z-score +2.07, p=0.038). Performance was strongest in vaccines (mean z-score -2.57) and cancer (-2.12), and weakest in stem cells (+1.25), athletic performance (+3.74) and nutrition (+4.35). Chatbot outputs were consistently expressed with confidence and certainty; from 250 total questions, there were only two refusals to answer (0.8%), both from Meta AI. Reference quality was poor, with a median completeness score of 40% (Q1-Q3: 20-67%). Chatbot hallucinations and fabricated citations precluded any chatbot from producing a fully accurate reference list. All readability scores were graded as 'Difficult' (30-50), equivalent to college sophomore-senior level. The audited chatbots performed poorly when answering questions in misinformation-prone health and medical fields. Continued deployment without public education and oversight risks amplifying misinformation.31:Tc4b, Objectives Artificial intelligence (AI)-driven chatbots have been rapidly adopted across research, education, business, marketing and medicine. Most interactions, however, come from non-experts using chatbots like search engines, including for everyday health and medical queries. Design We conducted an original study to audit chatbot responses in health and medical fields prone to misinformation. Methods Five popular chatbots were assessed: Gemini (Google), DeepSeek (High-Flyer), Meta AI (Meta), ChatGPT (OpenAI) and Grok (xAI). In February 2025, each chatbot was prompted with 10 questions from five categories: cancer, vaccines, stem cells, nutrition and athletic performance. We deployed an adversarial-like framework, using open- and closed-ended prompts designed to strain models toward misinformation or contraindicated advice. Two experts from each category rated responses as ‘non-problematic’, ‘somewhat problematic’ or ‘ highly problematic’ using a coding matrix based on objective, predefined criteria. Citations were scored for accuracy and completeness, and each response was given a Flesch Reading Ease score. Results Nearly half (49.6%) of responses were problematic: 30% somewhat problematic and 19.6% highly problematic. Response quality did not differ significantly among chatbots (p=0.566) but Grok generated significantly more highly problematic responses than would be expected under a random distribution (z-score +2.07, p=0.038). Performance was strongest in vaccines (mean z-score –2.57) and cancer (–2.12), and weakest in stem cells (+1.25), athletic performance (+3.74) and nutrition (+4.35). Chatbot outputs were consistently expressed with confidence and certainty; from 250 total questions, there were only two refusals to answer (0.8%), both from Meta AI. Reference quality was poor, with a median completeness score of 40% (Q1–Q3: 20–67%). Chatbot hallucinations and fabricated citations precluded any chatbot from producing a fully accurate reference list. All readability scores were graded as ‘Difficult’ (30–50), equivalent to college sophomore–senior level. Conclusions The audited chatbots performed poorly when answering questions in misinformation-prone health and medical fields. Continued deployment without public education and oversight risks amplifying misinformation. 25:[[["$","script","0",{"type":"application/ld+json","dangerouslySetInnerHTML":{"__html":"{\"@context\":\"https://schema.org\",\"@type\":\"CollectionPage\",\"name\":\"Audit / Content Analysis: study design Research & Clinical Takeaways\",\"description\":\"Explore peer-reviewed papers on Audit / Content Analysis (study design). Clinician-written summaries and actionable practice insights. Updated daily.\",\"url\":\"https://www.reachrx.ai/hub/tag/study_design/Audit%20%2F%20Content%20Analysis\",\"breadcrumb\":{\"@type\":\"BreadcrumbList\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https://www.reachrx.ai\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Research Hub\",\"item\":\"https://www.reachrx.ai/hub\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"Study Designs\",\"item\":\"https://www.reachrx.ai/hub/tag/study_design\"},{\"@type\":\"ListItem\",\"position\":4,\"name\":\"Audit / Content Analysis\",\"item\":\"https://www.reachrx.ai/hub/tag/study_design/Audit%20%2F%20Content%20Analysis\"}]}}"}}]],["$","$L2f",null,{"title":"Audit / Content Analysis: Research & Clinical Takeaways","description":"Browse Audit / Content Analysis research papers with clear, practical takeaways for your clinical practice.","papers":[{"doi":"10.1136/bmjopen-2025-112695","pmid":"41980854","title":"Generative artificial intelligence-driven chatbots and medical misinformation: an accuracy, referencing and readability audit","journal_title":"BMJ Open","journal_issn":"2044-6055","published_at":"2026-04-15T00:00:00.000Z","url":"https://doi.org/10.1136/bmjopen-2025-112695","source_score":828,"is_retracted":false,"is_open_access":true,"raw_metadata":{"ooir":{"day":"2026-04-19","doi":"10.1136/bmjopen-2025-112695","days":"30","issn":"2044-6055","rank":"6","field":"Clinical Medicine","score":"828","category":"Medicine, General & Internal","categories":["Medicine, General & Internal"],"_categories":["Medicine, General & Internal"]},"pubmed":{"doi":"10.1136/bmjopen-2025-112695","title":"Generative artificial intelligence-driven chatbots and medical misinformation: an accuracy, referencing and readability audit.","journal":"BMJ open","methods":"","results":"Nearly half (49.6%) of responses were problematic: 30% somewhat problematic and 19.6% highly problematic. Response quality did not differ significantly among chatbots (p=0.566) but Grok generated significantly more highly problematic responses than would be expected under a random distribution (z-score +2.07, p=0.038). Performance was strongest in vaccines (mean z-score -2.57) and cancer (-2.12), and weakest in stem cells (+1.25), athletic performance (+3.74) and nutrition (+4.35). Chatbot outputs were consistently expressed with confidence and certainty; from 250 total questions, there were only two refusals to answer (0.8%), both from Meta AI. Reference quality was poor, with a median completeness score of 40% (Q1-Q3: 20-67%). Chatbot hallucinations and fabricated citations precluded any chatbot from producing a fully accurate reference list. All readability scores were graded as 'Difficult' (30-50), equivalent to college sophomore-senior level.","abstract":"$30","keywords":["Artificial Intelligence","Health","Medicine"],"pubmed_id":"41980854","conclusions":"","publication_date":"2026-04-15T00:00:00"},"crossref":{"DOI":"10.1136/bmjopen-2025-112695","URL":"https://doi.org/10.1136/bmjopen-2025-112695","ISSN":["2044-6055","2044-6055"],"link":[{"URL":"https://syndication.highwire.org/content/doi/10.1136/bmjopen-2025-112695","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"page":"e112695","type":"journal-article","issue":"4","score":1,"title":["Generative artificial intelligence-driven chatbots and medical misinformation: an accuracy, referencing and readability audit"],"author":[{"given":"Nicholas B","family":"Tiller","sequence":"first","affiliation":[{"name":"The Lundquist Institute for Biomedical Innovation at Harbor-UCLA Medical Center, Torrance, California, USA"}]},{"given":"Alessandro R","family":"Marcon","sequence":"additional","affiliation":[{"name":"Health Law Institute, University of Alberta, Edmonton, Alberta, Canada"}]},{"given":"Marco","family":"Zenone","sequence":"additional","affiliation":[{"name":"Faculty of Health Sciences, University of Ottawa, Ottawa, Ontario, Canada"}]},{"given":"Kristin E","family":"Kidd","sequence":"additional","affiliation":[{"name":"Department of Social Sciences and Health Policy, Division of Public Health Sciences, Wake Forest School of Medicine, Winston-Salem, North Carolina, USA"}]},{"given":"Asker E","family":"Jeukendrup","sequence":"additional","affiliation":[{"name":"School of Sport, Exercise and Health Sciences, Loughborough University, Loughborough, UK"}]},{"given":"Zubin","family":"Master","sequence":"additional","affiliation":[{"name":"Department of Social Sciences and Health Policy, Division of Public Health Sciences, Wake Forest School of Medicine, Winston-Salem, North Carolina, USA"},{"name":"Wake Forest Institute of Regenerative Medicine, Wake Forest University School of Medicine, Winston-Salem, North Carolina, USA"},{"name":"Maya Angelou Center for Healthy Communities, Wake Forest University School of Medicine, Winston-Salem, North Carolina, USA"},{"name":"Center for Bioethics, Health & Society, Wake Forest University, Winston-Salem, North Carolina, USA"}]},{"given":"Timothy","family":"Caulfield","sequence":"additional","affiliation":[{"name":"Health Law Institute, University of Alberta, Edmonton, Alberta, Canada"}]}],"issued":{"date-parts":[[2026,4]]},"member":"239","prefix":"10.1136","source":"Crossref","volume":"16","created":{"date-time":"2026-04-15T00:05:12Z","timestamp":1776211512000,"date-parts":[[2026,4,15]]},"indexed":{"date-time":"2026-05-13T02:01:43Z","timestamp":1778637703549,"date-parts":[[2026,5,13]]},"license":[{"URL":"https://creativecommons.org/licenses/by-nc/4.0/","start":{"date-time":"2026-04-14T00:00:00Z","timestamp":1776124800000,"date-parts":[[2026,4,14]]},"delay-in-days":13,"content-version":"unspecified"}],"subject":[],"abstract":"$31","accepted":{"date-parts":[[2026,2,11]]},"language":"en","relation":{},"resource":{"primary":{"URL":"https://bmjopen.bmj.com/lookup/doi/10.1136/bmjopen-2025-112695"}},"subtitle":[],"deposited":{"date-time":"2026-04-15T00:05:20Z","timestamp":1776211520000,"date-parts":[[2026,4,15]]},"issn-type":[{"type":"print","value":"2044-6055"},{"type":"electronic","value":"2044-6055"}],"published":{"date-parts":[[2026,4]]},"publisher":"BMJ","reference":[{"DOI":null,"key":"2026041417050707000_16.4.e112695.1","year":null,"issue":null,"author":null,"volume":null,"first-page":null,"unstructured":"Statista . GenAI usage frequency business 2024. Available: https://www.statista.com/statistics/1482102/rate-of-generative-ai-utilization-globally/ [Accessed 23 Jul 2025].","article-title":null,"journal-title":null,"doi-asserted-by":null},{"DOI":"10.3389/feduc.2024.1374889","key":"2026041417050707000_16.4.e112695.2","year":null,"issue":null,"author":null,"volume":null,"first-page":null,"unstructured":"Paustian T , Slinger B . Students are using large language models and AI detectors can often detect their use. Front Educ 2024;9. doi:10.3389/feduc.2024.1374889","article-title":null,"journal-title":null,"doi-asserted-by":"crossref"},{"DOI":"10.1038/d41586-023-02477-w","key":"2026041417050707000_16.4.e112695.3","year":null,"issue":null,"author":null,"volume":null,"first-page":null,"unstructured":"Conroy G . Scientific sleuths spot dishonest ChatGPT use in papers. Nature New Biol 2023. doi:10.1038/d41586-023-02477-w","article-title":null,"journal-title":null,"doi-asserted-by":"crossref"},{"DOI":"10.5812/semj-161279","key":"2026041417050707000_16.4.e112695.4","year":null,"issue":null,"author":null,"volume":null,"first-page":null,"unstructured":"MohsseniPour M , Parsakian S , Mehraeen E . Potential Applications of ChatGPT in the Healthcare Industry of Low- and Middle-Income Countries. Shiraz E-Med J 2025;26. doi:10.5812/semj-161279","article-title":null,"journal-title":null,"doi-asserted-by":"crossref"},{"DOI":"10.3390/info16070549","key":"2026041417050707000_16.4.e112695.5","year":"2025","issue":null,"author":"Chow","volume":"16","first-page":"549","unstructured":null,"article-title":"Large Language Models in Medical Chatbots: Opportunities, Challenges, and the Need to Address AI Risks","journal-title":"Information","doi-asserted-by":"crossref"},{"DOI":"10.1038/s41562-024-02046-9","key":"2026041417050707000_16.4.e112695.6","year":"2025","issue":null,"author":"Luo","volume":"9","first-page":"305","unstructured":null,"article-title":"Large language models surpass human experts in predicting neuroscience results","journal-title":"Nat Hum Behav","doi-asserted-by":"crossref"},{"DOI":"10.1001/jamanetworkopen.2023.36483","key":"2026041417050707000_16.4.e112695.7","year":"2023","issue":null,"author":"Goodman","volume":"6","first-page":null,"unstructured":null,"article-title":"Accuracy and Reliability of Chatbot Responses to Physician Questions","journal-title":"JAMA Netw Open","doi-asserted-by":"crossref"},{"DOI":"10.1186/s43019-024-00218-5","key":"2026041417050707000_16.4.e112695.8","year":"2024","issue":null,"author":"Zhang","volume":"36","first-page":null,"unstructured":null,"article-title":"Evaluating the accuracy and relevance of ChatGPT responses to frequently asked questions regarding total knee replacement","journal-title":"Knee Surg Relat Res","doi-asserted-by":"crossref"},{"DOI":"10.21203/rs.3.rs-2566942/v1","key":"2026041417050707000_16.4.e112695.9","year":null,"issue":null,"author":null,"volume":null,"first-page":null,"unstructured":"Johnson D , Goodman R , Patrinely J , et al . Assessing the Accuracy and Reliability of AI-Generated Medical Responses: An Evaluation of the Chat-GPT Model. Res Sq 2023.:rs.3.rs-2566942. doi:10.21203/rs.3.rs-2566942/v1","article-title":null,"journal-title":null,"doi-asserted-by":"crossref"},{"DOI":"10.1038/s41433-023-02906-0","key":"2026041417050707000_16.4.e112695.10","year":"2024","issue":null,"author":"Cappellani","volume":"38","first-page":"1368","unstructured":null,"article-title":"Reliability and accuracy of artificial intelligence ChatGPT in providing information on ophthalmic diseases and management to patients","journal-title":"Eye (Lond)","doi-asserted-by":"crossref"},{"DOI":null,"key":"2026041417050707000_16.4.e112695.11","year":"2024","issue":null,"author":"van Nuland","volume":"15","first-page":null,"unstructured":null,"article-title":"Assessing accuracy of ChatGPT in response to questions from day to day pharmaceutical care in hospitals","journal-title":"Explor Res Clin Soc Pharm","doi-asserted-by":null},{"DOI":"10.2196/48023","key":"2026041417050707000_16.4.e112695.12","year":"2023","issue":null,"author":"Yanagita","volume":"7","first-page":null,"unstructured":null,"article-title":"Accuracy of ChatGPT on Medical Questions in the National Medical Licensing Examination in Japan: Evaluation Study","journal-title":"JMIR Form Res","doi-asserted-by":"crossref"},{"DOI":"10.1177/08850666241267871","key":"2026041417050707000_16.4.e112695.13","year":"2025","issue":null,"author":"Balta","volume":"40","first-page":"184","unstructured":null,"article-title":"Evaluating the Appropriateness, Consistency, and Readability of ChatGPT in Critical Care Recommendations","journal-title":"J Intensive Care Med","doi-asserted-by":"crossref"},{"DOI":"10.1186/s12903-025-06726-4","key":"2026041417050707000_16.4.e112695.14","year":"2025","issue":null,"author":"Bashah","volume":"25","first-page":null,"unstructured":null,"article-title":"Evaluation of deepseek, gemini, ChatGPT-4o, and perplexity in responding to salivary gland cancer","journal-title":"BMC Oral Health","doi-asserted-by":"crossref"},{"DOI":null,"key":"2026041417050707000_16.4.e112695.15","year":null,"issue":null,"author":null,"volume":null,"first-page":null,"unstructured":"What are AI hallucinations? Google Cloud, Available: https://cloud.google.com/discover/what-are-ai-hallucinations [Accessed 20 Aug 2025].","article-title":null,"journal-title":null,"doi-asserted-by":null},{"DOI":null,"key":"2026041417050707000_16.4.e112695.16","year":null,"issue":null,"author":null,"volume":null,"first-page":null,"unstructured":"Sharma M , Tong M , Korbak T , et al . Towards understanding sycophancy in language models. arXiv [Preprint] 2025. doi:10.48550/arXiv.2310.13548","article-title":null,"journal-title":null,"doi-asserted-by":null},{"DOI":"10.1007/s13304-025-02149-6","key":"2026041417050707000_16.4.e112695.17","year":"2026","issue":null,"author":"Mehraeen","volume":"78","first-page":"477","unstructured":null,"article-title":"Transforming plastic surgery: an innovative role of Chat GPT in plastic surgery practices","journal-title":"Updates Surg","doi-asserted-by":"crossref"},{"DOI":"10.3389/frai.2023.1166014","key":"2026041417050707000_16.4.e112695.18","year":"2023","issue":null,"author":"Chow","volume":"6","first-page":null,"unstructured":null,"article-title":"Impact of ChatGPT on medical chatbots as a disruptive technology","journal-title":"Front Artif Intell","doi-asserted-by":"crossref"},{"DOI":"10.3390/publications13010012","key":"2026041417050707000_16.4.e112695.19","year":"2025","issue":null,"author":"Spennemann","volume":"13","first-page":"12","unstructured":null,"article-title":"The Origins and Veracity of References ‘Cited’ by Generative Artificial Intelligence Applications: Implications for the Quality of Responses","journal-title":"Publications","doi-asserted-by":"crossref"},{"DOI":null,"key":"2026041417050707000_16.4.e112695.20","year":null,"issue":null,"author":null,"volume":null,"first-page":null,"unstructured":"Jaźwińska K , Chandrasekar A . AI Search Has A Citation Problem. Columbia Journal Rev 2025. Available: https://www.cjr.org/tow_center/we-compared-eight-ai-search-engines-theyre-all-bad-at-citing-news.php","article-title":null,"journal-title":null,"doi-asserted-by":null},{"DOI":"10.1148/radiol.230163","key":"2026041417050707000_16.4.e112695.21","year":"2023","issue":null,"author":"Shen","volume":"307","first-page":null,"unstructured":null,"article-title":"ChatGPT and Other Large Language Models Are Double-edged Swords","journal-title":"Radiology","doi-asserted-by":"crossref"},{"DOI":null,"key":"2026041417050707000_16.4.e112695.22","year":null,"issue":null,"author":null,"volume":null,"first-page":null,"unstructured":"Milmo D . Two US lawyers fined for submitting fake court citations from ChatGPT. The Guardian. 2023. Available: https://www.theguardian.com/technology/2023/jun/23/two-us-lawyers-fined-submitting-fake-court-citations-chatgpt","article-title":null,"journal-title":null,"doi-asserted-by":null},{"DOI":"10.1016/j.datak.2023.102182","key":"2026041417050707000_16.4.e112695.23","year":"2023","issue":null,"author":"Opdahl","volume":"146","first-page":"102182","unstructured":null,"article-title":"Trustworthy journalism through AI","journal-title":"Data Knowl Eng","doi-asserted-by":"crossref"},{"DOI":null,"key":"2026041417050707000_16.4.e112695.24","year":null,"issue":null,"author":null,"volume":null,"first-page":null,"unstructured":"Elon University News Bureau, staff . Survey: 52% of U.S. adults now use AI large language models like ChatGPT. Today Elon; 2025. Available: https://www.elon.edu/u/news/2025/03/12/survey-52-of-u-s-adults-now-use-ai-large-language-models-like-chatgpt/ [Accessed 4 Sep 2025].","article-title":null,"journal-title":null,"doi-asserted-by":null},{"DOI":"10.3389/fpubh.2023.1166120","key":"2026041417050707000_16.4.e112695.25","year":"2023","issue":null,"author":"De Angelis","volume":"11","first-page":null,"unstructured":null,"article-title":"ChatGPT and the rise of large language models: the new AI-driven infodemic threat in public health","journal-title":"Front Public Health","doi-asserted-by":"crossref"},{"DOI":null,"key":"2026041417050707000_16.4.e112695.26","year":null,"issue":null,"author":null,"volume":null,"first-page":null,"unstructured":"Fault lines: the expert panel on the socioeconomic impacts of science and health misinformation. Council of Canadian Academies, Available: https://cca-reports.ca/reports/the-socioeconomic-impacts-of-health-and-science-misinformation/ [Accessed 9 Sep 2025].","article-title":null,"journal-title":null,"doi-asserted-by":null},{"DOI":"10.1016/S0140-6736(25)00094-7","key":"2026041417050707000_16.4.e112695.27","year":null,"issue":null,"author":null,"volume":null,"first-page":null,"unstructured":null,"article-title":null,"journal-title":null,"doi-asserted-by":"publisher"},{"DOI":"10.1001/jamapediatrics.2023.5282","key":"2026041417050707000_16.4.e112695.28","year":"2024","issue":null,"author":"Wang","volume":"178","first-page":"109","unstructured":null,"article-title":"Health Misinformation on Social Media and Adolescent Health","journal-title":"JAMA Pediatr","doi-asserted-by":"crossref"},{"DOI":"10.1126/science.aap9559","key":"2026041417050707000_16.4.e112695.29","year":null,"issue":null,"author":null,"volume":null,"first-page":null,"unstructured":null,"article-title":null,"journal-title":null,"doi-asserted-by":"publisher"},{"DOI":"10.1038/s41591-024-03445-1","key":"2026041417050707000_16.4.e112695.30","year":null,"issue":null,"author":null,"volume":null,"first-page":null,"unstructured":null,"article-title":null,"journal-title":null,"doi-asserted-by":"publisher"},{"DOI":"10.1007/s43681-023-00289-2","key":"2026041417050707000_16.4.e112695.31","year":"2024","issue":null,"author":"Mökander","volume":"4","first-page":"1085","unstructured":null,"article-title":"Auditing large language models: a three-layered approach","journal-title":"AI Ethics","doi-asserted-by":"crossref"},{"DOI":"10.1038/s41746-025-01786-w","key":"2026041417050707000_16.4.e112695.32","year":"2025","issue":null,"author":"Templin","volume":"8","first-page":null,"unstructured":null,"article-title":"Framework for bias evaluation in large language models in healthcare settings","journal-title":"NPJ Digit Med","doi-asserted-by":"crossref"},{"DOI":"10.1001/jamanetworkopen.2024.57879","key":"2026041417050707000_16.4.e112695.33","year":"2025","issue":null,"author":"Huo","volume":"8","first-page":null,"unstructured":null,"article-title":"Large Language Models for Chatbot Health Advice Studies: A Systematic Review","journal-title":"JAMA Netw Open","doi-asserted-by":"crossref"},{"DOI":"10.1136/bmjmed-2025-001632","key":"2026041417050707000_16.4.e112695.34","year":"2025","issue":null,"author":null,"volume":"4","first-page":null,"unstructured":null,"article-title":"Reporting guideline for chatbot health advice studies: the Chatbot Assessment Reporting Tool (CHART) statement","journal-title":"BMJ Med","doi-asserted-by":"crossref"},{"DOI":null,"key":"2026041417050707000_16.4.e112695.35","year":null,"issue":null,"author":null,"volume":null,"first-page":null,"unstructured":"40+ Chatbot statistics (2025). Exploding Topics, 2024. Available: https://explodingtopics.com/blog/chatbot-statistics [Accessed 3 Sep 2025].","article-title":null,"journal-title":null,"doi-asserted-by":null},{"DOI":null,"key":"2026041417050707000_16.4.e112695.36","year":null,"issue":null,"author":null,"volume":null,"first-page":null,"unstructured":"Bilski D . The state of consumer AI. Menlo Ventures, 2025. Available: https://menlovc.com/perspective/2025-the-state-of-consumer-ai/ [Accessed 3 Sep 2025].","article-title":null,"journal-title":null,"doi-asserted-by":null},{"DOI":null,"key":"2026041417050707000_16.4.e112695.37","year":null,"issue":null,"author":null,"volume":null,"first-page":null,"unstructured":"Zhang L , Wang H , Cheng L , et al . Adversarial testing in LLMs: insights into decision-making vulnerabilities. arXiv [Preprint] 2025. doi:10.48550/arXiv.2505.13195","article-title":null,"journal-title":null,"doi-asserted-by":null},{"DOI":null,"key":"2026041417050707000_16.4.e112695.38","year":null,"issue":null,"author":null,"volume":null,"first-page":null,"unstructured":"Balazadeh V , Cooper M , Pellow D , et al . Red teaming large language models for healthcare. arXiv [Preprint] 2025. doi:10.48550/arXiv.2505.00467","article-title":null,"journal-title":null,"doi-asserted-by":null},{"DOI":"10.1111/j.1365-2648.2007.04569.x","key":"2026041417050707000_16.4.e112695.39","year":null,"issue":null,"author":null,"volume":null,"first-page":null,"unstructured":null,"article-title":null,"journal-title":null,"doi-asserted-by":"publisher"},{"DOI":"10.1177/1075547012458290","key":"2026041417050707000_16.4.e112695.40","year":null,"issue":null,"author":null,"volume":null,"first-page":null,"unstructured":null,"article-title":null,"journal-title":null,"doi-asserted-by":"publisher"},{"DOI":null,"key":"2026041417050707000_16.4.e112695.41","year":null,"issue":null,"author":null,"volume":null,"first-page":null,"unstructured":"Is bothsidesism killing us? (and why scientific consensus matters). Healthy Debate, Available: https://healthydebate.ca/2023/08/topic/bothsidesism-scientific-consensus-matters/ [Accessed 10 Sep 2025].","article-title":null,"journal-title":null,"doi-asserted-by":null},{"DOI":"10.1016/j.socscimed.2025.117795","key":"2026041417050707000_16.4.e112695.42","year":"2025","issue":null,"author":"Kawam","volume":"382","first-page":"117795","unstructured":null,"article-title":"Factors that influence unproven stem cell intervention seeking behavior: A qualitative analysis of U.S. patients considering or having undertaken unproven stem cell interventions","journal-title":"Soc Sci Med","doi-asserted-by":"crossref"},{"DOI":"10.1037/h0057532","key":"2026041417050707000_16.4.e112695.43","year":null,"issue":null,"author":null,"volume":null,"first-page":null,"unstructured":null,"article-title":null,"journal-title":null,"doi-asserted-by":"publisher"},{"DOI":"10.1080/10862967509547131","key":"2026041417050707000_16.4.e112695.44","year":null,"issue":null,"author":null,"volume":null,"first-page":null,"unstructured":null,"article-title":null,"journal-title":null,"doi-asserted-by":"publisher"},{"DOI":"10.1016/j.jhsa.2024.01.006","key":"2026041417050707000_16.4.e112695.45","year":null,"issue":null,"author":null,"volume":null,"first-page":null,"unstructured":null,"article-title":null,"journal-title":null,"doi-asserted-by":"publisher"},{"DOI":"10.1001/jamaoncol.2024.0836","key":"2026041417050707000_16.4.e112695.46","year":"2024","issue":null,"author":"Chen","volume":"10","first-page":"956","unstructured":null,"article-title":"Physician and Artificial Intelligence Chatbot Responses to Cancer Questions From Social Media","journal-title":"JAMA Oncol","doi-asserted-by":"crossref"},{"DOI":"10.1001/jamainternmed.2023.1838","key":"2026041417050707000_16.4.e112695.47","year":"2023","issue":null,"author":"Ayers","volume":"183","first-page":"589","unstructured":null,"article-title":"Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum","journal-title":"JAMA Intern Med","doi-asserted-by":"crossref"},{"DOI":"10.3390/nu16040469","key":"2026041417050707000_16.4.e112695.48","year":"2024","issue":null,"author":"Ponzo","volume":"16","first-page":"469","unstructured":null,"article-title":"Is ChatGPT an Effective Tool for Providing Dietary Advice?","journal-title":"Nutrients","doi-asserted-by":"crossref"},{"DOI":"10.1101/2024.04.10.24305470","key":"2026041417050707000_16.4.e112695.49","year":null,"issue":null,"author":null,"volume":null,"first-page":null,"unstructured":"Anibal J , Huth H , Gunkel J , et al . Simulated Misuse of Large Language Models and Clinical Credit Systems. medRxiv 2024.:2024.04.10.24305470. doi:10.1101/2024.04.10.24305470","article-title":null,"journal-title":null,"doi-asserted-by":"crossref"},{"DOI":"10.1186/s40900-020-0182-y","key":"2026041417050707000_16.4.e112695.50","year":"2020","issue":null,"author":"Day","volume":"6","first-page":null,"unstructured":null,"article-title":"Open to the public: paywalls and the public rationale for open access medical research publishing","journal-title":"Res Involv Engagem","doi-asserted-by":"crossref"},{"DOI":"10.1177/1833358321992683","key":"2026041417050707000_16.4.e112695.51","year":null,"issue":null,"author":null,"volume":null,"first-page":null,"unstructured":null,"article-title":null,"journal-title":null,"doi-asserted-by":"publisher"},{"DOI":"10.18653/v1/2020.findings-emnlp.301","key":"2026041417050707000_16.4.e112695.52","year":null,"issue":null,"author":null,"volume":null,"first-page":null,"unstructured":"Gehman S , Gururangan S , Sap M , et al . RealToxicityPrompts: evaluating neural toxic degeneration in language models. In: Cohn T , He Y , Liu Y , eds. Findings of the Association for Computational Linguistics; Online, 2020 doi:10.18653/v1/2020.findings-emnlp.301 Available: https://aclanthology.org/2020.findings-emnlp","article-title":null,"journal-title":null,"doi-asserted-by":"crossref"},{"DOI":"10.2196/43548","key":"2026041417050707000_16.4.e112695.53","year":"2023","issue":null,"author":"Zenone","volume":"3","first-page":null,"unstructured":null,"article-title":"Advertising Alternative Cancer Treatments and Approaches on Meta Social Media Platforms: Content Analysis","journal-title":"JMIR Infodemiology","doi-asserted-by":"crossref"},{"DOI":"10.1038/s41586-024-07930-y","key":"2026041417050707000_16.4.e112695.54","year":"2024","issue":null,"author":"Zhou","volume":"634","first-page":"61","unstructured":null,"article-title":"Larger and more instructable language models become less reliable","journal-title":"Nature New Biol","doi-asserted-by":"crossref"},{"DOI":null,"key":"2026041417050707000_16.4.e112695.55","year":null,"issue":null,"author":null,"volume":null,"first-page":null,"unstructured":"Bai Y , Kadavath S , Kundu S , et al . Constitutional AI: harmlessness from AI feedback. arXiv:2212.08073v1 [cs.CL]. 2022. doi:10.48550/arXiv.2212.08073","article-title":null,"journal-title":null,"doi-asserted-by":null},{"DOI":"10.1080/13538322.2012.711071","key":"2026041417050707000_16.4.e112695.56","year":"2012","issue":null,"author":"Bloxham","volume":"18","first-page":"185","unstructured":null,"article-title":"‘You can see the quality in front of your eyes’: grounding academic standards between rationality and interpretation","journal-title":"Quality in Higher Education","doi-asserted-by":"crossref"},{"DOI":"10.1007/978-3-030-02511-3_4","key":"2026041417050707000_16.4.e112695.57","year":null,"issue":null,"author":null,"volume":null,"first-page":null,"unstructured":"Delgado López-Cózar E , Orduña-Malea E , Martín-Martín A , et al . Google Scholar as a data source for research assessment. In: Glänzel W , Moed HF , Schmoch U , eds. Springer Handbook of Science and Technology Indicators. Cham: Springer International Publishing, 2019: 95–127.","article-title":null,"journal-title":null,"doi-asserted-by":"crossref"},{"DOI":"10.18653/v1/2023.emnlp-main.453","key":"2026041417050707000_16.4.e112695.58","year":null,"issue":null,"author":null,"volume":null,"first-page":null,"unstructured":"Chang K , Cramer M , Soni S , et al . Speak, memory: an archaeology of books known to ChatGPT/GPT-4. In: Bouamor H , Pino J , Bali K , eds. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing; Stroudsburg, PA, USA: Association for Computational Linguistics 2023:7312–27, Singapore. doi:10.18653/v1/2023.emnlp-main.453 Available: https://aclanthology.org/2023.emnlp-main","article-title":null,"journal-title":null,"doi-asserted-by":"crossref"},{"DOI":null,"key":"2026041417050707000_16.4.e112695.59","year":null,"issue":null,"author":null,"volume":null,"first-page":null,"unstructured":"Spennemann DH . What has ChatGPT read? The origins of archaeological citations used by a generative artificial intelligence application. arXiv:2308.03301v1 [cs.AI]. 2023. doi:10.48550/arXiv.2308.03301","article-title":null,"journal-title":null,"doi-asserted-by":null},{"DOI":null,"key":"2026041417050707000_16.4.e112695.60","year":null,"issue":null,"author":null,"volume":null,"first-page":null,"unstructured":"Roundtable on Health Literacy, Board on Population Health and Public Health Practice, Institute of Medicine . In: Health Literacy: Past, Present, and Future: Workshop Summary. Washington (DC): National Academies Press (US), 2015.","article-title":null,"journal-title":null,"doi-asserted-by":null},{"DOI":"10.3109/17538157.2010.542529","key":"2026041417050707000_16.4.e112695.61","year":null,"issue":null,"author":null,"volume":null,"first-page":null,"unstructured":null,"article-title":null,"journal-title":null,"doi-asserted-by":"publisher"},{"DOI":"10.2196/60939","key":"2026041417050707000_16.4.e112695.62","year":"2024","issue":null,"author":"Joshi","volume":"8","first-page":null,"unstructured":null,"article-title":"Ensuring Accuracy and Equity in Vaccination Information From ChatGPT and CDC: Mixed-Methods Cross-Language Evaluation","journal-title":"JMIR Form Res","doi-asserted-by":"crossref"},{"DOI":"10.1038/s42256-024-00976-7","key":"2026041417050707000_16.4.e112695.63","year":"2025","issue":null,"author":"Steyvers","volume":"7","first-page":"221","unstructured":null,"article-title":"What large language models know and what people think they know","journal-title":"Nat Mach Intell","doi-asserted-by":"crossref"},{"DOI":"10.1177/0963662514549688","key":"2026041417050707000_16.4.e112695.64","year":null,"issue":null,"author":null,"volume":null,"first-page":null,"unstructured":null,"article-title":null,"journal-title":null,"doi-asserted-by":"publisher"},{"DOI":"10.1371/journal.pone.0268320","key":"2026041417050707000_16.4.e112695.65","year":"2022","issue":null,"author":"Kara","volume":"17","first-page":null,"unstructured":null,"article-title":"Composite measures of quality of health care: Evidence mapping of methodology and reporting","journal-title":"PLoS ONE","doi-asserted-by":"crossref"},{"DOI":"10.1038/s41591-025-04074-y","key":"2026041417050707000_16.4.e112695.66","year":"2026","issue":null,"author":"Bean","volume":"32","first-page":"609","unstructured":null,"article-title":"Reliability of LLMs as medical assistants for the general public: a randomized preregistered study","journal-title":"Nat Med","doi-asserted-by":"crossref"}],"short-title":[],"journal-issue":{"issue":"4","published-print":{"date-parts":[[2026,4]]},"published-online":{"date-parts":[[2026,4,14]]}},"update-policy":"https://doi.org/10.1136/crossmarkpolicy","alternative-id":["10.1136/bmjopen-2025-112695"],"content-domain":{"domain":["bmj.com"],"crossmark-restriction":true},"original-title":[],"container-title":["BMJ Open"],"published-print":{"date-parts":[[2026,4]]},"reference-count":66,"published-online":{"date-time":null,"timestamp":null,"date-parts":[[2026,4,14]]},"references-count":66,"short-container-title":["BMJ Open"],"is-referenced-by-count":2},"unpaywall":{"doi":"10.1136/bmjopen-2025-112695","is_oa":true,"title":"Generative artificial intelligence-driven chatbots and medical misinformation: an accuracy, referencing and readability audit","updated":"2026-05-08T16:02:47+00:00","published_date":"2026-04-01","best_oa_location":{"url":"https://doi.org/10.1136/bmjopen-2025-112695","pmh_id":null,"url_for_pdf":null,"url_for_landing_page":"https://doi.org/10.1136/bmjopen-2025-112695"}}},"created_at":"2026-05-07T22:50:59.674Z","updated_at":"2026-05-14T15:13:26.918Z","tags":[{"paper_doi":"10.1136/bmjopen-2025-112695","tag_type":"age_group","value":"mixed","value_numeric":null},{"paper_doi":"10.1136/bmjopen-2025-112695","tag_type":"category","value":"Medicine, General & Internal","value_numeric":null},{"paper_doi":"10.1136/bmjopen-2025-112695","tag_type":"condition","value":"Cancer","value_numeric":null},{"paper_doi":"10.1136/bmjopen-2025-112695","tag_type":"condition","value":"Medical Misinformation","value_numeric":null},{"paper_doi":"10.1136/bmjopen-2025-112695","tag_type":"condition","value":"Stem Cell Therapy","value_numeric":null},{"paper_doi":"10.1136/bmjopen-2025-112695","tag_type":"condition","value":"Vaccine-Preventable Disease","value_numeric":null},{"paper_doi":"10.1136/bmjopen-2025-112695","tag_type":"impact_flag","value":"safety_signal","value_numeric":null},{"paper_doi":"10.1136/bmjopen-2025-112695","tag_type":"level_of_evidence","value":"4","value_numeric":null},{"paper_doi":"10.1136/bmjopen-2025-112695","tag_type":"modality","value":"Generative Artificial Intelligence","value_numeric":null},{"paper_doi":"10.1136/bmjopen-2025-112695","tag_type":"sample_size","value":"n=250","value_numeric":250},{"paper_doi":"10.1136/bmjopen-2025-112695","tag_type":"sex","value":"mixed","value_numeric":null},{"paper_doi":"10.1136/bmjopen-2025-112695","tag_type":"study_design","value":"Audit / Content Analysis","value_numeric":null},{"paper_doi":"10.1136/bmjopen-2025-112695","tag_type":"warning","value":"Results reflect chatbot versions available in February 2025 — findings may not apply to current or future model releases.","value_numeric":null},{"paper_doi":"10.1136/bmjopen-2025-112695","tag_type":"warning","value":"Study design is an audit with expert raters, not a controlled trial; level of evidence is approximate.","value_numeric":null},{"paper_doi":"10.1136/bmjopen-2025-112695","tag_type":"warning","value":"The adversarial prompt framework was designed to stress-test models toward misinformation, which may not reflect typical real-world patient query patterns.","value_numeric":null}],"summaries":{"limitations":"Only 10 questions per category were tested, limiting generalizability; expert rater subjectivity may influence 'problematic' classifications despite predefined criteria; findings reflect a single snapshot in February 2025 and may not apply to updated model versions.","practice_takeaway":"Clinicians should warn patients that AI chatbots frequently produce inaccurate, confidently worded health information—especially in areas like nutrition and stem cells—and that chatbot-cited references cannot be trusted without independent verification.","what_studied":"An audit of five AI chatbots (Gemini, DeepSeek, Meta AI, ChatGPT, Grok) responding to 50 health/medical questions across five misinformation-prone categories (cancer, vaccines, stem cells, nutrition, athletic performance) in February 2025, using adversarial prompts rated by domain experts for accuracy, citation quality, and readability.","key_result":"49.6% of responses were problematic (30% somewhat, 19.6% highly problematic); Grok produced disproportionately more highly problematic responses (z=+2.07, p=0.038); nutrition and athletic performance were the weakest categories; citation completeness had a median of only 40%; all responses were at college-level reading difficulty, and hallucinated/fabricated citations appeared in every chatbot."}}],"initialOffset":1,"fetchConfig":{"kind":"tags","tags":[{"tag_type":"study_design","value":"Audit / Content Analysis"}]},"breadcrumbItems":[{"name":"Home","href":"/"},{"name":"Research Hub","href":"/hub"},{"name":"Study Designs","href":"/hub/tag/study_design"},{"name":"Audit / Content Analysis"}],"hrefConfig":{"basePath":"/hub/papers","webViewQueryParams":{"hub_ctx":"tag","hub_tag_type":"study_design","hub_tag_value":"Audit / Content Analysis"}}}]]

Papers