Welcome to the forefront of conversational AI as we explore the fascinating world of AI chatbots in our dedicated blog series. Discover the latest advancements, applications, and strategies that propel the evolution of chatbot technology. From enhancing customer interactions to streamlining business processes, these articles delve into the innovative ways artificial intelligence is shaping the landscape of automated conversational agents. Whether you’re a business owner, developer, or simply intrigued by the future of interactive technology, join us on this journey to unravel the transformative power and endless possibilities of AI chatbots.
Study Finds AI Chatbots Offer Epididymitis Information Beyond Patient Reading Levels
In an exploratory cross-sectional investigation examining how artificial intelligence systems answer patient questions about epididymitis, medical researchers discovered that while widely accessible chatbots produce structured and informative explanations, every evaluated system exceeded standard patient-education reading benchmarks and demonstrated poor source transparency. The comparative study, led by Baokun Fan and Cheng Zhou from the Department of Urology and Nephrology at The First Affiliated Hospital of Ningbo University, evaluated 54 outputs across six prominent conversational models. Their findings revealed that while overall educational value was often rated favorably, not a single response achieved the recognized sixth-grade reading level recommended for public health communication, and several models introduced clinically actionable advice regarding antimicrobial regimens that could pose risks if followed without clinician guidance.
Epididymitis is an inflammation of the epididymis that commonly presents with unilateral scrotal or testicular pain, localized tenderness, and swelling. In acute clinical scenarios, the condition is frequently associated with sexually transmitted pathogens such as Chlamydia trachomatis and Neisseria gonorrhoeae in younger individuals, as well as enteric and genitourinary organisms. Because acute epididymitis can present with symptoms that closely mimic testicular torsion—a time-critical urological emergency requiring rapid surgical intervention—accurate, accessible, and cautious patient information is essential. When individuals experience sudden or worsening scrotal symptoms, particularly involving sensitive reproductive or sexual health concerns, many turn to online resources for immediate explanations before consulting a healthcare professional.
To capture realistic patient inquiries, the research team designed a standardized query selection protocol relying on Google Trends. Using the standardized Medical Subject Headings term epididymitis, the authors identified the top 25 queries worldwide over a five-year window spanning August 1, 2021, to August 1, 2026. Two independent reviewers screened these candidate search terms according to predefined eligibility criteria, eliminating nonspecific entries, substantial semantic duplicates, and terms that merely reflected anatomical laterality without a distinct information need. Following adjudication by a third reviewer when necessary, nine core queries were retained: epididymitis pain, epididymitis symptoms, epididymitis cause, what is epididymitis, orchitis epididymitis, epididymitis infection, epididymitis antibiotics, chronic epididymitis, and treatment of epididymitis.
The researchers evaluated six widely available conversational platforms: ChatGPT 5.5, Gemini 3.5 Flash, Perplexity, Copilot, Kimi K2.6, and DeepSeek V4-Flash. Each of the nine retained queries was submitted exactly once to every chatbot through official web interfaces on August 1, 2026, in Ningbo, China. To replicate typical consumer interactions, the platforms were accessed using their basic default operational modes with internet search enabled, avoiding advanced configurations such as Pro tiers or specialized deep research modes. In total, 54 raw responses were gathered without adding clinical background details, health literacy prompts, or customized instructions designed to adjust medical complexity or stylistic tone.
To evaluate the responses, the investigators applied four recognized health communication instruments alongside six mathematical readability formulas. Two medically trained evaluators, blinded to the identity of each chatbot during scoring, evaluated consumer health information quality using DISCERN, educational presentation with the Ensuring Quality Information for Patients (EQIP) tool, attribution using the Journal of the American Medical Association (JAMA) Benchmark Criteria, and overall communication quality with the Global Quality Scale (GQS). Readability indices—including the Automated Readability Index, Flesch Reading Ease Score, Gunning Fog Index, Flesch-Kincaid Grade Level, Coleman-Liau Index, and Simple Measure of Gobbledygook—were calculated using textstat in Python after stripping formatting markers, citation numbers, and standalone web links.
Analysis of the quality metrics revealed notable differences in overall distributions across platforms, though post hoc comparisons did not separate individual models following statistical correction. On the DISCERN scale, which rates treatment choices, uncertainty, and source relevance, overall model scores differed significantly with a Friedman chi-square of 17.655 and a Kendall’s W effect size of 0.392. Median scores reached 40.00 for Copilot, Kimi, and DeepSeek, placing them in the fair quality tier, while Perplexity, Gemini, and ChatGPT registered medians of 34.00, 35.00, and 33.00, falling into the poor quality category. On the EQIP assessment, overall scores also varied significantly, with Copilot achieving the highest median score of 70.00 compared to medians of 55.00 across the other five systems. However, after applying Holm correction for multiple comparisons, no pairwise post hoc contrast between models remained statistically significant.
Despite relatively favorable flow and structure—with 50 of the 54 responses receiving a GQS score of 4 out of 5—the chatbots demonstrated an almost complete absence of verifiable source attribution. On the JAMA Benchmark Criteria, which inspect authorship, citation of evidence, financial disclosure, and content currency, 52 out of 54 responses (96.3%) scored zero points, with only two individual Copilot responses achieving a single point. The researchers noted that while large language models readily construct authoritative-sounding text with structured headings and coherent transitions, they rarely provide verifiable links, publication dates, or clear medical origins for their assertions, making it difficult for non-specialist readers to determine whether clinical guidance reflects current guidelines.
The evaluation of text complexity exposed an unbroken barrier to general consumer comprehension. All six readability indices demonstrated statistically significant between-model variations, but every single generated response failed to meet standard sixth-grade patient-education thresholds. Across all 54 text outputs, every grade-level metric exceeded the target threshold of 6, and every Flesch Reading Ease Score fell below the benchmark minimum of 80, yielding a 0% attainment rate across all evaluated models. Copilot recorded the comparatively lowest median grade levels on several metrics, including an Automated Readability Index of 12.11 and a Simple Measure of Gobbledygook of 12.30, whereas Gemini generated the most complex text with a median Flesch-Kincaid Grade Level of 17.74 and a Flesch Reading Ease Score of 9.21. These elevated reading levels were attributed to multisyllabic anatomical terminology, complex drug names, and lengthy sentence structures combining differential diagnoses, symptom explanations, and therapeutic mechanisms.
In an exploratory safety assessment conducted by two urologists using a conservative threshold for actionable harm, four of the 54 responses (7.4%) were found to contain potentially harmful misinformation. The identified errors were concentrated exclusively in recommendations regarding antimicrobial regimens, involving one response from Perplexity, one from Kimi, and two from DeepSeek, while ChatGPT, Gemini, and Copilot produced no flagged safety events. These instances included presenting single-dose azithromycin as an equivalent substitute for doxycycline, citing outdated ceftriaxone dosing recommendations, or suggesting antimicrobial regimens inconsistent with established empirical management for enteric epididymitis. The investigators emphasized that this 7.4% frequency does not represent an overall factual error rate, as benign inaccuracies, minor omissions, and non-preferred options lacking direct potential for physical harm were intentionally excluded.
The authors contextualized their observations within several methodological boundaries. The study represents a single cross-sectional snapshot taken on August 1, 2026, capturing dynamic consumer models that undergo frequent structural and retrieval modifications. The evaluation was limited to English-language queries and responses, and each query was submitted once, precluding analysis of run-to-run output variability. Furthermore, the standardized queries could not incorporate individualized patient demographics, diagnostic confirmations, or clinical nuance. Fan and Zhou concluded that while modern conversational AI systems can serve as helpful supplemental overviews of general medical concepts, their lack of verifiable sourcing, advanced reading difficulty, and potential for erroneous pharmacological guidance mean they cannot substitute for professional urological consultation or guideline-directed clinical evaluation.
Beyond model-level averages, the researchers tracked substantial variation across individual search queries within each chatbot system. When examining heatmaps representing the evaluation dataset, ratings fluctuated depending on whether an inquiry addressed general definitions, acute pain, chronic inflammation, or specific infectious causes. This query-level variability occurred alongside the pronounced floor effect observed on the JAMA criteria and the narrow distribution on the Global Quality Scale, where the vast majority of outputs clustered tightly at identical score values. The authors noted that summary scores across an entire model must be weighed alongside these underlying query-specific differences, as a system might generate well-structured overviews for basic definitions yet deliver problematic or inconsistent responses when handling complex therapeutic decisions.
The study also evaluated inter-rater reliability among independent assessors before resolving differences by consensus. Pre-consensus agreement demonstrated substantial consistency across the evaluated dimensions, yielding an intraclass correlation coefficient of 0.800 for the DISCERN instrument and 0.770 for the EQIP scale. For the JAMA criteria, unweighted Cohen’s kappa reached 0.913 with an observed agreement of 98.1 percent, while the Global Quality Scale produced a quadratic-weighted kappa of 0.690 with 92.6 percent agreement. In the initial independent safety classification of potentially harmful misinformation, Cohen’s kappa was 0.626 with 92.6 percent observed agreement between the two evaluating urologists.
Placing their results in the context of broader digital health research, the investigators highlighted that conventional assessment tools were originally constructed for static consumer brochures and web pages rather than generative algorithms. Because instruments such as DISCERN, EQIP, and JAMA predate interactive artificial intelligence, they do not directly evaluate algorithmic hallucination risks, citation validity, or guideline currency. The authors suggested that emerging evaluation frameworks will need to combine traditional readability and quality indices with domain-specific measures of clinical actionability. In the interim, they cautioned that polished digital interfaces should not diminish clinician vigilance regarding automated medical guidance.
Subject of Research: Evaluation of AI chatbot information quality, readability, and safety for epididymitis
Article Title: Quality, readability, and potential safety risks of epididymitis information generated by six artificial intelligence chatbots: a cross-sectional comparative study
Article References: Fan, B., & Zhou, C. (2026). Quality, readability, and potential safety risks of epididymitis information generated by six artificial intelligence chatbots: a cross-sectional comparative study. Frontiers in Medicine, 13, Article 1991323. https://doi.org/10.3389/fmed.2026.1991323
Image Credits: AI Generated
DOI: 10.3389/fmed.2026.1991323
Keywords: artificial intelligence, chatbots, epididymitis, urology, health literacy, readability, patient education, DISCERN, EQIP, JAMA benchmark criteria, Global Quality Scale, medical misinformation, ChatGPT, Gemini, Perplexity, Copilot, Kimi, DeepSeek
News Source: Ophelia Keating. (November 14, 2026). Study Finds AI Chatbots Offer Epididymitis Information Beyond Patient Reading Levels. Scienmag.
We bring you the latest biotechnology news from best research centers and universities around the world. Check our website.
Enter your email address to subscribe to this blog and receive notifications of new posts by email.
Bioengineer.org © Copyright 2023 All Rights Reserved.
Login to your account below
Please enter your username or email address to reset your password.
Bioengineer.org © Copyright 2023 All Rights Reserved.
