Welcome to the forefront of conversational AI as we explore the fascinating world of AI chatbots in our dedicated blog series. Discover the latest advancements, applications, and strategies that propel the evolution of chatbot technology. From enhancing customer interactions to streamlining business processes, these articles delve into the innovative ways artificial intelligence is shaping the landscape of automated conversational agents. Whether you’re a business owner, developer, or simply intrigued by the future of interactive technology, join us on this journey to unravel the transformative power and endless possibilities of AI chatbots.
Six AI Chatbots Show Variable Performance Answering Diabetic Kidney Disease Questions
In a cross-sectional comparative study guided by the CHART framework, researchers found that publicly accessible large language models can provide plausible information about diabetic kidney disease but demonstrate notable variability in performance and clinical safety risks. The investigation revealed that safe-response rates across six evaluated chatbots ranged between 83.3 percent and 92.9 percent, with no statistically significant differences in overall safety among the systems. However, approximately 11.9 percent of all generated answers were classified as potentially unsafe upon qualitative examination.
The research was conducted by investigators from Hunan Medical College and Huaihua Central Hospital in Huaihua, China, to address uncertainty regarding whether generative artificial intelligence tools are suitable for patient education in diabetic kidney disease. The study evaluated six widely accessible chatbot platforms: ChatGPT, Gemini, DeepSeek, Doubao, Kimi, and Qwen. Safety was established as the primary outcome measure, alongside structured assessments of accuracy, empathy, information quality, and textual readability.
To test the models, the researchers developed 42 standardized English questions focusing on diabetic kidney disease. Each inquiry was submitted exactly once to each of the six chatbots in independent, single-turn query sessions, producing a dataset of 252 total responses for formal evaluation.
Evaluation of the generated texts was conducted by five clinical raters who were masked to the source models to minimize assessment bias. Readability metrics were evaluated automatically using computational formulas, while inter-rater reliability was measured using Fleiss’ kappa and intraclass correlation coefficients ICC (2,1) and ICC (2,5).
Analysis of safety evaluations demonstrated substantial agreement among the clinical raters, yielding a Fleiss’ kappa of 0.854. Reliability coefficients showed ICC (2,1) values ranging from 0.552 to 0.855 and ICC (2,5) values ranging from 0.860 to 0.967 across the evaluation dimensions. While safe-response rates varied numerically from 83.3 percent to 92.9 percent across the six models, statistical testing using Cochran’s Q test confirmed that the differences in safety between models were not statistically significant with a P-value of 0.726.
In contrast to overall safety, evaluations of response accuracy and clinician-rated empathy showed statistically significant differences across the six chatbots, with both metrics reaching significance at P < 0.001. ChatGPT achieved the highest scores for factual accuracy, whereas Gemini scored highest among the models for clinician-rated empathy. Information quality scores also differed significantly across the platforms, with ChatGPT demonstrating the best overall performance and Doubao recording the lowest scores.
Automated readability assessments revealed that all six evaluated readability indices differed significantly among the chatbots, with all tests showing P < 0.001 and Kendall’s W values ranging from 0.139 to 0.255.
A qualitative review of the 30 responses identified as potentially unsafe, accounting for 11.9 percent of the 252 generated answers, revealed that hazards most frequently stemmed from incomplete medication-related information or insufficiently individualized advice regarding fluid intake. The authors concluded that while publicly accessible large language models can support patient learning, they must not replace professional clinical guidance, underscoring that guideline anchoring, source verification, readability enhancements, and continuous clinical oversight remain essential safeguards.
To evaluate performance differences across the six chatbots, the researchers implemented Friedman tests alongside Cochran’s Q tests, incorporating Kendall’s W to establish effect sizes and applying the Benjamini–Hochberg correction procedure to adjust for multiple testing. While the primary outcome of safety showed no statistically significant differences among the models, secondary evaluations revealed marked disparities, with factual accuracy, clinician-rated empathy, overall information quality, and automated readability metrics all demonstrating statistically significant differences across the evaluated systems.
Although chatbots can generate plausible information, their variable performance across individual quality dimensions and the presence of clinical safety hazards illustrate why their role remains supplementary. The persistence of readability estimates above the sixth-grade reference level further indicates that language complexity remains an obstacle for patient comprehension across all tested models. Consequently, the authors emphasized that clinical oversight, guideline anchoring, source verification, and targeted readability improvements remain indispensable prerequisites before these conversational tools can safely support diabetic kidney disease management.
Subject of Research: Performance and safety evaluation of artificial intelligence chatbots answering patient-facing diabetic kidney disease questions
Article Title: Safety, accuracy, empathy and information quality of six large language models answering patient-facing questions about diabetic kidney disease: a cross-sectional comparative study
Article References: Yu, S., Huang, R., Zhou, W., Duan, N., & Huang, W. (2026). Safety, accuracy, empathy and information quality of six large language models answering patient-facing questions about diabetic kidney disease: a cross-sectional comparative study. Frontiers in Medicine, 13, Article 2007153. https://doi.org/10.3389/fmed.2026.2007153
Image Credits: AI Generated
DOI: 10.3389/fmed.2026.2007153
Keywords: diabetic kidney disease, artificial intelligence, large language models, patient education, chatbots, clinical safety, medical accuracy, health communication, Safety, accuracy, empathy, information
News Source: Jerry Hayes. (November 8, 2026). Six AI Chatbots Show Variable Performance Answering Diabetic Kidney Disease Questions. Scienmag.
We bring you the latest biotechnology news from best research centers and universities around the world. Check our website.
Enter your email address to subscribe to this blog and receive notifications of new posts by email.
Bioengineer.org © Copyright 2023 All Rights Reserved.
Login to your account below
Please enter your username or email address to reset your password.
Bioengineer.org © Copyright 2023 All Rights Reserved.
