Comparative Analysis of the Readability of Artificial Intelligence-Supported Educational Materials for Menengitis: Chatgpt and Gemini

Background: Artificial intelligence (AI)-based large language models (LLMs) are increasingly used to generate patient educational materials. However, the readability of AI-generated information is a key determinant of patient comprehension and health literacy. This study compared the readability of educational materials on meningitis generated by ChatGPT and Gemini.
Methods: This cross-sectional comparative study evaluated 80 AI-generated educational texts on meningitis, including 40 responses generated by ChatGPT and 40 by Gemini using an identical standardized prompt. Readability was assessed using nine validated readability indices: Automated Readability Index (ARI), Flesch Reading Ease, Flesch-Kincaid Grade Level, Gunning Fog Index, Coleman-Liau Index, SMOG Index, Linsear Write Formula, Dale-Chall Readability Score, and Spache Readability Formula. Continuous variables were expressed as mean ± standard deviation, and comparisons between groups were performed using appropriate statistical tests. A p-value <0.05 was considered statistically significant.
Results: Most readability indices did not differ significantly between ChatGPT and Gemini. ARI, Flesch Reading Ease, Gunning Fog Index, SMOG Index, Linsear Write Formula, Dale-Chall Readability Score, and Spache Readability Formula were comparable between the two platforms (all p >0.05). However, ChatGPT-generated materials demonstrated significantly higher Flesch-Kincaid Grade Level scores than Gemini (9.25 ± 2.64 vs. 7.66 ± 2.42, p <0.001) and significantly higher Coleman-Liau Index scores (16.74 ± 2.11 vs. 12.76 ± 2.12, p =0.03), indicating greater reading complexity. Overall, both AI models generated educational materials above the sixth-grade reading level recommended for patient education.
Conclusions: ChatGPT and Gemini produced patient educational materials on meningitis with generally similar readability; however, ChatGPT generated significantly more complex texts according to the Flesch-Kincaid Grade Level and Coleman-Liau Index. Despite their potential as educational tools, both AI platforms produced content that exceeded recommended readability levels for the general population. Further refinement of AI-generated health information is needed to improve accessibility and support patients with diverse health literacy levels.

Meningitis; Artificial intelligence; ChatGPT; Gemini; Readability; Health literacy; Patient education.

1. van de Beek D, Brouwer MC, Thwaites GE, Tunkel AR. Advances in treatment of bacterial meningitis. Lancet. 2012;380(9854):1693-702. [Google Scholar] [Crossref]

2. Tunkel AR, Hartman BJ, Kaplan SL, Kaufman BA, Roos KL, Scheld WM, et al. Practice guidelines for the management of bacterial meningitis. Clin Infect Dis. 2004;39(9):1267-84. [Google Scholar] [Crossref]

3. Brouwer MC, Tunkel AR, van de Beek D. Epidemiology, diagnosis, and antimicrobial treatment of acute bacterial meningitis. Clin Microbiol Rev. 2010;23(3):467-92. [Google Scholar] [Crossref]

4. World Health Organization. Defeating meningitis by 2030: global road map. Geneva: WHO; 2021. [Google Scholar] [Crossref]

5. OpenAI. GPT-4 Technical Report. arXiv. 2023;2303.08774. [Google Scholar] [Crossref]

6. Google DeepMind. Gemini: A family of highly capable multimodal models. arXiv. 2023;2312.11805. [Google Scholar] [Crossref]

7. Sallam M. ChatGPT utility in healthcare education, research, and practice: Systematic review. Healthcare (Basel). 2023;11(6):887. [Google Scholar] [Crossref]

8. Kung TH, Cheatham M, Medenilla A, Sillos C, De Leon L, Elepaño C, et al. Performance of ChatGPT on USMLE: Potential for AI-assisted medical education. PLoS Digit Health. 2023;2(2):e0000198. [Google Scholar] [Crossref]

9. Berkman ND, Sheridan SL, Donahue KE, Halpern DJ, Crotty K. Low health literacy and health outcomes: An updated systematic review. Ann Intern Med. 2011;155(2):97-107. [Google Scholar] [Crossref]

10. Institute of Medicine. Health Literacy: A Prescription to End Confusion. Washington, DC: National Academies Press; 2004. [Google Scholar] [Crossref]

11. Weiss BD. Health Literacy and Patient Safety: Help Patients Understand. 2nd ed. Chicago: American Medical Association Foundation; 2007. [Google Scholar] [Crossref]

12. Doak CC, Doak LG, Root JH. Teaching Patients with Low Literacy Skills. 2nd ed. Philadelphia: JB Lippincott; 1996. [Google Scholar] [Crossref]

13. Flesch R. A new readability yardstick. J Appl Psychol. 1948;32(3):221-33. [Google Scholar] [Crossref]

14. Kincaid JP, Fishburne RP Jr, Rogers RL, Chissom BS. Derivation of new readability formulas for Navy enlisted personnel. Res Branch Rep. 1975;8-75. [Google Scholar] [Crossref]

15. Mc Laughlin GH. SMOG grading: A new readability formula. J Read. 1969;12(8):639-46. [Google Scholar] [Crossref]

16. Gunning R. The Technique of Clear Writing. New York: McGraw-Hill; 1952. [Google Scholar] [Crossref]

17. Coleman M, Liau TL. A computer readability formula designed for machine scoring. J Appl Psychol. 1975;60(2):283-4. [Google Scholar] [Crossref]

18. Chall JS, Dale E. Readability Revisited: The New Dale-Chall Readability Formula. Cambridge (MA): Brookline Books; 1995. [Google Scholar] [Crossref]

19. Badarudeen S, Sabharwal S. Assessing readability of patient education materials: Current role in orthopaedics. Clin Orthop Relat Res. 2010;468(10):2572-80. [Google Scholar] [Crossref]

20. Rooney MK, Santiago G, Perni S, Horowitz DP, McCall AR, Einstein AJ, et al. Readability of patient education materials in radiation oncology. Pract Radiat Oncol. 2021;11(3):e263-e271. [Google Scholar] [Crossref]

21. Ayre J, Muscat DM, Mac O, Batcup C, Cvejic E, Pickles K, et al. Mainstream readability formulas underestimate understanding of patient education materials. Health Expect. 2020;23(3):687-96. [Google Scholar] [Crossref]

22. Lee P, Bubeck S, Petro J. Benefits, limits, and risks of GPT-4 as an AI chatbot for medicine. N Engl J Med. 2023;388(13):1233-9. [Google Scholar] [Crossref]

23. Johnson D, Goodman R, Patrinely J, Stone C, Zimmerman E, Donald R, et al. Assessing the accuracy and readability of AI-generated medical responses. NPJ Digit Med. 2023;6:156. [Google Scholar] [Crossref]

24. Temsah MH, Aljamaan F, Alhasan K, et al. ChatGPT and large language models in healthcare: Opportunities and challenges. Cureus. 2023;15(8):e43614. [Google Scholar] [Crossref]

25. Eysenbach G. Improving the quality of web-based health information. J Med Internet Res. 2002;4(3):e15. [Google Scholar] [Crossref]

26. Nielsen-Bohlman L, Panzer AM, Kindig DA, editors. Health Literacy: Improving Health, Health Systems, and Health Policy Around the World. Washington (DC): National Academies Press; 2004 [Google Scholar] [Crossref]

• Treasure of 6th Edition of Organon of Medicine- Lm Potency
• Neurobehavioral and Immune-Histochemical Effects of Daucus Carota Ethanolic Leaf Extract in Cadmium-Induced Toxicity of the Hippocampus and Prefrontal Cortex of Adult Wistar Rats
• Unveiling the Antimicrobial Potential of Homoeopathic Sarcode and Bowel Nosode: A Study on Histaminum and Dysenteriae Co against Escherichia Coli And Klebsiella Bacteria
• Incidence of Cardiovascular Events in Patients Receiving Hematopoietic Stem Cell Transplant (HSCT) in HCTM – (A Retrospective Cohort Study)
• A Prospective Case Study Examining the Effectiveness of Homoeopathic Medicine in Treating Degenerative Changes in the Lumbar Spine-A Case Report