Fine-Tuning Domain-Specific LLMs for Medical Named Entity Recognition (NER) and Context-Aware Summarization

One of the biggest challenges faced by to-day’s healthcare system is the reliance on legacy systems which rely on paper-based data storage systems. Health practitioners tend to document critical details such as medication, lab reports, and discharge information manu¬ally within the current fragmented system. In the absence of an integrated digital system, there would be increased workload among clinicians to enter data manually. Manual data entry is prone to error as the process can be exhausting for clinicians and involves legible writing. To address this persistent challenge within our healthcare system, we present an intelligent system called CareTrack. In essence, the architecture of this prototype levarage the use of multimodal vision transformers along with a large language model. Most notably, it adopts the technique of retrieval-augmented generation (RAG) that supports contextual understanding of the medical record in question instead of basic text recognition. When tested on various medical documents, the results were highly encouraging. In particular, the proposed model was able to score 98.

Named Entity Recognition, Vision Transformers, Healthcare Digitization

1. T. Davenport and R. Kalakota, “The potential for artificial intelligence in healthcare,” Nature Medicine, vol. 25, no. 1, pp. 94–98, 2023. [Google Scholar] [Crossref]

2. R. Miotto, L. Li, B. A. Kidd, and J. T. Dudley, “Deep Patient: An Unsupervised Representation to Predict the Future of Pa¬tients from the Electronic Health Records,” Scientific Reports, vol. 6, p. 26094, 2016. [Google Scholar] [Crossref]

3. Y. Si, J. Du, Z. Li, X. Jiang, T. Miller, F. Wang, W. Zheng, and K. Roberts, “Deep Representation Learning of Patient Data from Electronic Health Records,” Journal Biomedical Informatics, vol. 112, p. 103594, 2020. [Google Scholar] [Crossref]

4. A. Dosovitskiy et al., “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale,” in Int. Conf. on Learning Representations (ICLR), 2021. [Google Scholar] [Crossref]

5. S. Eslami, G. de Melo, and C. Meinel, “Aligning Text and Image for Medical Visual Question Answering,” in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition, 2022, pp. 2145–2154. [Google Scholar] [Crossref]

6. P. Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,” in Advances in Neural In¬formation Processing Systems, vol. 33, pp. 9459–9474, 2020. [Google Scholar] [Crossref]

7. C. Zakka, R. Shad, and A. Hiesinger, “Retrieval-Augmented Generation for Clinical Decision Support: A Case Study,” Journal of Biomedical Informatics, vol. 142, p. 104289, 2023. [Google Scholar] [Crossref]

8. C. Yang et al., “POPDx: Automated Patient Phenotyping across 392,246 Individuals in UK Biobank,” Journal of the American Medical Informatics Association, vol. 30, no. 5, pp. 892–902,2023. [Google Scholar] [Crossref]

9. A. Neuraz et al., “Facilitating Phenotyping from Clinical Texts: The Medkit Library,” Bioinformatics, vol. 40, no. 2, p. btae064, 2024. [Google Scholar] [Crossref]

10. X. Garcia, L. Chen, and M. Smith, “Improving Automated Deep Phenotyping through Large Language Models and RAG-HPO,” BMC Medical Genomics, vol. 18, p. 42, 2025. [Google Scholar] [Crossref]

11. J. Clusmann, J. Kolbinger, and H. Mussmann, “Generative AI for Medical Report Summarization: Performance and Clinical Utility,” NPJ Digital Medicine, vol. 7, no. 1, 2024. [Google Scholar] [Crossref]

12. A. Moor et al., “Foundation models for generalist medical artificial intelligence,” Nature, vol. 616, no. 7956, pp. 259–265,2023. [Google Scholar] [Crossref]

13. J. Devlin, M. W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” in Proc. NAACL, 2019, pp. 4171–4186. [Google Scholar] [Crossref]

14. A. Al-Shammari and A. Al-Ghamdi, “Secure Cloud-Based Personal Health Record System Using Firebase,” IEEE Access, vol. 9, pp. 12345–12356, 2021. [Google Scholar] [Crossref]

15. S. Sebastian, FastAPI: A Modern, Fast (High-Performance) Web Framework for Building APIs with Python. Sebastopol, CA: O’Reilly Media, 2022. [Google Scholar] [Crossref]

• What the Desert Fathers Teach Data Scientists: Ancient Ascetic Principles for Ethical Machine-Learning Practice
• Comparative Analysis of Some Machine Learning Algorithms for the Classification of Ransomware
• Comparative Performance Analysis of Some Priority Queue Variants in Dijkstra’s Algorithm
• Transfer Learning in Detecting E-Assessment Malpractice from a Proctored Video Recordings.
• Dual-Modal Detection of Parkinson’s Disease: A Clinical Framework and Deep Learning Approach Using NeuroParkNet