Real-Time Multilingual Closed Captioning System with Simplified Captions for Deaf Person

Real-time closed captioning is essential for improving accessibility for Deaf and Hard-of-Hearing (DHH) users in live communication. This paper presents a low-latency multilingual closed-captioning system designed for Indian languages. The proposed system integrates streaming Automatic Speech Recognition (ASR), automatic language identification, text simplification, punctuation restoration, and speaker segmentation. It supports code-switching and optional translation or transliteration across scripts.
The system is optimized for sub-second end-to-end latency and robustness in noisy environments. Experimental results show that simplified captions significantly improve readability and comprehension while maintaining acceptable recognition accuracy, making the system suitable for real-time educational and public communication applications.

Real-time Captioning, Streaming ASR, Multilingual Systems, Accessibility, Deaf and Hard-of-Hearing, Text Simplification

1. R.Ranchaletal.,"UsingSpeech Recognition for RealTimeCaptioningandLectureTranscription in theClassroom,"IEEETransactionson Learning Technologies. [2] A.A.R.Ambilietal.,"TheEffect of Synthetic Voice DataAugmentationonSpokenLanguage Identification on IndianLanguages,"IEEEAccess, 2023. [Google Scholar] [Crossref]

2. M.H.etal. Identification in UnseenTargetDomainUsingWithin-Sample Similarity Loss," ICASSP, 2021 [Google Scholar] [Crossref]

3. B.Pulugundlaetal.,"BUTSystem for Low Resource IndianLanguageASR, "Interspeech, 2018. [Google Scholar] [Crossref]

4. J.Billaetal.,"ISIASRSystem for Low Resource SpeechRecognitionChallengefor Indian Languages, " Interspeech,2018. [Google Scholar] [Crossref]