Real Time Sign Language Conversation Using AI

This project introduces a real-time Indian Sign Language (ISL) recognition and translation system to improve communication between hearing-impaired and non-signing individuals. Using OpenCV and MediaPipe, the system accurately detects and tracks hand landmarks to recognize gestures made with one or both hands. Each gesture is mapped to a predefined ISL phrase, which is then converted into text and spoken output using a non-blocking text-to-speech engine (pyttsx3) for smooth and simultaneous operation. The system also supports text-to-sign translation, displaying pre-recorded ISL videos with synchronized speech for interactive learning. Developed entirely in Python, the model provides real-time performance and requires minimal hardware. It can be applied in educational, public, and assistive communication systems. Overall, the proposed system offers an efficient, affordable, and inclusive solution that enhances accessibility and bridges communication barriers for individuals with hearing and speech disabilities.

Indian Sign Language, gesture recognition, OpenCV, MediaPipe, text-to-speech, accessibility

1. S. Kumar, A. Singh, and R. Patel, “CNN-based Indian Sign Language Recognition,” [Google Scholar] [Crossref]

2. International Journal of Computer Applications, [Google Scholar] [Crossref]

3. R. Sharma and P. Gupta, “Real-Time Indian Sign Language Recognition Using MediaPipe and OpenCV,” International Research Journal of Engineering and Technology [Google Scholar] [Crossref]

4. P. Gupta and N. Kaur, “Vision-Based Gesture Translation with Text-to-Speech Output for ISL,” International Journal of Innovative Science and Research Technology (IJISRT), 2020. [Google Scholar] [Crossref]

5. M. Raj and D. S. Reddy, “Hand Gesture Recognition for Indian Sign Language Translation,” International Journal of Advanced Research in Computer and Communication Engineering, 2021. [Google Scholar] [Crossref]

6. K. S. Ramesh and P. Kumar, “Design and Implementation of Real-Time ISL to Speech Conversion System,” International Journal of Engineering Trends and Technology (IJETT), 2022. [Google Scholar] [Crossref]

7. Z. Yan et al., “Spatial-Temporal Graph Convolutional Networks for Action Recognition,” IEEE TPAMI, , 2019. [Google Scholar] [Crossref]

8. F. Zhang and S. Chen, “Real-Time Hand Pose Estimation Using Deep Learning,” IEEE Access, 2020. [Google Scholar] [Crossref]

9. T. Starner and A. Pentland, “Real-Time American Sign Language Recognition Using Hidden Markov Models, 1997. [Google Scholar] [Crossref]

10. MediaPipe Hands, Google AI, “Real-Time Hand Tracking,” 2022. [Google Scholar] [Crossref]

11. G. Bradski, “The OpenCV Library,” Dr. Dobb’s Journal of Software Tools, 2000. [Google Scholar] [Crossref]

12. M. S. Hossain and G. Muhammad, “Hand [Google Scholar] [Crossref]

13. Gesture Recognition Using CNN and Transfer [Google Scholar] [Crossref]

14. Learning,” IEEE Access, 2019 Kumar and S. Mehta, “Machine Learning Techniques for Gesture Recognition,” IJERT, 2020. Graves, “Speech Synthesis with Deep Neural Networks, 2013. [Google Scholar] [Crossref]

15. D. P. Kingma and J. Ba, “Adam: A Method for Stochastic Optimization,” ICLR, 2015. [Google Scholar] [Crossref]

16. J. B. Tiwari and R. Shukla, “A Review on AIbased Assistive Systems for Hearing-Impaired Users,” IJCSE, 2021. [Google Scholar] [Crossref]

17. A. Mittal and V. Bhatia, “Assistive Communication System for Deaf and Dumb Using Computer Vision,” IJACSA, 2019 [Google Scholar] [Crossref]

18. S. Rautaray and A. Agrawal, “Vision-Based Hand Gesture Recognition for Human–Computer Interaction: A Survey,” Artificial Intelligence Review, 2015. [Google Scholar] [Crossref]

19. M. Elmezain, A. Al-Hamadi, and B. Michaelis, “Robust Methods for Hand Gesture Recognition Using Vision-Based Systems,” IEEE Transactions on Systems, Man, and Cybernetics, 2009. [Google Scholar] [Crossref]

20. J. Shotton et al., “Real-Time Human Pose Recognition in Parts from Single Depth Images,” IEEE CVPR, 2011. [Google Scholar] [Crossref]

21. D. Molchanov, S. Gupta, K. Kim, and J. Kautz, “Hand Gesture Recognition with 3D Convolutional Neural Networks,” IEEE CVPR Workshops, 2015. [Google Scholar] [Crossref]

22. T. Ong and S. Ranganath, “Automatic Sign Language Analysis: A Survey and the Future Beyond Lexical Meaning,” IEEE TPAMI, , 2005. [Google Scholar] [Crossref]

23. P. Premaratne and P. Meegama, “Hand Gesture Recognition Using Feature Extraction,” International Journal of Computer Science and Network Security, 2006. [Google Scholar] [Crossref]

24. R. Poppe, “A Survey on Vision-Based Human Action Recognition,” Image and Vision Computing, 2010. [Google Scholar] [Crossref]

25. S. Mitra and T. Acharya, “Gesture Recognition: A Survey,” IEEE Transactions on Systems, Man, and Cybernetics, 2007. [Google Scholar] [Crossref]

26. H. Cooper, B. Holt, and R. Bowden, “Sign Language Recognition,” IEEE Signal Processing Magazine, 2012. [Google Scholar] [Crossref]

27. M. Koller, H. Ney, and R. Bowden, “Deep Learning of Mouth Shapes for Sign Language,” IEEE ICCV,2015.Chaudhary, J. Raheja, and K. Das, “Intelligent Approaches for Hand Gesture Recognition,” Pattern Recognition and Image Analysis,2013 [Google Scholar] [Crossref]

28. J. F. C. Mendes et al., “Vision-Based Assistive Technology for Deaf People,” Sensors, 2019 [Google Scholar] [Crossref]

29. S. Bheda and A. Radpour, “Using Deep Learning for Sign Language Recognition,” arXiv preprint arXiv:1708.07687, 2017. [Google Scholar] [Crossref]

30. K. Sahoo and S. Mishra, “Computer Vision Based Sign Language Recognition System,” IJCSI, 2018. [Google Scholar] [Crossref]

• Assessment of the Role of Artificial Intelligence in Repositioning TVET for Economic Development in Nigeria
• Teachers’ Use of Assure Model Instructional Design on Learners’ Problem Solving Efficacy in Secondary Schools in Bungoma County, Kenya
• “E-Booksan Ang Kaalaman”: Development, Validation, and Utilization of Electronic Book in Academic Performance of Grade 9 Students in Social Studies
• Analyzing EFL University Students’ Academic Speaking Skills Through Self-Recorded Video Presentation
• Major Findings of The Study on Total Quality Management in Teachers’ Education Institutions (TEIs) In Assam – An Evaluative Study