Sign2text: Real-time Sign Language Translator Using Deep Learning

Abstract

This thesis examines the Sign language translator, one of the most useful computer vision and action recognition applications. Previous sign language translators often do not offer action gesture translation, lack stability, and only translate letters or words, resulting in slow communication. This application aims to equip special people of our society with a powerful tool, so they do not feel they are missing out on anything. Upon our research, we found that many sign language interpreters have good accuracy, but many were word-based or letter-based interpreters. Therefore, the need evolved to create a sign language interpreter that not only interprets signs limited to just hand gestures and positioning but also involves a dynamic approach to include body movement. Our two-sign language-based interpreters, static and dynamic, are based on neural networks such as CNN and LSTM respectively. In the static sign language-based interpreter, we use CNN to translate different signs into words and then send the words into GPT API to create meaningful sentences by prompt. Dynamic, on the other hand, works using the MediaPipe holistic function, which is used to draw body landmarks and then detect actions done by the user using temporal modeling and long short-term memory cells. Results showed that our trained model was able to provide 94dataset and 90.8% accuracy on the dynamic dataset. The result of our pre-trained model for the static dataset was then fed into the GPT transformer by an API call, which provided meaningful sentences as output. Finally, we concluded that by using the neural net, we will be able to train our model on more words as we go along and can generate more meaningful sentences, as well as train more actions on the dynamic temporal model. Additionally, Sign2Text will be made readily available online, anywhere and anytime, for fluent communication.