Deaf Talk

Abstract

The world hardly lives without communication, no matter whether it is in the form of texture, voice or visual expression. The communication among the deaf and dumb people is carried by hand gestures and visual expressions. Gestural communication is always in the scope of confidential and secure communication. Hands and facial parts are immensely important in expressing the thoughts of humans in confidential communication. The aim of our project is to develop a communication system for the deaf and mute people. The problem we are investigating is sign language recognition. In this project we have detected and understood the activities and actions of deaf and mute people that have specific meaning and translated these gestures into a text using different machine learning libraries and technologies. The proposed system consists of Translation Mode in which we have translated signs into text. We have used Supervised Machine Learning which is a subcategory of machine learning and artificial intelligence. It is defined by its use of labeled datasets to train algorithms that to classify data or predict outcomes accurately. For this purpose we have made our own data set consisting of 15 signs that are commonly used in daily life. In order to label the data set we used the LabelImg directory, which is an open source graphic image annotation tool. It’s written in Python and has a graphical user interface built with Qt. Annotations are saved as XML(Extensible Markup Language) files in PASCAL VOC(PASCAL Visual Object Classes) format, we chose 15 words out of many that we use in our daily conversations, for each word we took 15 images of 3 different volunteers 5 images each, with different angles/positions and involving both left and right hands. Then we labelled these images with LabelImg directory in Pascal Voc format for training purpose. After model training and testing we integrated our trained model in android. We customized an open source Tensor flow lite object detection android application by adding our custom tflite model file and label map file with custom labels. We also improved the interface of the application using Android Studio and added a new feature that will tell the user the last detected sign. The backend of the application is written in Kotlin, and we have also created our own logo and layout for a better user experience.