Signify: Gesture to Speech Sign Language Recognition

Abstract

Individuals with speech impairments often face considerable barriers in expressing themselves, leading to communication challenges in daily life. This project presents Signify, an Android application designed to bridge this gap by converting hand gestures into audible speech. Instead of relying on image-based classification, the system uses MediaPipe to extract 63 hand landmarks in real-time, normalizes them through translation and scale transformations, and feeds them into a pre-trained Dense Neural Network (DNN) converted to TensorFlow Lite for on-device inference. Recognized gestures are collected into a sequence, which is then passed to Cohere’s Lite language model to generate grammatically correct sentences. Finally, the generated text is spoken aloud using Text-to-Speech (TTS) APIs. The app is optimized for Android using CameraX and MediaPipe’s asynchronous frame processing to ensure low latency. With a focus on accuracy, responsiveness, and accessibility, Signify offers a practical communication tool for non-verbal users, promoting greater social inclusion.