Real-time deep visual semantic alignment for automatic image captioning (P-0007) (MFN 8646)

Abstract

Humans are capable of easily describing the environment they are in because they possess cognitive abilities. However, it is difficult for machines to infer the visual world around them. By blending the concepts of computer vision, Natural Language Processing (NLP) and deep learning, we implement an embedded system that understands the spatial relationship of objects in the images and describe them in natural language. The proposed system can be utilized in many different scenarios mainly in natural robot human interactions, navigation for the blind, image retrieval, creating social media content and for early childhood development. Cross modal retrieval is performed to generate semantically correct image descriptions. Convolution Neural Network (CNN) is used for image pre-processing and Long Short-Term Memory (LSTM) module is use for text pre-processing followed by word embeddings. Then both, as input vectors are provided to Feed Forward Neural Network (FFNN) to create semantically and syntactically correct sentence. The algorithms are trained using Flickr8k dataset and after performing model selection the system is implemented on hardware as a stand-alone device. Our systems performance is calculated by the NLP scores namely, BLEU, METEOR and CIDER score. These scores measure precision, recall and accuracy respectively.