REAL TIME SPEECH DRIVEN FACE ANIMATION SYSTEM

Abstract

We have opted a paper “LIPNET: END-TO-END SENTENCE-LEVEL LIPREADING” as a base paper of our Final Year Project. Lip-reading is the task of decryption text from the movement of a speaker’s mouth. Ancient approaches separated the issue into 2 stages: planning or learning visual options, and prediction. Newer deep lip-reading approaches are end-to-end trainable (Wand et al„ 2016; Chung & Zisserman, 20j6a). However, existing work on models trained end-to-end perform solely word classification, instead of sentence-level sequence prediction. Studies have shown that human lip-reading performance will for extended words (Easton & Basala, 1982), indicating the importance of options capturing temporal context in an ambiguous communication. Intended by this observation, our project presents, a model that maps a video frames to text, creating of spatial-temporal convolutions, a neural network, and therefore the connection temporal classification loss, trained entirely end-to-end. End-to-end sentence-level lip reading model that at the same time learns spatial-temporal visual options and a sequence model.