Multi-modal Vehicular Control (Speech & Gesture)

Abstract

This project is about incorporating gesture and speech based input with an IoT device to increase ease of access to similar devices. This report elaborates on the techniques used to recognise speech and gesture based input and then using them to control an IoT based device. To demonstrate this a robotic vehicle built with Raspberry Pi 3 is being used. The device is designed to be remotely controlled through a laptop over Wi-Fi using gesture or speech based input. For this 3 main components had to be accomplished: Firstly, the Input Application (The one running on the laptop); that reads and processes the gesture and speech based input before transmitting it to the device, this section has been implemented in C#. Secondly, the Device Application (the one running on Raspberry Pi), that utilises the incoming commands to dictate the robotic cars movements, this part of the code is implemented in Python. Thirdly, the hardware; that is the robotic vehicle made using Raspberry Pi 3, DC motors, motor drivers, chassis, wheels and other components. It also utilises a sonar sensor for obstacle detection during forward motion. This system is designed to provide multiple modes of interaction for a device, ensuring the availability of the same functionality regardless of the chosen interaction mode. Recommendations for future development and conclusions are also included in this report.