Analysis of Physiological Signals for Emotion Charting

Abstract

Human emotions are complex mental and physiological states that arise in response to various internal and external stimuli. Emotional health is an important part of human physical and psychological health. An emotionally healthy person can perform multiple roles in his life in the best possible way, however, due to emotional imbalance and cognitive disorders a person can face so many physical and mental health issues. Therefore, the timely diagnosis of these mental illness can prevent severe mental disorders and can improve the quality of medical care. Recently, emotion recognition has gained great attention in affective computing and different modalities have been used for emotion recognition i.e. human physical signals and human physiological signals. Human physiological signals are considered to be most reliable source for emotion recognition as compared to human physical signals because they can’t be manipulated. Emotion charting using multimodal signals have grown in popularity due to wide multidisciplinary applications such as health care department, neuromarketing,, robotics, safety, security and e-gaming etc. There are number of physiological markers such as heart rate, respiration, electrodermal activity, conductance and brain activity , which can be used for performing emotion recognition. However, physiological signals such as Electrocardiogram (ECG) signals and Electroencephalograms (EEG) signals measure the cardiac and neuronal activities respectively, connected with different human emotional states . In addition to this, Galvanic Skin Response (GSR) signals are also very highly correlated with the emotional states of human. Therefore, these physiological signals are incorporated in the proposed method and they can be analyzed using different techniques of advanced signal processing and machine learning in order to identify the hidden patterns and classify the emotional states. Previously, researchers have developed different methods for classification of these signals for emotion detection but still there is a need to bridge the connection between the anatomy of human physiological signals and cognitive behaviors by critically analyzing the variation in the waveforms of physiological signals with respect to human emotions. Keeping this in view, this research work proposes two deep learning-based approaches for emotion charting using physiological signals. First approach is an Ensemble method using customized convolutional Stacked Autoencoder (ESA) for Emotion Charting. This approach performs preprocessing of physiological signals (EEG, ECG and GSR) using bandpass filtering and Independent Component Analysis (ICA) followed by Discrete Wavelet Transform (DWT). Then convolutional stacked autoencoder has been employed for feature extraction from the scalograms of physiological signals. Feature vector obtained from stacked autoencoder is then fed to three classifers SVM (Support Vector Machine), RF (Random Forest), and LSTM (Long Short-Term Memory). The outputs of classifers are combined using majority voting scheme for the fnal classifcation of signals into four emotional classes i.e. High Valence and High Arousal (HVHA), Low Valence and Low Arousal (LVLA), High Valence and Low Arousal (HVLA) and Low Valence High Arousal (LVHA). However, second approach is CNN-Vision Transformer (CVT) based emotion charting using ensemble classifcation. In this approach, initially signals are decomposed into non-overlapping segments, the noise is removed using bandpass fltering followed by ICA. Then two feature sets are obtained from 1D CNN and Vision Transformer, which are then combined to generate a single feature vector. Finally feature vector is fed to an ensemble classifer composed of LSTM (Long Short-Term Memory), ELM (Extreme Learning Machine) and SVM (Support Vector Machines) classifiers. The probabilities generated from each classifer is fed as input to few shot learning based technique Model Agnostic Meta Learning (MAML) which combines classifers outputs and generates a single output in the form of emotional classes. The proposed system is validated on AMIGOS and DEAP datasets with 10-fold cross validation and obtained the highest accuracy of 94.75 % , sensitivity of 99.15% and specifcity of 97.61 % with ESA based emotion charting. However, on the other hand the proposed system achieved the highest accuracy of 98.2 %, sensitivity of 98.4% and specifcity of 99.53% with CVT based approach. The proposed system outperforms the state-of-the-art emotion charting methods in terms of accuracy, sensitivity and specifcity.