Stress / Workload Assessment Using Multimodal Approach

Abstract

The growing interest in monitoring cognitive workload has sped up the research in intelligent systems that utilize physiological signals. Mental workload (MWL), if unmanaged, can degrade human performance in safety-critical applications. This thesis introduces a deep learning (DL) approach that utilizes both EEG and ECG signals in MWL classification of the publicly available N-back dataset. Methodology begins with a multimodal-based cognitive workload classification system with four DL models and one machine learning (ML) baseline approach. GEL-RF is a hybrid combination designed to provide a lightweight and efficient baseline. The core models are an LSTM with self-attention to learn temporal dependencies and improve interpretability; an LSTM-HASTF model without self-attention to reduce computational complexity but maintain spatial-temporal learning; a proposed hybrid LSTM model with both self-attention and HASTF to jointly learn temporal and spatial interactions; and a final model that employs VAE on fusion features to reduce dimensionality and latency. All models are trained using subject-independent validation on EEG–ECG features extracted from the N-back dataset for both binary and multiclass cognitive workload classification tasks. The results show that the GEL-RF approach provided the best performance with binary accuracy of 97.0% and multiclass accuracy of 96.3%, thus showing its efficiency and generalizability. Deep model performance was best with LSTM+HASTF+SA (Model 3), achieving binary and multiclass accuracies of 94.46% and 90.63%, respectively. VAE outperformed with binary and multiclass accuracies of 94.58% and 90.65%. Model 2, LSTM with HASTF without self-attention, achieved binary and multiclass accuracies of 90.11% and 90.34%, thus showing a balance between efficiency and performance. Model 1, which combined LSTM with self-attention, showed good performance with binary and multiclass accuracies of 89.85% and 90.10%. Though DL models showed improved performance in temporal learning, the GEL-RF approach was more appropriate for real-time or restricted scenarios. This work makes the novel contribution of a scalable and interpretable model for MWL assessment in safety-critical settings like aviation, education, and healthcare.