Advanced Malware Detection for Proactive Defense Against Evolving Cyber Threat

Abstract

The rising complexity of malware, particularly in environments of high security like air-gapped networks, requires sophisticated detection mechanisms that can detect sophisticated and obfuscated threats. This thesis proposes a dynamic malware detection model that employs deep learning models to examine Windows API call sequences derived from executable binaries. The initial dataset, obtained from Kaggle, had a significant class imbalance (malicious: 42,797; benign: 1,079) that was alleviated by applying the SMOTE method to balance training data. Comparative analysis of seven deep models—Simple ANN, MLP, DropConnect Improved ANN, Residual ANN, DenseNet ANN, RBF Network, and hybrid CNN-LSTM—was performed on a range of metrics including accuracy, precision, recall, F1-score, and ROC-AUC for both 50 and 150 training epochs. Of these, the CNN-LSTM model, supplemented by an attention mechanism, showed better performance in distinguishing between benign and malicious samples. In this regard, accuracy gain is minor +0.08%, the significantly largest number in the Class 0 recall is +4.1% and F1-score is +2.7%. The attention-augmented architecture is the central new contribution of this study, which reduces interpretability and improves concentration on important behavioral features. The top-performing model was incorporated into a web-based malware sanitization tool for use in standalone network environments. In total, this work closes the gap between experimental academia and practical use of deep-learning-based intelligent and robust malware detection systems.