Image Based PE File Detection and Classification

Abstract

The rapid evolution of malware and its increasing complexity necessitate innovative detection and classification methods to ensure robust cybersecurity. Traditional methods, which largely rely on signature-based and heuristic approaches, struggle to keep pace with the sophisticated tactics employed by modern malware. These conventional techniques are often thwarted by obfuscation strategies and polymorphic behaviors, which allow malware to evade detection by altering their signatures. Consequently, there is a pressing need for advanced methodologies capable of overcoming these limitations. This thesis introduces a novel approach to malware detection and classification through the use of image-based deep learning techniques. By converting Portable Executable (PE) malware binaries into grayscale images, we exploit the visual pattern recognition strengths of Convolutional Neural Networks (CNNs). This transformation allows the models to detect intricate structural patterns within the malware data that are not readily apparent through traditional text-based analysis. We conducted experiments using six state-of-the-art deep learning models: XceptionNet, EfficientNetB0, ResNet50, VGG16, DenseNet169, and a custom-designed Conv2D model. The Malimg dataset, which contains 9,342 samples spanning 25 distinct malware families, provided a diverse and comprehensive basis for evaluation. Each sample in the dataset was transformed into a grayscale image, enabling the CNNs to learn from and identify unique visual features associated with different malware families. The Conv2D model demonstrated exceptional performance, achieving a test accuracy of 99.24%. This result surpasses the accuracy of the current leading system, which stands at 98.65%. The superior performance of the Conv2D model can be attributed to its ability to capture detailed visual patterns and nuances within the malware images, which are critical for accurate classification. The success of this model underscores the potential of image-based analysis in enhancing the precision and effectiveness of malware detection systems. In addition to demonstrating high accuracy, the use of image-based deep learning models offers several advantages over traditional methods. These models reduce the reliance on manual feature extraction, as they autonomously learn and identify relevant features during the training process. This capability is particularly beneficial in handling the dynamic nature of malware, where new variants frequently emerge. The adaptability and scalability of CNNs make them well-suited for continuous learning and improvement, ensuring that the detection system remains effective against evolving threats Furthermore, this research highlights the feasibility of integrating advanced machine learning techniques into cybersecurity frameworks. The successful application of CNNs to image-based malware detection paves the way for future research to explore additional deep learning architectures and their potential in other domains of cybersecurity. For instance, expanding this approach to analyze different types of malware datasets or incorporating reinforcement learning could further enhance the robustness and versatility of malware detection systems. In conclusion, this thesis presents a significant advancement in the field of malware detection and classification. By leveraging the power of deep learning and image-based analysis, we provide a robust solution that addresses the limitations of traditional methods. The high accuracy achieved by the Conv2D model demonstrates the efficacy of this approach, offering a promising direction for future research and development in cybersecurity. The integration of these techniques into practical applications can significantly bolster the defenses against the ever-evolving landscape of cyber threats, ensuring enhanced protection for computer systems and networks.