A Framework for Detection of Fileless and Cryptojacking Malware

Abstract

The rise of stealthy and evasive mal ware particularly fileless mal ware and cryptojacking poses a significant challenge to traditional security mechanisms that rely on file-based signatures and static analysis. This thesis presents a hybrid detection framework that leverages memory forensics and deep learning to identify sophisticated in-memory threats on Windows systems. Integrating runtime telemetry (via Sysmon and Event Tracing for Windows) with forensic data extracted from memory dumps using the Volatility 3 framework, the proposed system enables both real-time detection and post-compromise investigation. To ensure timely capture of volatile evidence, a lightweight Python-based parser monitors critical event IDs and triggers automated memory acquisition. Extracted features are then passed to Transformerbased deep learning models trained independently for each threat class on datasets including CIC-MalMem2022 (fileless malware) and a custom-labeled cryptojacking dataset collected from Windows 10 hosts. The framework demonstrated 99.69% accuracy for fileless malware and 99.79% accuracy for cryptojacking, with Fl-scores of 0.98+ and false positive rates under 1.2% outperforming recent baselines. Test scenarios confirmed its ability to recognize hybrid and polymorphic attack vectors, including LOLBins, process injection, and stealthy in-memory miners. Explainable AI tools such as LIME further enhance analyst trust by aligning model decisions with observable behaviors mapped to the MITRE ATT&CK framework. This study underscores the critical role of memory-centric analytics and demonstrates a scalable, explainable, and high-performing detection pipeline tailored for modern threat landscapes.