AI-Based Threat Intelligence and Anomaly Detection Framework for Hybrid Cloud Security Using Azure and Mitre Att&Ck

Abstract

Traditional cyber security monitoring tools are heavily dependent on a static, signaturebased approach to identifying malicious activity. These can deal with known and knownto-be-known attacks, but are less effective in recognizing new or changing attack patterns, and often give false positives (alarms when there are no attacks) and false negatives (alarms when there are attacks). But this is exacerbated in hybrid cloud deployments, where on-premises systems and public cloud resources like Microsoft Azure contribute to a large and expanse of attack surface that is complex to monitor. This thesis proposes the design, implementation and evaluation of an AI-based Threat Intelligence and Anomaly Detection framework that can overcome these limitations. The system adds a machine learning layer which can learn the normal and abnormal network behaviors directly from the data without requiring hand-written rules, and then maps the generated anomaly predictions to structured threat intelligence that can be understood by analysts in the MITRE ATT&CK knowledge base. As part of this project, due to the lack of availability of real-time Azure Monitor or Azure Sentinel logs for training, the benchmark UNSW-NB15 intrusion detection dataset was used as a substitute data source. The system pipeline was intentionally designed to be data-source agnostic, meaning that the UNSWNB15 dataset can be replaced with a real dataset of Azure telemetry without changing the overall design of the system. Three machine learning models were used and empirically evaluated: an unsupervised model called the Isolation Forest baseline model, a supervised model called the Random Forest Classifier, and an unsupervised model called the Auto encoder-based deep learning model. The accuracy of the Isolation Forest is 54.82%, the Random Forest gives accuracy of 90.09% and F1-Score 0.9224, and the Auto encoder with tuning the reconstruction-error threshold gives accuracy of 68.34%. For the framework's main detection engine therefore, the Random Forest classifier was chosen. The detections created by the Random Forest model are then passed through the MITRE ATT&CK mapping layer which adds a tactic, technique, technique ID, severity rating and contextualizes a prediction of attack/normal (for instance, a prediction of DenialofService is mapped to the tactic, technique, technique ID, and a high severity rating to the event). All processed results are stored in a structured CSV file, and presented in aninteractive Stream lit dashboard with charts showing total events processed, alerts generated, severity distribution and MITRE tactic distribution, and a table of alerts that can be filtered. Experimental results show that supervised machine learning significantly outperforms the unsupervised baseline and the deep learning auto encoder on this dataset, and that the incorporation of MITRE ATT&CK into the anomaly detection meaningfully adds to the interpretability and operational utility of ML-based anomaly detection. The thesis ends with a discussion of the limitations of the framework, namely that it currently relies on an offline benchmark data set, not on the real Azure telemetry, and a roadmap for future work, in which the framework will be extended with real-time data ingestion, federated learning and automated response actions