PETECT: Malicious PE File Detection using Function Call Graph

Abstract

Computers and laptops have become essential tools for many people to store and access personal information, accounts, contacts, and communication services. With the increasing popularity of Windows as an operating system, data privacy has become a major concern due to the rise in cybercrimes and security breaches globally. These breaches often occur when individuals are tricked into downloading malicious software on their devices, such as viruses, Trojans, and other types of malware. To combat this, static analysis is employed on Windows executables to determine whether they are malware or not. Static analysis is a method used to analyze the code of an executable without running it. This is done by analyzing the structure of the code and the instructions it contains. This approach is particularly useful in detecting malware since malware typically has specific features and characteristics that can be identified through static analysis. In static malware analysis, the Windows executable is analyzed based on static parameters such as functions and API calls imported from the PE header file. A Portable Executable (PE) file is the format used for executable files, object code, and DLLs in Windows. The PE header file contains important information about the file, such as the location and size of various sections of the executable. This information can be used to determine whether the file is malware or not. When analyzing an executable using static analysis, the file is disassembled into machine code instructions, and the code is analyzed to identify any suspicious patterns or structures that are commonly found in malware. For example, malware often contains code that attempts to evade detection, such as anti-debugging techniques and anti-VM checks. Static analysis can identify such code and flag it as potentially malicious. Once the static analysis is complete, the information obtained is input into a mathematical model that classifies the file as either malware or good-ware. The mathematical model can use various techniques, such as rule-based systems, decision trees, or machine learning algorithms, to determine whether the file is malicious or not. Rule-based systems use a set of pre-defined rules to determine whether a file is malicious or not, while decision trees use a series of questions to determine the file’s classification. Machine learning algorithms, on the other hand, use statistical models to identify patterns in the data and classify the file based on those patterns. In conclusion, static analysis is a valuable technique for detecting malware on Windows executables. By analyzing the code structure and using mathematical models to classify the file as either malware or goodware, it can help identify and prevent malicious software from causing harm to a user’s device and data. While it has some limitations, static analysis is a fast and efficient way to detect malware and can be combined with dynamic analysis for more accurate results.