Malware Analysis

Malware analysis is the process of determining the functionality, origin, and potential impact of a malicious software sample. It answers critical questions: What does this malware do? How does it communicate? What data does it target? How can we detect and block it?

Malware analysis is a core capability for incident response. When a suspicious binary is found on a compromised system, analysis determines whether it is a known commodity malware, a custom tool, or — most importantly — whether it has exfiltrated data and how to block its C2 infrastructure.

Analysis Approaches

Malware analysis is divided into two complementary approaches:

The Analysis Decision Tree

Not every sample requires full reverse engineering. The analysis depth should match the threat:

Sample Collected
├── Is it already known? (VT lookup, hash databases)
│   ├── Yes → Check analysis reports; update detections
│   └── No → Proceed
├── Static analysis
│   ├── Identified as known family?
│   │   ├── Yes → Check family TTPs; update detections
│   │   └── No → Proceed
├── Dynamic analysis (sandbox)
│   ├── Clear C2, persistence, exfiltration observed?
│   │   ├── Yes → Block IOCs; update detections; done
│   │   └── No → Proceed to full reverse engineering
└── Reverse engineering (Ghidra/IDA)
    └── Full understanding of capability and intent

Static Analysis

1. File Type Identification

# Linux — identify file type
file suspicious_sample.bin
# Output: PE32 executable (GUI) Intel 80386, for MS Windows

# Windows — identify file type (Sysinternals)
sigcheck -a suspicious_sample.exe

# Check if the file is packed (UPX, themida, etc.)
# Packed files have few readable strings and high entropy

2. Hash Calculation and Lookup

# Calculate hashes
sha256sum suspicious_sample.exe
md5sum suspicious_sample.exe

# VirusTotal lookup via API
curl -X POST 'https://www.virustotal.com/api/v3/files/{hash}' \
  -H 'x-apikey: YOUR_API_KEY'

3. String Analysis

Strings reveal hardcoded configuration data, URLs, IPs, registry keys, and commands:

# Extract all printable strings
strings -n 8 suspicious_sample.exe > strings.txt

# Look for suspicious patterns
grep -i "http\|https\|ftp" strings.txt          # URLs
grep -i "\\\\" strings.txt                        # Network paths
grep -i "HKEY\|CurrentVersion\\Run" strings.txt   # Registry persistence
grep -i "C:\\\\(Users\\|Windows\\|Program)" strings.txt  # File paths
grep -i "Mutant\|Mutex\|Event" strings.txt         # Named objects
grep -i "CreateRemoteThread\|VirtualAllocEx\|WriteProcessMemory" strings.txt  # Injection APIs

Example strings output from a TrickBot sample:

TrickBot
config.bin
https://evil-c2.com/gate.php
user_agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:35.0) Gecko/20100101 Firefox/35.0
server_1: 185.130.5.213:443
server_2: 203.0.113.50:8443
group_tag: finance_2026
C:\Users\*\AppData\Roaming\TrickBot\

4. PE (Portable Executable) Analysis

# PE analysis with pecheck
pecheck suspicious_sample.exe

# PE analysis with pefile (Python)
python3 -c "
import pefile
pe = pefile.PE('suspicious_sample.exe')
print('Sections:')
for section in pe.sections:
    print(f'  {section.Name.decode().strip():<10} | VA: {section.VirtualAddress:08x} | Size: {section.SizeOfRawData:08x} | Entropy: {section.get_entropy():.2f}')
print('Imports:')
for dll in pe.DIRECTORY_ENTRY_IMPORT:
    print(f'  {dll.dll.decode()}')
    for imp in dll.imports:
        print(f'    {imp.name.decode() if imp.name else f\"Ordinal: {imp.ordinal}\"}')
"

Suspicious imports to look for:

YARA Rules

YARA (Yet Another Recursive Acronym) is a tool for creating and matching malware signatures based on textual or binary patterns. YARA rules are the industry standard for malware classification and detection.

YARA Rule Structure

rule Suspicious_CreateRemoteThread
{
    meta:
        description = "Detects samples using CreateRemoteThread for process injection"
        author = "Malware Analysis Team"
        date = "2026-01-16"
        reference = "https://attack.mitre.org/techniques/T1055/"
        severity = "high"

    strings:
        $s1 = "OpenProcess" fullword
        $s2 = "VirtualAllocEx" fullword
        $s3 = "WriteProcessMemory" fullword
        $s4 = "CreateRemoteThread" fullword
        $s5 = "NtUnmapViewOfSection" fullword

    condition:
        // Any 3 of the 5 injection APIs indicate malicious intent
        3 of ($s*)
}

YARA — Real-World Rule: TrickBot

rule TrickBot_Loader
{
    meta:
        description = "Detects TrickBot loader based on known strings and PE characteristics"
        author = "Malware Analysis Team"
        date = "2026-01-16"
        reference = "https://attack.mitre.org/software/S0266/"

    strings:
        $str1 = "TrickBot" wide ascii
        $str2 = "config.bin" wide
        $str3 = "/gate.php" wide
        $str4 = "group_tag" wide
        $str5 = "windows_update_" wide

    condition:
        (2 of ($str*) or $str1) and pe.imports("kernel32.dll", "CreateProcess") and pe.sections[0].name == ".text"
}

Running YARA

# Compile rules
yarac rules.yara rules.yarc

# Scan a single file
yara rules.yara suspicious_sample.exe

# Scan a directory recursively
yara -r rules.yara /evidence/malware/

# Scan with metadata output
yara -e rules.yara suspicious_sample.exe

Dynamic Analysis (Sandboxing)

Dynamic analysis executes the malware in a controlled, isolated environment to observe its behaviour. This is the most efficient way to understand what a malware sample actually does.

Sandbox Setup Requirements

Cuckoo Sandbox / CAPE

Cuckoo is the open-source standard for automated malware analysis. CAPE (Config And Payload Extraction) is its modern successor:

# Submit a sample to Cuckoo
cuckoo submit suspicious_sample.exe

# Submit with options
cuckoo submit --timeout 120 --enforce-timeout \
  --machine win10-vm \
  --options "procmem=yes,filewrite=yes" \
  suspicious_sample.exe

# Check analysis status
cuckoo status

ANY.RUN (Cloud Sandbox)

ANY.RUN is an interactive cloud-based sandbox that shows real-time analysis:

Key features:

• Live process tree with screen recording
• Network traffic viewer (HTTP, DNS, TCP/UDP)
• Registry and file system changes
• MITRE ATT&CK technique mapping
• YARA and Suricata rule matching
• IoC extraction (IPs, domains, hashes, mutexes)

Dynamic Analysis — What to Look For

Reverse Engineering

Reverse engineering disassembles the binary to understand its logic at the code level.

Ghidra (Free, Open-Source)

Ghidra is the NSA’s reverse engineering framework, available for free:

Ghidra Workflow:
1. Create a new project
2. Import the binary
3. Auto-analysis (Ghidra identifies functions, decompiles to C)
4. Navigate the decompiled code
5. Identify key functions (C2, encryption, installation)
6. Rename variables and functions for clarity
7. Extract configuration data (embedded C2 URLs, encryption keys)

Key features:

• Decompiler (produces readable C-like pseudocode)
• Multi-platform (Windows, Linux, macOS binaries)
• Scriptable (Python, Java, JavaScript)
• Collaborative (multi-user project support)
• Extensible (large plugin ecosystem)

IDA Pro (Commercial)

IDA Pro is the industry standard for professional reverse engineering:

• Superior decompiler (Hex-Rays)
• Extensive plugin ecosystem
• Debugger integration
• Type library support (FLIRT signatures)

Case Study: Analyzing a TrickBot Sample

TrickBot is a banking trojan that evolved into a modular crimeware platform used for initial access, credential theft, and ransomware delivery.

Static Analysis Findings

Dynamic Analysis Findings

MITRE ATT&CK Mapping

Key Takeaways

• Static analysis (file type, hashes, strings, PE headers, YARA) provides initial classification without executing the sample — always start here
• YARA rules enable automated malware detection and classification — write rules for samples you analyse so you can detect them across your environment
• Dynamic analysis (Cuckoo, ANY.RUN) reveals what the malware actually does: C2 communication, persistence, injection, data theft — this is the most efficient analysis method for IR
• Reverse engineering (Ghidra, IDA Pro) is required only for unknown, custom, or heavily obfuscated malware — do not spend days reversing a commodity trojan
• The TrickBot case study shows how static, dynamic, and reverse engineering analysis combine to build a complete picture of a malware sample’s capabilities
• Every malware analysis should produce detection artifacts: Sigma rules for SIEM, YARA rules for files, network signatures for NIDS, and MITRE ATT&CK mappings for IR