Malware Analysis 101: Examine Malicious Files

Why Analyze Malware?

Malware analysis is the process of dissecting malicious software to understand what it does, how it spreads, what it communicates with, and what damage it can cause. This information feeds directly into incident response (what systems are affected?), threat intelligence (is this a known actor?), and detection engineering (how do we write a signature to catch this in the future?).

You don't need to reverse-engineer every byte to produce valuable analysis. Knowing how to safely run a sample, observe its behavior, and extract indicators of compromise (IOCs) — domains, IPs, file hashes, registry keys — is a practical skill that non-specialists can develop.

Setting Up a Safe Analysis Environment

This is the most critical step. Never analyze malware on your primary machine or production network. A proper isolation setup requires:

• A dedicated virtual machine (VMware or VirtualBox) with snapshots enabled. Always revert to a clean snapshot after each analysis session.
• The VM's network adapter set to host-only or isolated network mode. Never bridge it to your real LAN.
• A separate analysis OS — Windows 10/11 is ideal since most malware targets Windows. Use a licensed copy specifically for analysis.
• Disable shared folders and clipboard sharing between host and guest — some malware actively looks for these channels to escape the VM.
• Consider a dedicated analysis machine that is physically isolated if you're working with particularly sophisticated samples.

Tools like REMnux (a Linux-based analysis distro) and FLARE VM (a Windows analysis environment by Mandiant) come pre-loaded with analysis tools and are the fastest way to get started.

Static Analysis: Examining Without Executing

Static analysis examines the malware file without running it. Start here — it's safe and often reveals a great deal.

File Identification and Hashing

Before anything else, compute the file's hash:

sha256sum suspicious_file.exe

Search the hash on VirusTotal, Malware Bazaar, or Hybrid Analysis. If it's a known sample, you'll immediately get family name, behavior reports, and prior analysis. The hash also serves as your primary IOC for detection rules.

Use file on Linux or PEiD/Detect-It-Easy on Windows to identify the file type. Malware frequently uses misleading extensions — a .pdf might actually be a PE executable.

String Extraction

Running strings against a binary extracts all printable character sequences. You'll often find:

• Hardcoded domain names and IP addresses (C2 infrastructure)
• File paths and registry keys the malware interacts with
• Error messages that reveal developer language/environment
• Encryption keys or configuration data

strings -n 8 malware.exe | grep -Ei 'http|cmd|reg|powershell'

PE Header Analysis

For Windows executables, tools like PE-bear, CFF Explorer, or Python's pefile library reveal the import table (which Windows API functions the binary calls), section names and entropy (high entropy suggests packing/encryption), compilation timestamps, and embedded resources. A binary importing CreateRemoteThread, VirtualAllocEx, and WriteProcessMemory is almost certainly doing process injection.

Dynamic Analysis: Watching Malware Run

Dynamic analysis executes the sample in your isolated VM while monitoring its behavior. Set up monitors before running the sample:

• Process Monitor (Procmon): logs every file, registry, and network event by process
• Wireshark: captures all network traffic
• Process Hacker: shows running processes, injected threads, loaded modules
• Regshot: takes registry snapshots before and after execution for diff comparison

Run the sample, wait 2–5 minutes, then review what you've captured. Key questions:

1. What processes did it create or inject into?
2. What files did it create, modify, or delete?
3. What registry keys did it set (persistence mechanisms)?
4. What network connections did it attempt — domains, IPs, ports, protocols?
5. Did it attempt to disable security tools or modify the Windows Firewall?

Automated Sandboxes

For quick triage, submit the sample to an automated sandbox:

• Any.run (interactive, browser-based)
• Hybrid Analysis (free, powered by CrowdStrike Falcon Sandbox)
• Joe Sandbox (detailed reports)
• Cuckoo Sandbox (self-hosted, open-source)

Sandboxes automate the dynamic analysis workflow and produce structured reports with IOCs, MITRE ATT&CK mappings, and network captures — often in minutes. Their limitation is that sophisticated malware actively detects sandbox environments (checking for unusual usernames, minimal uptime, VM artifacts) and stays dormant to evade detection.

Extracting Indicators of Compromise

The practical output of malware analysis is a set of IOCs for detection and blocking:

• File hashes (MD5, SHA-256) for AV/EDR signatures
• Domain names and IPs for DNS sinkholes and firewall blocks
• URLs for web proxy blocking
• Registry keys for detection rules
• Mutex names (malware often creates mutexes to avoid double-infection — these make excellent IOCs)
• YARA rules for pattern-based detection across file collections

Document everything in a structured format (STIX/TAXII if sharing with the community) and share findings on platforms like MISP or OpenCTI to benefit the broader security community.