Skip to content

Malware Analysis

Malware analysis is the process of examining suspicious files or behavior to understand what the malware is, what it does, how it persists, how it communicates, and how to detect it.

Malware analysis usually combines static analysis, dynamic analysis, reverse engineering, and detection engineering.

Analysis Goals

Malware analysis helps answer:

  • What type of file is this?
  • Is it packed or obfuscated?
  • What capabilities does it have?
  • Does it persist?
  • Does it contact C2 infrastructure?
  • Does it inject into other processes?
  • What files, registry keys, or services does it create?
  • What indicators can we turn into detections?

Analysis Types

Type Description Risk
Static analysis Analyze without running the sample Lower
Dynamic analysis Execute in a controlled lab and observe behavior Higher
Reverse engineering Inspect disassembly or debug the program Medium to high
Detection engineering Convert findings into YARA/Sigma/SIEM rules Low

Windows Internals for Malware Analysis

To analyze Windows malware, understand the environment it runs in.

Windows separates execution into:

Mode Description Malware Relevance
User mode Restricted mode where applications run Most malware starts here
Kernel mode Privileged mode where drivers and the OS kernel run Rootkits and advanced malware may target this

User Mode

User mode includes:

  • User applications
  • Browsers
  • Office applications
  • Service processes
  • Logon-related processes
  • Malware running with normal user privileges

User-mode malware commonly interacts with:

  • Files
  • Registry keys
  • Processes
  • Services
  • Network APIs
  • User credentials available to the session

Kernel Mode

Kernel mode includes:

  • Windows kernel
  • Device drivers
  • Hardware Abstraction Layer
  • Kernel-level security components

Kernel-mode malware is dangerous because it can:

  • Hide files
  • Hide processes
  • Intercept security tools
  • Manipulate memory
  • Resist removal
  • Tamper with system behavior

Windows API Call Flow

User-mode programs request privileged actions through Windows APIs.

Example flow for a memory-related API:

Malware calls Windows API
  -> kernel32.dll
  -> ntdll.dll
  -> syscall instruction
  -> kernel mode
  -> SSDT lookup
  -> permission checks
  -> result returned to user mode

Important components:

Component Purpose
kernel32.dll Common Windows API layer
ntdll.dll Low-level interface to syscalls
syscall CPU transition from user mode to kernel mode
SSDT System Service Descriptor Table
Kernel Performs privileged operation

Portable Executable Format

Windows executables and DLLs commonly use the PE format.

Common file types:

.exe
.dll
.sys
.scr

PE Sections

Section Purpose Analysis Value
.text Executable code Main program logic
.data Writable variables Runtime data
.rdata Read-only data Strings and constants
.rsrc Resources Icons, dialogs, embedded payloads
.idata Imports API capabilities
.edata Exports DLL-provided functions
.reloc Relocation data Memory loading support

Process Anatomy

A process is a container for execution.

Important components:

Component Description
PID Process ID
Virtual Address Space Process memory
Threads Actual execution units
Handles References to files, registry keys, processes, etc.
Access Token Security context and privileges

Imports and Exports

Imports show which external functions a program uses.

Examples:

Import Possible Meaning
CreateFile File creation/access
WriteFile Write data to disk
InternetOpenUrl Network download
RegOpenKeyEx Registry access
VirtualAllocEx Allocate memory in another process
WriteProcessMemory Write into another process
CreateRemoteThread Execute code in another process

Exports are functions a DLL makes available to other programs.

Process Injection Indicators

A common process injection pattern:

OpenProcess
  -> VirtualAllocEx
  -> WriteProcessMemory
  -> CreateRemoteThread

This may indicate one process is writing code into another process and forcing it to execute.

Common injection targets:

  • notepad.exe
  • explorer.exe
  • svchost.exe
  • Browser processes
  • Office processes

Static Analysis

Static analysis means examining a sample without executing it.

Common goals:

  • Identify file type
  • Calculate hashes
  • Extract strings
  • Inspect imports
  • Inspect PE sections
  • Detect packing
  • Extract embedded files
  • Identify possible capabilities

Static Analysis on Linux

Identify File Type

Use file:

file sample.exe

Example output:

PE32 executable (GUI) Intel 80386, for MS Windows

Verify Header

Windows PE files begin with MZ.

hexdump -C sample.exe | head

Look for:

4d 5a
MZ

Hashing

Hashes identify exact files.

md5sum sample.exe
sha256sum sample.exe
Hash Use
MD5 Legacy lookup
SHA256 Preferred IOC hash

Similarity Hashing

Standard hashes change completely when one byte changes. Similarity hashing helps identify related samples.

Method Purpose
IMPHASH Hashes imported APIs
SSDEEP Fuzzy file similarity
Section hash Hashes individual PE sections

SSDEEP example:

ssdeep sample.exe

String Analysis

Extract readable strings:

strings -n 15 sample.exe

Look for:

  • URLs
  • IP addresses
  • Domains
  • File paths
  • Registry paths
  • PDB paths
  • Error messages
  • User-Agent strings
  • Commands

FLOSS

FLOSS extracts obfuscated strings and stack strings.

floss sample.exe

Use FLOSS when basic strings produces limited output.

Packing

Packing compresses or encrypts malware to make static analysis harder.

Signs of packing:

  • Few imports
  • High entropy
  • Section names like UPX0, UPX1
  • Very few readable strings
  • Suspicious section permissions

UPX unpacking:

upx -d -o unpacked_sample.exe packed_sample.exe

After unpacking, repeat:

strings -n 15 unpacked_sample.exe

Static Analysis on Windows

Useful Windows tools:

Tool Purpose
PowerShell Get-FileHash File hashing
Sysinternals strings String extraction
FLOSS Obfuscated string extraction
UPX Unpacking UPX-packed files
CFF Explorer PE structure inspection
PeStudio Static triage
Hex editor Raw byte inspection

Windows Hashing

Get-FileHash -Algorithm MD5 .\sample.exe
Get-FileHash -Algorithm SHA256 .\sample.exe

Windows Strings

strings.exe sample.exe

FLOSS:

floss.exe sample.exe

Dynamic Analysis

Dynamic analysis means executing malware in a controlled environment and observing behavior.

It can reveal:

  • Created files
  • Deleted files
  • Registry changes
  • Network connections
  • Child processes
  • Persistence
  • Sandbox checks
  • Payload downloads
  • Process injection
  • C2 behavior

Dynamic Analysis Lab

Use an isolated lab.

Recommended setup:

  • Virtual machine
  • No connection to production network
  • Snapshot before execution
  • Fake internet services if needed
  • Monitoring tools running before execution
  • Safe sample handling process

Danger

Never run malware on your host machine or production network.

Dynamic Analysis Workflow

Phase Action
Environment setup Prepare isolated VM
Baseline Take snapshot and record clean state
Tool deployment Start monitoring tools
Execution Run sample
Observation Let behavior occur
Collection Save logs and captures
Analysis Review artifacts
Reset Revert VM snapshot

Dynamic Analysis Tools

Tool Purpose
ProcMon File, registry, process activity
Noriben Filters ProcMon output
Wireshark Network traffic
INetSim Fake internet services
Regshot Registry before/after comparison
ProcDot Visualize ProcMon activity
Cuckoo Sandbox Automated sandbox
Joe Sandbox Automated sandbox

Noriben

Noriben is a Python wrapper around Sysinternals ProcMon.

It:

  • Starts ProcMon
  • Captures activity
  • Filters common system noise
  • Produces a readable report

Basic workflow:

python Noriben.py

Then:

  1. Run the malware.
  2. Wait for behavior.
  3. Press CTRL+C in Noriben.
  4. Review the generated report.

Noriben reports may include:

  • File activity
  • Registry activity
  • Process activity
  • Network activity

Noriben Caveat

Noriben filters noise, but filtering can hide important details.

If malware exits immediately or appears inactive:

  • Review raw ProcMon logs
  • Check for sandbox evasion
  • Check registry/environment queries
  • Check failed network checks
  • Check parent process checks
  • Check VM artifacts

Automated Sandboxes

Automated sandboxes execute malware and generate reports.

Benefits:

  • Fast triage
  • Repeatable execution
  • Behavior summaries
  • Screenshots
  • Network indicators
  • Dropped file extraction

Limitations:

  • Malware may detect sandboxing
  • Behavior may differ from real systems
  • Network simulation may be incomplete
  • Advanced samples may sleep or evade

Reverse Engineering

Reverse engineering analyzes compiled code when source code is unavailable.

It helps uncover:

  • Hidden logic
  • Control flow
  • Encryption routines
  • Obfuscated strings
  • Sandbox evasion
  • Process injection
  • C2 protocol details
  • Configuration extraction

Reverse Engineering Tools

Tool Type Examples Use
Disassembler IDA, Ghidra, Cutter Static code analysis
Debugger x64dbg, OllyDbg, IDA debugger Step through execution
Decompiler Ghidra, IDA Hex-Rays Pseudocode recovery

IDA Basics

Common IDA views:

View Purpose
Graph View Visual control flow
Text View Linear disassembly

Shortcuts:

Shortcut Action
Space Toggle graph/text view
Enter Jump into function or reference
Esc Go back
N Rename function or variable

Assembly Indicators

Pattern / API Meaning
test eax, eax + jz Check if return value is zero
lea Load address of string or structure
RegOpenKeyExA Registry access
InternetOpenUrlA URL access/download
CreateFileA File creation/access
WriteFile File write
CreateProcessA Process creation
VirtualAllocEx Remote memory allocation
WriteProcessMemory Remote memory write
CreateRemoteThread Process injection indicator

Common Malware Behaviors

Behavior Possible Indicator
Persistence Run keys, services, scheduled tasks
Defense evasion Sandbox checks, packed code
Credential access LSASS access
Discovery whoami, ipconfig, net commands
C2 Periodic outbound connections
Payload download HTTP request to executable
Injection Remote thread creation
Exfiltration Archive creation then upload

Detection Engineering After Analysis

After analysis, create detections.

Finding Detection Type
Unique file strings YARA
PE structure YARA
Malware behavior Sigma
Network domain/IP SIEM/DNS/Proxy rule
Process chain Sigma/SIEM
File drop path Sigma/Sysmon
Registry persistence Sigma/Sysmon
C2 pattern IDS/SIEM

YARA for Malware Files

Example:

rule Shell_Sandbox_Detection
{
    strings:
        $sandbox_string = "Sandbox detected"

    condition:
        $sandbox_string
}

A stronger rule should include multiple unique strings and file constraints.

Example:

rule Suspicious_Dropper
{
    strings:
        $s1 = "Sandbox detected"
        $s2 = "Connection sent to C2"
        $s3 = "WindowsUpdater"
        $s4 = "http://ms-windows-update.com/svchost.exe"

    condition:
        uint16(0) == 0x5A4D and 2 of them
}

yarGen

yarGen can generate candidate YARA rules from samples.

Workflow:

mkdir Test
cp sample.exe Test/
sudo python3 yarGen.py -m ./Test/

Then review:

yargen_rules.yar

Always review generated rules before using them.

Sigma for Malware Behavior

Sigma detects behavior in logs.

Example: suspicious file creation in temp:

title: Suspicious File Drop In User Temp
status: experimental
description: Detects suspicious file creation in a user temp location
logsource:
  product: windows
  category: file_event
detection:
  selection:
    TargetFilename|contains: '\AppData\Local\Temp\'
    TargetFilename|endswith: '\svchost.exe'
  condition: selection
level: high

Useful Sysmon events:

Event ID Meaning
1 Process creation
3 Network connection
7 Image loaded
10 Process access
11 File creation
12/13/14 Registry activity

Malware Analysis Cheat Sheet

Task Tool / Command
Identify file file sample.exe
View header hexdump -C sample.exe \| head
SHA256 hash sha256sum sample.exe
Windows hash Get-FileHash -Algorithm SHA256 .\sample.exe
Fuzzy hash ssdeep sample.exe
Extract strings strings -n 15 sample.exe
Decode strings floss sample.exe
Unpack UPX upx -d sample.exe
Dynamic behavior ProcMon / Noriben
Network behavior Wireshark
Reverse engineering IDA / Ghidra
File detection YARA
Behavior detection Sigma

Notes to Remember

  • Static analysis is safer but may miss runtime behavior.
  • Dynamic analysis reveals behavior but requires isolation.
  • Packed malware should be unpacked before deeper static analysis.
  • Imports reveal possible capabilities.
  • Strings are useful but easy for attackers to change.
  • Process injection often involves VirtualAllocEx, WriteProcessMemory, and CreateRemoteThread.
  • Detection rules should be tested and tuned before production.