Why Emulation — The Missing Half of Detection Engineering

The most common failure mode in detection programs isn’t bad rules — it’s untested rules. Rules that look correct in YAML but fail silently in production because of a field mapping gap, a missing telemetry source, or a SIEM configuration mismatch. Adversary emulation closes this gap by running the actual attack behavior and checking if the alarm goes off.

The Validation Gap

Detection engineering has two halves:

Half 1: Author                Half 2: Validate
─────────────────────         ──────────────────────────────
Write Sigma rule         →    Run the attack in a test lab
Convert to SIEM query    →    Confirm the alert fires
Add to SIEM library      →    Verify telemetry is present
Write the ADS            →    Test FP exclusions don't suppress TPs

Most detection programs invest heavily in Half 1 and almost nothing in Half 2. The result: a detection library full of rules with unknown firing status. Some fire correctly. Some fire but with high FP rates. Some never fire at all — because the environment is missing a precondition the rule author assumed.

The Three Silent Failure Modes

Failure Mode 1: Missing Telemetry

A rule for Sysmon Event 10 (LSASS access) does nothing if:

Sysmon isn’t installed on the servers where credential dumping typically occurs
The Sysmon config excludes the event types the rule needs
The endpoint agent collects Sysmon events but only from workstations, not servers

Emulation test reveals this immediately: run the test on a server, check the SIEM — zero events. The gap is visible.

Failure Mode 2: Field Mapping Gap

A Sigma rule uses OriginalFileName to detect renamed binaries. The field is valid in the Sysmon spec. But:

Your Splunk pipeline maps Sysmon Event 1 fields but never added OriginalFileName to the mapping
sigma-cli silently drops unmapped fields during conversion
The converted SPL query checks Image but has no OriginalFileName condition
A renamed mimikatz.exe passes straight through

Emulation test reveals this: run the test, check the SIEM, find an alert — but inspect the query. The OriginalFileName condition is absent. Coverage is weaker than the rule implies.

Failure Mode 3: Logsource Mismatch

A rule targets logsource: category: process_creation, product: windows. Your SIEM indexes Sysmon data under index=sysmon but the pipeline converts the rule to query index=windows. Different index — no results.

Or: the rule was written for Sysmon but your environment uses the Windows Security provider (4688). Different event structure — conditions that reference Sysmon-specific fields like ProcessGuid return nothing.

Emulation vs Penetration Testing vs Red Teaming

	Emulation	Pen Test	Red Team
Goal	Validate detections	Find vulnerabilities	Test full defense
Who controls it	Detection engineer	Security team	Red team + blue team
Scope	Specific ATT&CK techniques	System attack surface	Full kill chain
Environment	Isolated test lab	Target network (scoped)	Production (authorized)
Output	Coverage matrix + detection gaps	Vuln report + remediation	TTX findings + MTTD
Risk	Very low (isolated)	Low-medium	Highest

Detection engineers run emulation — controlled, scoped, in a lab. Pen testing and red teaming are broader and require more organizational coordination. Don’t conflate them.

DetectionLab: A Ready-Made Test Environment

DetectionLab (github.com/clong/DetectionLab) provisions a complete Windows detection lab:

DC (dc.windomain.local)  ← Domain Controller, Windows Server 2019
WEF (wef.windomain.local) ← Windows Event Forwarder
Win10 (win10.windomain.local) ← Client workstation
Logger (logger.windomain.local) ← Splunk + ELK

Pre-installed on all Windows hosts:
  - Sysmon (SwiftOnSecurity config)
  - Winlogbeat (ships events to Logger)
  - Osquery
  - velociraptor (optional)

Pre-installed on Logger:
  - Splunk Enterprise (trial)
  - Elastic + Kibana

Build time: 30–60 minutes with Vagrant + VirtualBox.

Lighter alternative: Single Windows 10 VM + Sysmon (from sysmon.xml config) + Winlogbeat → free Elastic Cloud trial.

The Emulation Workflow

1. Choose technique
   → Pick ATT&CK technique to test (e.g., T1003.001)
   → Confirm you have a detection rule for it

2. Prepare the test environment
   → Verify Sysmon is running and generating events
   → Confirm events are flowing to the SIEM (check count)
   → Document the environment state

3. Run the emulation test
   → Execute the atomic test / CALDERA operation
   → Note exact execution time

4. Query the SIEM
   → Search for events in ±5 minute window around execution time
   → Check raw events: are the fields populated?
   → Check alert: did the detection rule fire?

5. Record the outcome
   → FIRE: rule fired correctly → mark as "Covered"
   → NO FIRE + events present: rule logic or mapping issue → mark as "Miscovered"
   → NO FIRE + no events: telemetry gap → mark as "Gap"

6. Remediate and retest
   → Fix the identified issue (deploy Sysmon, fix field mapping, fix rule logic)
   → Re-run the test to confirm the fix works

DetectionLab— GitHub Atomic Red Team— Red Canary

Retrieval Practice

A detection engineer says: 'We have Sigma rules covering all 50 techniques in our threat model.' A colleague responds: 'That doesn't mean much without emulation results.' Who is right, and what specific evidence would change the engineer's claim into a defensible statement?

Reveal answer

The colleague is right. 'We have rules' is a claim about the detection library, not about detection capability. The same claim expressed as a defensible statement: 'We have rules covering all 50 techniques, and we have emulation test results confirming 43 of those rules fire correctly in production. The remaining 7 have identified gaps: 3 are telemetry gaps (Sysmon not on servers), 2 are field mapping issues, 2 are untested.' The defensible version includes: which rules fire (tested), which don't (tested and failed), and why the gaps exist. This is the output of an emulation program — it converts 'we have rules' into 'we have measured coverage.'

Retrieval Practice

You run an Atomic Red Team test for T1059.001 on a Windows 10 workstation. The test completes. You check the SIEM — no alert, but you can see the powershell.exe process creation event in the raw logs. What failure mode is this, and what are the two most likely root causes?

Reveal answer

This is Failure Mode 2 (field mapping gap) or a rule logic issue — the telemetry IS present (process creation event visible) but the rule didn't fire. Two most likely causes: (1) Field mapping gap: the Sigma rule uses a field (e.g., CommandLine) that wasn't mapped by the SIEM pipeline. The converted query references a non-existent field name that returns no results, even though the data is present under a different field name. Check the converted query field names against the actual event field names in the raw log. (2) Rule condition mismatch: the CommandLine value in the raw event doesn't match the expected pattern in the rule — e.g., the test uses -ec (abbreviation) but the rule only checks for -EncodedCommand (full flag). Review the actual CommandLine value in the raw event against every condition in the rule.