Testing Detection Rules — Unit, Integration, and Regression

A detection rule without tests is a claim about behavior that no one has verified. Testing gives you two guarantees: the rule fires for real attacks, and the rule doesn’t fire for known-legitimate events. Without both guarantees, you’re deploying hope, not security.

The Three Testing Levels

Level 1: Unit Test — Does the rule convert?

# Validate Sigma YAML syntax and semantics
sigma check rules/windows/process_creation/proc_creation_powershell_encoded_cmd.yml

# Output (success):
# [OK] rules/windows/process_creation/proc_creation_powershell_encoded_cmd.yml

# Output (failure):
# [WARN] proc_creation_powershell_encoded_cmd.yml: field 'OriginalFileName' not in
#         recommended field list for logsource windows/process_creation
# [ERROR] proc_creation_powershell_encoded_cmd.yml: condition references undefined
#         selection 'selection_encoded' (defined: selection_img, filter_sccm)

# Validate conversion succeeds for all target backends
sigma convert -t splunk -p sysmon \
  rules/windows/process_creation/proc_creation_powershell_encoded_cmd.yml
sigma convert -t elasticsearch -p ecs-windows \
  rules/windows/process_creation/proc_creation_powershell_encoded_cmd.yml
sigma convert -t microsoft365defender \
  rules/windows/process_creation/proc_creation_powershell_encoded_cmd.yml

Conversion failures indicate: field not in pipeline mapping, modifier not supported by backend, or unsupported condition syntax.

Level 2: Integration Test — Does the rule fire for attacks?

Run the converted query against a known-TP dataset:

# test_rule.py (simplified)
import json
import subprocess

def test_rule_fires_on_tp():
    # Load TP fixture
    with open("tests/windows/process_creation/proc_creation_powershell_encoded_cmd-tp.json") as f:
        tp_events = json.load(f)
    
    # Convert rule
    result = subprocess.run(
        ["sigma", "convert", "-t", "test-backend", 
         "rules/windows/process_creation/proc_creation_powershell_encoded_cmd.yml"],
        capture_output=True, text=True
    )
    query = result.stdout
    
    # Run query against TP events (using test-backend evaluation)
    matches = evaluate_query(query, tp_events)
    assert len(matches) >= 1, f"Rule should match TP events, got {len(matches)} matches"

def test_rule_excludes_fp():
    # Load FP fixture
    with open("tests/windows/process_creation/proc_creation_powershell_encoded_cmd-fp.json") as f:
        fp_events = json.load(f)
    
    query = convert_rule(...)
    matches = evaluate_query(query, fp_events)
    assert len(matches) == 0, f"Rule should not match FP events, got {len(matches)} matches"

Level 3: Regression Test — Still fires after changes?

After any rule modification (adding exclusion, changing condition), run the TP test again to confirm the modification didn’t accidentally suppress TPs:

# Run the full test suite after modifying a rule
python -m pytest tests/ -v -k "powershell_encoded"

# Output:
# PASSED test_rule_fires_on_tp[proc_creation_powershell_encoded_cmd]
# PASSED test_rule_excludes_fp[proc_creation_powershell_encoded_cmd]

If the TP test starts failing after a rule change — the change is a regression.

Writing TP and FP Fixtures

Where to get TP data

Source	Quality	Effort
Atomic Red Team test output (saved from your lab)	High	Low — run the test, save the event
EVTX-ATTACK-SAMPLES (sbousseaden GitHub repo)	High	Low — pre-built EVTX files
Mordor dataset (Roberto Rodriguez)	Very high	Low — JSON format, pre-labeled
Synthetic (hand-crafted)	Medium	Medium — craft the event fields manually

Preferred: Run the Atomic Red Team test in your lab, capture the Sysmon Event from your SIEM as JSON, store it as the TP fixture. This is the highest-fidelity test because it’s actual telemetry from the real attack.

# Capture Sysmon event from Splunk after atomic test
| search index=sysmon EventCode=1 Image="*powershell*" earliest="-5m"
| table _time, EventCode, Image, CommandLine, ParentImage, User
| outputlookup tp_fixture.csv

Writing a synthetic TP fixture

When you don’t have real attack data, craft a minimal fixture with exactly the fields the rule checks:

// proc_creation_powershell_encoded_cmd-tp.json
[
  {
    "EventID": 1,
    "Image": "C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe",
    "CommandLine": "powershell.exe -EncodedCommand SGVsbG8gV29ybGQ=",
    "ParentImage": "C:\\Windows\\explorer.exe",
    "User": "CORP\\testuser",
    "ComputerName": "WIN10-TEST"
  },
  {
    "EventID": 1,
    "Image": "C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe",
    "CommandLine": "powershell.exe -enc SGVsbG8gV29ybGQ=",
    "ParentImage": "C:\\Windows\\explorer.exe",
    "User": "CORP\\testuser",
    "ComputerName": "WIN10-TEST"
  }
]

Include multiple variants (full flag + abbreviated) to test recall within the fixture.

Writing FP fixtures

FP fixtures should represent the actual known-legitimate events that previously generated FPs:

// proc_creation_powershell_encoded_cmd-fp.json
[
  {
    "EventID": 1,
    "Image": "C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe",
    "CommandLine": "powershell.exe -ExecutionPolicy Bypass -NonInteractive -File C:\\SCCM\\deploy.ps1",
    "ParentImage": "C:\\Windows\\CCM\\CcmExec.exe",
    "User": "SYSTEM",
    "ComputerName": "WIN10-TEST"
  }
]

This tests that the filter_sccm exclusion works — the SCCM parent triggers the exclusion and no alert fires.

The sigma-testing Framework

sigma-testing provides a test manifest format that wraps the process above:

# tests/proc_creation_powershell_encoded_cmd.test.yml
title: PowerShell Encoded Command Test Cases
rules:
  - rules/windows/process_creation/proc_creation_powershell_encoded_cmd.yml

tests:
  - title: Encoded PowerShell execution (full flag)
    rule: proc_creation_powershell_encoded_cmd.yml
    result: positive
    event:
      EventID: 1
      Image: 'C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe'
      CommandLine: 'powershell.exe -EncodedCommand SGVsbG8='
      ParentImage: 'C:\Windows\explorer.exe'

  - title: Encoded PowerShell execution (abbreviated flag)
    rule: proc_creation_powershell_encoded_cmd.yml
    result: positive
    event:
      CommandLine: 'powershell.exe -enc SGVsbG8='

  - title: Normal PowerShell (should not fire)
    rule: proc_creation_powershell_encoded_cmd.yml
    result: negative
    event:
      CommandLine: 'powershell.exe -Command "Write-Host hello"'

  - title: SCCM PowerShell deployment (filtered)
    rule: proc_creation_powershell_encoded_cmd.yml
    result: negative
    event:
      CommandLine: 'powershell.exe -enc SGVsbG8='
      ParentImage: 'C:\Windows\CCM\CcmExec.exe'

Run with:

sigma test tests/proc_creation_powershell_encoded_cmd.test.yml

sigma-cli— SigmaHQ EVTX-ATTACK-SAMPLES— GitHub Mordor / Security Datasets— OTRF

Retrieval Practice

A CI pipeline runs `sigma check` and `sigma convert` on every PR. A rule passes both checks. An analyst then deploys it and finds that it never fires in production. What is the gap in the CI pipeline, and what would catch this?

Reveal answer

sigma check and sigma convert are syntax-level tests — they verify the YAML is valid and conversion succeeds, but they don't verify the converted query produces correct results against real data. The gap: no integration test was run. The rule could have: (1) a field that converts successfully but maps to a non-existent field name in the production SIEM; (2) a condition that's syntactically correct but logically wrong (e.g., AND instead of OR for alternatives); (3) an exclusion that accidentally suppresses all events. What would catch this: an integration test that runs the converted query against a known-TP fixture and asserts at least one result. If the field mapping is wrong, the converted query returns zero results from the fixture, and the CI pipeline fails before merge.

Retrieval Practice

You add an exclusion to suppress FPs from a monitoring agent: `filter_agent: ParentImage|contains: '\\Monitor\\agent.exe'`. Your FP test passes (the excluded event no longer matches). Your TP test also passes. However, after deployment, a real attack using a renamed binary at `C:\\Monitor\\evil.exe` is missed. What was wrong with the FP fixture?

Reveal answer

The FP fixture tested the exclusion working correctly for the legitimate case, but didn't test the edge case where the exclusion path could be exploited. The fixture should include a 'near-miss' event: the same attack pattern (encoded PowerShell in CommandLine) but from a process at C:\\Monitor\\evil.exe (a different binary in the same directory). The exclusion uses |contains: '\\Monitor\\' which would match both the legitimate agent.exe AND any other binary in that directory. A FP fixture with evil.exe in the Monitor path would have caught this: the exclusion incorrectly suppresses it (fixture event returns no results, but it should have returned 1). Fix: tighten the exclusion to the full binary path: `ParentImage: 'C:\\Program Files\\Monitor\\agent.exe'`.