Precision, Recall, and Robustness — The Detection Quality Triangle

A detection rule is a binary classifier: for every event, it decides “alert” or “no alert.” Two classic machine-learning metrics — precision and recall — measure how good that classifier is. But in adversarial settings, a third dimension matters just as much: robustness. This lesson defines all three, shows how they interact, and builds the vocabulary you’ll use throughout the rest of this module.

Precision: How Often Are Your Alerts Real?

Precision = TP / (TP + FP)

A rule with 100 alerts per day — 20 real threats, 80 noise — has 20% precision. Analysts spend 80% of their time on alerts that lead nowhere.

What drives low precision:

Broad pattern matching (CommandLine|contains: 'powershell' matches everything)
Lack of contextual conditions (no parent process, no user filter)
Missing false-positive filters for known-legitimate behavior

What drives high precision:

Multi-condition rules (process + parent + command line + user)
Targeted scope (specific system paths, specific event patterns)
Tested FP filters based on baseline data from your environment

Recall: How Many Attacks Are You Catching?

Recall = TP / (TP + FN)

A rule that fires on net.exe user /add has 0% recall for attackers using net1.exe, dsadd, the Active Directory Users GUI, or direct LDAP writes. All four are valid ways to add a user — none trigger the rule.

What drives low recall:

Tool-specific patterns (only catches specific binary names)
Exact-string matching on arguments that have many equivalent forms
Logsource gaps (rule targets Sysmon Event 1, but Sysmon isn’t deployed everywhere)

What drives high recall:

Behavioral patterns (detect the action, not the tool)
Multiple detection variants covering different techniques for the same goal
Coverage across data sources (process creation + script block logging + event 4104)

Robustness: Does the Rule Survive an Attacker Who Knows It Exists?

Precision and recall describe how the rule performs against a passive dataset. Robustness describes how it performs when the attacker actively tries to evade it.

Robustness = (steps required to evade the detection)

This is Summiting the Pyramid in practice — L0–L4.

Level	What the rule targets	Evasion cost
L0	Atomic indicator (hash, IP)	Seconds
L1	Specific tool variant (filename, mutex)	Minutes
L2	Common capabilities (specific function calls)	Hours
L3	Behavioral indicator (actions regardless of tool)	Days to weeks
L4	Implementation-agnostic (OS primitives, kernel-level)	Months

The net.exe example has low recall AND low robustness (L1). An attacker running T1136.001 through any other binary path — or renaming net.exe — evades it completely.

The Triangle: You Can’t Maximize All Three Simultaneously

         Precision
            /\
           /  \
          /    \
         /      \
        /________\
    Recall      Robustness

The three dimensions pull in different directions:

Increase recall → often decreases precision: casting a wider net catches more attacks but also more legitimate activity.
Increase robustness → often decreases precision: behavioral rules (detect any process opening LSASS with high-privilege access rights) fire for legitimate tools (Windows Defender, crash reporters) that share the same behavior.
Increase precision → often decreases recall and robustness: narrow patterns miss attack variants and evade easily.

There is no perfect rule. Every detection is a deliberate trade-off. The engineering discipline is knowing which corner of the triangle to sacrifice for your specific threat model and analyst capacity.

Working Example: Scoring a Net User Rule

detection:
  selection:
    Image|endswith: '\net.exe'
    CommandLine|contains: 'user /add'
  condition: selection

Dimension	Score	Why
Precision	Medium	`net.exe user /add` runs legitimately in enterprise IT; domains provisioning new staff will trigger this
Recall	Low	Misses net1.exe, dsadd, PowerShell AD cmdlets, direct LDAP, GUI-based account creation
Robustness	L1	Rename net.exe or use any alternative tool → complete evasion

Improving recall without destroying precision:

detection:
  selection_net:
    Image|endswith:
      - '\net.exe'
      - '\net1.exe'
    CommandLine|contains: 'user /add'
  
  selection_dsadd:
    Image|endswith: '\dsadd.exe'
    CommandLine|contains: 'user'
  
  condition: 1 of selection_*

This raises recall (catches net1.exe and dsadd) but still misses PowerShell and GUI paths. Robustness remains L1-L2.

Improving robustness (at precision cost):

logsource:
  product: windows
  category: user_management  # Sysmon Event 4720 or Security Event 4720

detection:
  selection:
    EventID: 4720            # A user account was created
  filter_expected:
    SubjectUserName|endswith: '$'   # machine accounts
  condition: selection and not filter_expected

This targets the outcome (account created, Event 4720) regardless of what tool was used. Recall and robustness jump to L3+. Precision drops: 4720 fires for all legitimate IT provisioning — you need additional context (time of day, who created it, domain vs local) to separate malicious from legitimate.

On Detection: Why Detections Fail— SpecterOps Summiting the Pyramid— CTID / MITRE Engenuity

Retrieval Practice

You write a Sigma rule for ransomware that detects: `CommandLine|contains|all: ['vssadmin', 'delete', 'shadows']`. An attacker uses: `wmic shadowcopy delete`. Does the rule fire? Is this a recall failure or a robustness failure?

Reveal answer

The rule does not fire — wmic.exe is a completely different binary with different command syntax. This is a recall failure: the rule only covers one tool for one technique (T1490 — Inhibit System Recovery). The attacker doesn't need to evade the rule; they simply use a different built-in Windows tool that achieves the same result. A robustness failure would be: the rule targets vssadmin.exe and the attacker renames vssadmin.exe to evade the filename check. Recall failure = missed technique variants. Robustness failure = defeated by evasion within the same tool path.