Hypothesis Failure Modes — Why Hunts Miss and How to Fix Them

A hunt that misses the adversary is worse than no hunt — it creates false confidence. Knowing the failure modes is as important as knowing the hunting techniques. Every missed hunt has a root cause; most root causes are in the hypothesis or the data source, not in the analysis.

The Taxonomy of Hunt Failure Modes

Hunt Failure
├── Hypothesis failure
│   ├── Too broad (candidate flood)
│   ├── Too narrow (excludes valid attacker variations)
│   ├── Unfalsifiable (no success/failure criteria)
│   ├── Premature closure (stopped before exhausting evidence)
│   └── Wrong TTP mapping (hunting for what the attacker didn't do)
│
├── Data source failure
│   ├── Log source not available
│   ├── Log source sampled / low fidelity
│   ├── Insufficient retention (time window exceeds data age)
│   └── Field not populated (e.g., command line logging disabled)
│
└── Analysis failure
    ├── Threshold overconfidence (treating 0.31 as definitively benign)
    ├── Scope creep (hypothesis drift)
    ├── Confirmation bias (looking for evidence to support, not refute)
    └── False negative acceptance (null result without coverage statement)

Failure Mode 1: Too Broad

Hypothesis (too broad):
"Are there any PowerShell commands in our environment that could be malicious?"

Results: 847,000 events in 30 days
Analyst can triage: ~200 events/day
Time to clear: ~4,200 analyst-days

Problem: This is not a hypothesis — it's a data dump with an implied question.

Fix — Progressive narrowing:

Round 1 (scope reduction):
"PowerShell spawned by Office applications (Word, Excel, Outlook) in past 30 days"
→ Results: 3 events
→ Triageable in < 1 hour

Round 2 (if Round 1 is clean, widen hypothesis):  
"PowerShell spawned by any user application not in approved parent list"
→ Results: 47 events
→ Triageable in 4 hours

Round 3 (statistical filter):
"PowerShell commands > 500 characters OR containing -enc flag"
→ Apply across all parent processes
→ Results: 23 events
→ Triageable in 2 hours

Narrowest specific hypothesis first. Widen only if clean.

Failure Mode 2: Too Narrow / Wrong Variation

Hypothesis:
"Alert on PowerShell with -encodedcommand flag (exact string match)"

Attacker command: powershell -enc [base64]

Hunt misses: powershell /encodedcommand [base64]
             powershell -e [base64]
             powershell -EnC [base64]  (case insensitive)
             powershell.exe -ENCODEDCOMMAND [base64]

PowerShell accepts abbreviated flag forms (-e, -en, -enc, -encod…) and the parameter names are case-insensitive. An exact string match hypothesis misses 80% of real usage.

Fix: Use regex matching for known abbreviations:

| where CommandLine matches regex @"(?i)-e(n(c(o(d(e(d(c(o(m(m(a(n(d)?)?)?)?)?)?)?)?)?)?)?)? "
// Or simpler: check for the base64 pattern (long alphanum string)
| where CommandLine matches regex @"-[eE]\S* [A-Za-z0-9+/=]{50,}"

Failure Mode 3: Wrong Data Source

Example scenario: Hunt for certutil download execution.

Hypothesis: "certutil.exe with -urlcache flag appears in Windows Security Event Log"

Hunt result: 0 events

Actual state: certutil.exe ran 14 times in 30 days
Problem: Windows Security Event Log 4688 (process creation) requires audit 
         policy "Process Creation" with command line logging enabled.
         This environment has process creation auditing enabled but command 
         line capture disabled → Event 4688 shows certutil.exe ran but shows
         CommandLine: "(null)" — the -urlcache flag is invisible.

Data source validation before hunting:

Before hypothesis:
1. Check: Is Event 4688 present with non-null CommandLine? 
   → query: SecurityEvent | where EventID == 4688 | where isnotempty(CommandLine) | take 1
   → If 0 results: command line logging not enabled → switch to Sysmon Event 1

2. Check: Is Sysmon Event 1 collected in SIEM?
   → query: Sysmon | where EventID == 1 | take 1
   → If results: Sysmon is the correct source for this hunt

Failure Mode 4: Premature Closure

Pattern:

Hunt: "Find beacons to external IPs with CV < 0.30"

Step 1: Rate filter → 2,847 results
Step 2: Peer cohort filter → 43 results
Step 3: CV calculation on top 5 (by conn_count) → all clean

Analyst conclusion: "No beacons found."

MISTAKE: Top 5 by conn_count were safe harbored after calculation. 
         Candidates 6–43 were never evaluated because analyst ran out of time.

Prevention — Define explicit closure criteria in the hypothesis:

Hunt is COMPLETE when:
  - All 43 candidates have been evaluated OR
  - All remaining candidates have conn_count < 100 AND are in safe harbor
Hunt is INCONCLUSIVE if:
  - Time limit reached with candidates 6–43 unevaluated
  - Document as partial hunt: "candidates 1–5 evaluated, 6–43 pending"

Premature closure without documentation is the same as not hunting at all — it produces false confidence without the work to back it.

Worked Example

Diagnosing a failed hunt and rebuilding it correctly

Original hunt report: “We hunted for Cobalt Strike in our environment. Found nothing. We are not compromised.”

Failure mode analysis:

Hypothesis quality check:
  Specific?      NO — "Cobalt Strike" covers 20+ techniques
  Testable?      PARTIALLY — depends on what was actually queried
  Time-bounded?  NOT STATED
  Data-supported?  UNKNOWN

Data source:
  Not documented — unknown if Sysmon or security events
  Unknown if command line logging was enabled
  
Analysis:
  Null result stated without coverage statement
  No mention of data gaps, safe harbor list, or CV threshold used
  No list of candidates evaluated vs. pending
  
Confidence in null result: VERY LOW
(No hypothesis, no data documentation, no coverage statement)

Rebuilt hunt with correct structure:

Hypothesis: "If Cobalt Strike is operating in our environment, it would appear as 
             a host with conn_count > 100 to a rare external destination (host_count ≤ 3)
             with interval CV < 0.30 and service=ssl in Zeek conn.log, 
             between 2024-06-01 and 2024-09-01."

Data validation:
  - Zeek conn.log: full-fidelity confirmed (no sampling), 90-day retention ✓
  - ssl.log: JA3 field populated, 90-day retention ✓
  - Endpoint Sysmon: 90-day retention, command line logging enabled ✓
  - Coverage gap: branch office (2 of 8 Zeek sensors) — 30-day retention only

Hunt execution:
  - 43 candidates after rate + peer cohort filter
  - CV < 0.30 applied: 2 candidates flagged
  - Both investigated: 1 confirmed TP (C2), 1 FP (vendor monitoring tool)

Confidence in null result for remaining 41: HIGH (within HQ scope)
Coverage gap acknowledged: branch office has only 30-day data; 60-90 day C2 not covered

Conclusion: "One confirmed Cobalt Strike beacon found. No additional beacons in 
             HQ segments with CV < 0.30 over 90 days. Branch office: incomplete 
             coverage — separate hunt required."

Retrieval Practice

A hunter reports: 'We ran the beacon hunt for 30 days and found 847 candidates. We evaluated the top 50 by connection count and cleared them all. Hunt complete — no beacons.' What are the failure modes present in this conclusion?

Reveal answer

Multiple failure modes: (1) Premature closure — 797 of 847 candidates were never evaluated. The statement 'hunt complete' after evaluating only the top 50 by connection count is incorrect. High-jitter C2 (lower connection count) would appear in the uninvestigated 797. (2) Selection bias — sorting by connection count prioritizes high-rate beacons. A sophisticated C2 with lower frequency (daily check-in, conn_count = 30) would be ranked at the bottom and never evaluated. (3) Missing coverage statement — no mention of the safe harbor list, the time window, data source quality, or what would make a candidate 'cleared.' (4) False confidence conclusion — 'no beacons' overstates the result. The correct conclusion: 'Top 50 candidates by connection count evaluated and cleared. 797 candidates below the investigation threshold remain unevaluated. High-jitter or low-frequency C2 (CV > 0.30 or conn_count < threshold) not covered by this hunt run.'

Retrieval Practice

A threat hunter suspects the data for a beacon hunt is being sampled (only 10% of connections recorded). How does this affect the hypothesis, and what alternative hypothesis should be used?

Reveal answer

Sampled conn.log data makes interval-based beacon detection unreliable. If only 10% of connections are recorded, a 60-second beacon's interval distribution will show gaps and inconsistencies — the CV calculation will reflect the sampling artifact, not the beacon's actual jitter. A beacon with CV=0.03 (very regular) may appear as CV=0.8 (irregular) due to missing connection records. Alternative hypothesis: switch from interval analysis to long-connection hunting. A persistent C2 session maintained as a single long TCP connection (instead of frequent reconnects) appears as one conn.log entry with duration > 4 hours — immune to sampling artifacts because the connection itself is recorded, not its frequency. The hypothesis becomes: 'If C2 is active via a persistent connection, it would appear as conn_count < 10 but duration > 14,400 seconds (4 hours) to a rare external destination, which is robust to 10% sampling.' Document the data fidelity issue in the hunt journal so subsequent analysts know the statistical method is unavailable until Zeek sampling is resolved.