Calibrated Assessment — Confidence Language, Hunt Journals, and Closing Hunts

Confidence is earned, not assumed. A null hunt result is only as valuable as the analyst’s documented certainty about what they covered. Calibrated assessment is the last step — and the one most frequently skipped.

ICD 203 Confidence in Hunt Outputs

The Intelligence Community uses standardized confidence language to prevent consumers from over-interpreting uncertain assessments. Threat hunters should adopt the same discipline.

Confidence Mapping for Hunt Findings

Finding Type	Data Quality	Corroboration	Confidence
TP: malfind MZ header + VT 47/72 + ESTABLISHED C2 connection	Full-fidelity Zeek + Sysmon	3 independent signals agree	HIGH
TP: beacon CV = 0.024 + self-signed cert	Full-fidelity Zeek	2 signals agree, no endpoint pivot	MODERATE
Lead: slightly elevated NXDOMAIN ratio, no other signals	Full-fidelity Zeek DNS	1 signal, no corroboration	LOW
Null: no beacons found (full coverage)	Full-fidelity, full retention	Complete data, all candidates evaluated	HIGH (null)
Null: no beacons found (partial coverage)	Some segments missing	Not all candidates evaluated	MODERATE (null)

Language Conventions (ICD 203-style)

Phrase	Confidence Level	Use When
”We assess with high confidence…”	HIGH	All signals corroborate, data complete
”We assess with moderate confidence…”	MODERATE	2 of 3 signals present, minor data gap
”We assess with low confidence…”	LOW	1 signal, significant data gap
”We cannot assess…”	N/A	Data not available for this hypothesis
”We found no evidence of…”	Null (state confidence separately)	Null result with explicit scope

The Hunt Journal Entry

Every hunt, regardless of result, requires a journal entry that can stand alone.

Complete Entry Template

## Hunt [YYYY-MM-DD-NN] — [TTP Name] — [Hypothesis Source]

**Date:** 2024-09-02
**Analyst:** M. Torres
**Hunt type:** Hypothesis-Driven — from CTI report [UNC3524 campaign, 2024-08-28]

---

### Hypothesis
"If an adversary used DNS tunnelling (T1048.003) via TXT records against our 
 M365 email environment between 2024-06-01 and 2024-09-02, we would observe 
 TXT query volume > 50/hour from a single internal host in Zeek dns.log, with 
 average first-label length > 30 characters."

**Null hypothesis:** "No adversary is using DNS tunnelling via TXT records in 
 our environment during this period."

---

### Data Source Validation
- Zeek dns.log: full-fidelity (no sampling) ✓
- Retention: 90 days — covers full hypothesis window ✓
- TXT qtype field populated: confirmed (sampled 100 TXT queries in past 7 days) ✓
- Coverage gap: branch office Zeek sensors — only 7-day retention
  → Branch office excluded from this hunt's null result

---

### Queries Run
1. TXT query volume per host (30-day window) — see: dns_txt_volume_2024-09-02.kql
2. Unique FQDN count per registered domain (tunnelling signature) — see: dns_fqdn_ratio.kql
3. Average first-label length filter (>30 chars) — see: dns_label_len.kql

---

### Findings
**Result:** NULL — no DNS tunnelling found in HQ segments for 30-day window

Candidates evaluated: 4 hosts with TXT query count > 50 in 30 days
- 10.1.1.5: 134 TXT queries to _dmarc.corp.com — EXCLUDED (mail server, legitimate SPF)
- 10.1.1.6: 89 TXT queries to various domains — EXCLUDED (weather app, SPF checks, mixed)
- 10.4.2.15: 67 TXT queries to google.com TXT — EXCLUDED (DNS lookup tool, known)
- 10.3.3.1: 52 TXT queries (avg label len: 12 chars) — NOT TUNNELLING (labels too short)

All 4 candidates triaged. None met tunnelling criteria.

---

### Assessment
**Confidence in null result:** MODERATE
- High-quality full-fidelity data for HQ (full retention, no sampling)
- Branch office excluded — 7-day retention insufficient for this 30-day hunt
  → A tunnelling channel present only in branch offices and older than 7 days 
    would NOT be caught by this hunt

**Conclusion:** "We assess with MODERATE confidence that no DNS tunnelling via TXT 
 records occurred in HQ segments between 2024-06-01 and 2024-09-02. Branch office 
 is an unvalidated gap. This technique is not currently evidenced in the covered scope."

---

### Next Hypothesis Generated
Hypothesis [2024-09-02-02]: "Check if branch office Zeek sensors can be upgraded 
 to 30-day retention, then re-run this hunt with branch coverage for Q4 2024."

Closing a Hunt: The Completion Checklist

□ All success criteria met (all candidates evaluated OR threshold documented)
□ Hunt journal entry written (hypothesis, data, queries, findings, confidence)
□ Coverage gaps explicitly stated
□ PEAK Act outputs produced:
    □ IR ticket (if TP found)
    □ Detection rule + ABLE documentation (if TP → durable pattern)
    □ CTI deliverable (if novel finding)
    □ Null result documented (if no TP — still required)
□ Next hypothesis spawned (even if: "no follow-up needed — this TTP covered")
□ ATT&CK Navigator layer updated
□ Hunt report shared with detection engineering team

A hunt is officially closed when all boxes are checked. An unclosed hunt — findings documented but no detection rule, or null result with no journal entry — produces half the value of a properly closed one.

Worked Example

Calibrating two analysts' conclusions on the same hunt data

Hunt data: DNS tunnelling hunt, 847 TXT queries from host 10.4.2.11 to cdn-telemetry.io, avg label length 55 chars, WHOIS: registered 6 days ago, decoded sample = base64 text.

Analyst A conclusion:

"We have definitely found DNS exfiltration. This host is leaking data."

Problems: (1) No confidence level stated. (2) “Definitely” is uncalibrated. (3) “Leaking data” is the conclusion, not the finding — the finding is TXT queries with encoded content.

Analyst B conclusion (calibrated):

"We assess with HIGH confidence that host 10.4.2.11 is exfiltrating data via 
 DNS TXT tunnelling to cdn-telemetry.io, based on:
   - 847 TXT queries over 24h (anomalous for a workstation)
   - 612 unique subdomains, average label length 55 chars (base64 signature)
   - WHOIS: cdn-telemetry.io registered 6 days ago (fresh infrastructure)
   - Decoded sample: plaintext data in expected base64 format (confirmed encoding)

 Volume estimate: 25 KB exfiltrated in observed 24h window.
 
 The remaining uncertainty: we do not know WHAT data was exfiltrated — only that 
 structured data was encoded and transmitted. Endpoint pivot required to identify 
 source process and what files it accessed.
 
 Confidence caveat: if the decoded data was a legitimate application's telemetry 
 encoded in base64, this would be a FP. Endpoint process identification will resolve."

Analyst B’s assessment gives IR: (1) the confidence level, (2) the evidence for that confidence, (3) the remaining uncertainty, (4) what next step resolves the uncertainty. Analyst A’s assessment gives IR nothing actionable beyond “someone found something.”

Retrieval Practice

An analyst reports: 'We hunted for T1059.001 (PowerShell abuse) and found no evidence. We are confident our environment is clean.' What information is missing and how should the statement be rewritten?

Reveal answer

Missing: (1) Data source — which log source was queried? Sysmon Event 1? Windows Security 4688? PowerShell ScriptBlock logging (Event 4104)? (2) Time window — 7 days, 30 days, Q3 2024? (3) Scope — all endpoints, HQ only, a specific VLAN? (4) Confidence level — HIGH/MODERATE/LOW and why. (5) Coverage gaps — what was excluded and why. (6) Query approach — what defined 'PowerShell abuse'? Any -enc flag? Office parent? All PowerShell? Rewritten correctly: 'We assess with MODERATE confidence that no PowerShell abuse (T1059.001) via Office application parent spawning occurred on HQ endpoints between 2024-07-01 and 2024-09-30, based on Sysmon Event 1 queries for Office-parent PowerShell processes (full-fidelity Sysmon, 90-day retention, HQ only). Branch offices not covered — Sysmon not deployed there. PowerShell execution via non-Office parents (direct user launch, service execution) was not in scope for this hunt run.'

Retrieval Practice

What is the purpose of 'spawning a next hypothesis' as a required step in closing a hunt, even when the result is null?

Reveal answer

Spawning the next hypothesis serves three functions: (1) Converts the hunt into a learning cycle rather than a dead end — every result (TP or null) generates information that points toward the next hunt. A null result for DNS tunnelling might generate: 'Hunt T1048.001 (exfiltration over non-standard port) — the data coverage gap in branch offices means we should hunt there specifically.' (2) Prevents hunt debt — documents outstanding hypotheses in the journal so they can be scheduled and tracked, not forgotten. (3) Maintains momentum in the hunt program — a team that only hunts when they have active leads is reactive; a team that generates hypotheses from every result maintains proactive coverage. Even the most complete null result generates a hypothesis: 'This TTP is covered — validate that coverage quarterly by confirming Zeek sensor health and retention.' That's a follow-up hypothesis for a future hunt cycle.