Vanity vs. Signal Metrics — What to Measure in a Detection Program

You can’t improve what you don’t measure. But measuring the wrong things creates the illusion of progress while the real problems compound.

The Two Types of Detection Metrics

Vanity metrics measure activity. Signal metrics measure outcomes.

Vanity Metric	What it seems to show	What it actually measures
Total rules in SIEM	Detection breadth	Rule count, regardless of quality or coverage
Total alerts per day	Detection activity	Alert volume, mostly FP rate
Mean time to respond (MTTR)	Analyst efficiency	How fast analysts close tickets, not how fast attacks are detected
Number of playbooks documented	Process maturity	Documentation completeness, not execution quality
Rules deployed this sprint	Team velocity	How many rules were added, not whether they work

Signal metrics measure what matters to the business:

Signal Metric	What it measures	Why it matters
MTTD (Mean Time to Detect)	Time from attacker action to first detection	Lower = less time for attacker to achieve objectives
Alert-to-ticket ratio	Precision at program level	Tracks analyst fatigue and FP rate
Detection coverage %	ATT&CK technique coverage	Breadth of detection across the threat model
Rule health rate	% of rules with TP+FP fixtures passing in CI	Rule quality and maintenance status
Dwell time trend	Historical MTTD over time	Is detection improving or degrading?

Calculating MTTD

Incident A: Initial attacker action at 09:00, first alert at 09:45 → MTTD = 45 min
Incident B: Initial attacker action at Day 1 14:00, first alert at Day 3 09:00 → MTTD = 43 hr
Incident C: Initial attacker action at 16:00, first alert at 17:15 → MTTD = 75 min

MTTD (average) = (45 min + 43 hr + 75 min) / 3 = ~ 15 hr average
MTTD (median) = 75 min (Incident C)

Use median rather than mean when you have outliers — a single 72-hour dwell incident shouldn’t inflate the MTTD to make everything else look worse. Track both: median for day-to-day program health, mean for trend analysis and SLA discussions.

Retrospective MTTD: Calculate from incident post-mortem data — when did the attacker gain initial access (based on forensic artifacts) vs. when did the first alert fire? This is often weeks after the incident, not real-time.

Alert-to-Ticket Ratio: Reading the Signal

Healthy state: 1 ticket per 3–5 alerts (30–60% alert precision)

Week 1: 1,000 alerts → 50 tickets → ratio 1:20 (warning)
Week 2: 1,200 alerts → 45 tickets → ratio 1:27 (deteriorating)
Week 3: 1,500 alerts → 40 tickets → ratio 1:38 (alert fatigue state)

When the ratio deteriorates:

Identify the FP contributors — which rules are generating the most alerts with the lowest ticket rate?
Tune or retire — if a rule generates 200 alerts/week with 0 tickets, it is a pure FP generator. Either tune it to fire less or retire it.
Do not compensate with automation — auto-closing FP alerts without fixing root cause just hides the problem while analyst trust in the system erodes.

Coverage Percentage: The Breadth View

Threat model: 45 techniques relevant to your sector with available telemetry
Rules in place: 28 techniques covered with at least one tested rule
Coverage: 28/45 = 62%

Breakdown by tactic:
  Initial Access: 5/6 techniques = 83%
  Execution: 4/7 techniques = 57%
  Persistence: 6/9 techniques = 67%
  Credential Access: 3/8 techniques = 38%  ← gap
  Lateral Movement: 4/5 techniques = 80%
  Collection: 3/6 techniques = 50%
  Exfiltration: 3/4 techniques = 75%

Coverage gaps in high-risk tactic areas (Credential Access, Collection) are more urgent than gaps in low-risk areas. Report coverage by tactic, not just total percentage.

Rule Health Rate: Quality Over Quantity

Total rules: 150
Rules with passing TP fixture in CI: 82
Rules with passing FP fixture in CI: 61
Rules with both TP + FP fixtures: 55
Rules deployed without any fixtures: 68

Rule health rate: 55/150 = 37%

37% rule health is common in detection programs that have accumulated rules over years without systematic testing. A rule without a TP fixture may have broken due to field mapping changes, SIEM updates, or log source format changes — it’s silently failing without generating any alerts.

Rule health sprint: Quarterly, pick the 10 oldest rules without fixtures and either add fixtures or retire the rule. Trend toward 80%+ rule health over 12 months.

M-Trends (Annual Dwell Time Data)— Mandiant/Google

Retrieval Practice

Your detection program has 350 rules and fires 5,000 alerts per week. Your team creates 50 tickets per week. Your CISO says 'great, 350 rules means excellent coverage.' What are the two metrics that would challenge this assessment?

Reveal answer

(1) Alert-to-ticket ratio: 50 tickets / 5,000 alerts = 1:100 ratio. This means 98% of alerts are FPs — analysts are drowning in noise and are likely missing real detections in the queue. A 350-rule program generating 1% actionable alerts is not 'excellent coverage' — it's a liability that causes alert fatigue and analyst burnout. (2) Coverage percentage: the 350 rules may cover only 15 ATT&CK techniques, each with 20+ tuning variants. A better question is: how many of the 45 techniques in the relevant threat model have at least one tested, working detection rule? Rule count ≠ technique coverage. Challenge the CISO by presenting: 'We have 350 rules covering 15 techniques (33% coverage) with a 1:100 alert-to-ticket ratio. The program needs fewer, better-tested rules and FP tuning, not more rules.'

Retrieval Practice

After a red team engagement, you calculate MTTD for the simulated attack: the red team gained initial access at Day 1 09:00, moved laterally at Day 2 14:00, accessed sensitive data at Day 3 16:00, and the detection team first flagged suspicious activity at Day 4 11:00. What is the MTTD and what does this tell you about detection coverage gaps?

Reveal answer

MTTD = Day 4 11:00 − Day 1 09:00 = 74 hours. During this 74-hour window, the red team completed the full kill chain (initial access → lateral movement → data access) without detection. This tells you: (1) Initial access detection is missing or silent — phishing/initial vector generated no alert. (2) Lateral movement detection is absent or below threshold — post-compromise movement should generate higher-fidelity alerts than initial access. (3) Data access detection (file access, S3 download, credential access) generated no alert despite occurring 19 hours before detection. The 74-hour MTTD means a real attacker could have exfiltrated data, encrypted systems, or established persistence before any response began. Gap analysis: audit which specific events during the red team's kill chain had no corresponding detection rule — those are your highest-priority gap closures.