Measuring Tuning Impact — Precision Deltas and MTTD Models

You can’t improve what you don’t measure. After a tuning pass, you need to answer two questions: did this make the rule better, and by how much? This lesson establishes the measurement vocabulary and methods for making tuning decisions data-driven.

The Precision Delta Calculation

Before any tuning change, establish a baseline:

Period: Past 30 days (or whatever window represents steady-state behavior)
Total alerts: 480
Sample size: 40 (random sample)
  TPs in sample: 4 (10%)
  FPs/BTPs in sample: 36 (90%)

Estimated Precision_before = 10%
Estimated TP rate per day = 480 × 10% = 48 TPs/month
Estimated FP/BTP rate per day = 480 × 90% = 432 FPs/month

After adding exclusions:

Period: Next 30 days
Total alerts: 120 (75% reduction)
Sample size: 40
  TPs in sample: 16 (40%)
  FPs/BTPs in sample: 24 (60%)

Estimated Precision_after = 40%
TP rate per day = 120 × 40% = 48 TPs/month (SAME — TPs preserved)
FP/BTP rate per day = 120 × 60% = 72 FPs/month (83% reduction)

Precision delta = 40% - 10% = +30pp improvement

The precision delta (+30 percentage points) tells the story: same TP detection, dramatically fewer FPs. This is the measurement to report to the detection team lead.

Validating That TPs Are Preserved

A precision delta only tells part of the story. You also need to verify TP count didn’t drop:

TP count before: ~48/month
TP count after:  ~48/month (same)

If after tuning the estimated TP count drops significantly, an exclusion is too broad:

BAD: TP count before: 48/month → TP count after: 12/month
Precision_before: 10% → Precision_after: 75%

This looks like great precision improvement — but 36 real threats per month
are now being suppressed. This is a regression.

Always check both precision AND TP count. Run regression tests with known-bad samples to verify.

Regression Testing

Regression testing confirms that a tuning change didn’t accidentally suppress TPs.

Manual Regression Test

Collect a set of confirmed true-positive samples — events that should trigger the rule
- Previous incident events (extracted from SIEM as EVTX or JSON)
- Atomic Red Team test output (from a controlled test environment)
- Synthetically generated events matching the detection pattern
Apply the updated rule (with new exclusions) to the sample set
Verify: every known-TP event still generates an alert

# Using sigma-cli with a test dataset:
sigma convert -t splunk -p sysmon rule.yml > query.spl
# Then run query.spl against known-TP dataset in test Splunk instance
# Expected: all 20 TP samples generate alerts

CI/CD Integration (Detection-as-Code)

For teams using detection-as-code pipelines:

# .github/workflows/rule-test.yml
- name: Test against known-good TP dataset
  run: |
    sigma convert -t splunk -p sysmon rules/lsass-dump.yml > query.spl
    python test_rule.py \
      --query query.spl \
      --tp-samples tests/lsass-dump/true-positives/*.json \
      --fp-samples tests/lsass-dump/false-positives/*.json \
      --expected-tp-rate 1.0 \
      --expected-fp-rate 0.0

This fails the CI pipeline if any TP sample no longer fires, or if any known-FP sample now fires (regression in either direction).

The Alert Volume Curve

Plot daily alert counts for 90 days before and after a major tuning change:

Alert Volume: Rule "LSASS Memory Access"

Day 1-60 (pre-tuning):
    ████████████████████████████  avg 16/day

Day 61 (exclusions added):
    ░░░░░░░░░░░░░░  avg 4/day

Day 61-90 (post-tuning):
    ████  avg 4/day (stable — with occasional spikes on incident days)

Healthy pattern: Sharp drop on tuning day, stable plateau, occasional spikes for real events.

Warning patterns:

Pattern 1: Gradual rise post-tuning
Day 61-90: ████ ███████ ████████████ ████████████████
New FP source emerged. Sample alerts from the rising period and identify the new source.

Pattern 2: Drop to near-zero
Day 61-90: ░
An exclusion is too broad — it's suppressing TPs as well as FPs.
Run regression test immediately. Check which exclusion is matching real attacks.

The MTTD Impact Model

Reducing FP rate has a compounding effect on MTTD:

Assumptions:
  - 2 analysts, each handling 50 alerts/hour
  - Total capacity: 100 alerts/hour
  - Real attacks occur at: 3/day (1 every 8 hours)

Scenario A (Pre-tuning):
  Alert volume: 400/day
  Time for one analyst to work through queue: 400/100 = 4 hours
  A TP arriving in a full queue waits, on average, 2 hours for an analyst
  MTTD contribution from queue depth: +2 hours

Scenario B (Post-tuning, 75% FP reduction):
  Alert volume: 100/day
  Queue clearance: 100/100 = 1 hour
  A TP arriving in the queue waits, on average, 30 minutes
  MTTD contribution from queue depth: +30 minutes

Delta: 1.5 hours reduction in MTTD from queue depth alone

In organizations with a 4-hour MTTD SLA, this tuning change can be the difference between meeting and missing SLA — without writing any new detection rules.

Building a Tuning Dashboard

A simple tuning dashboard tracks four metrics per rule, updated weekly:

Metric	Formula	Alert threshold
Alert volume (7d)	COUNT(alerts)	>2× baseline
Estimated precision	Sample TP/total	<70% → tune
TP delta (30d)	TP_this_month - TP_last_month	<-20% → regression check
Last tuned	Date of last exclusion change	>90 days without tuning → review

Sort by estimated precision ascending — the least precise rules at the top need attention first.

Detection Engineering Weekly— Detection Engineering Weekly Atomic Red Team— Red Canary

Retrieval Practice

A rule had 200 alerts last month (estimated 80% FP rate = 40 TPs, 160 FPs). After adding 3 exclusions, it fires 40 times this month (estimated 50% FP rate = 20 TPs, 20 FPs). Calculate: (a) precision delta, (b) TP delta, and (c) what does the TP delta tell you about the exclusions?

Reveal answer

(a) Precision_before = 40/200 = 20%. Precision_after = 20/40 = 50%. Precision delta = +30pp — improvement. (b) TP_before = 40/month. TP_after = 20/month. TP delta = -20 (50% fewer TPs). (c) The exclusions suppressed roughly 20 real attacks per month in addition to 140 FPs. This is a partial regression — precision improved but TP count dropped significantly. The correct response: review each exclusion against the known-TP sample set. One or more exclusions is too broad — it's matching both legitimate activity AND real attack patterns. Narrow the most recent exclusion first (it's most likely the problem), run regression tests, and remeasure.

Retrieval Practice

Your detection team ships 10 new rules and schedules a tuning review 'when time permits.' Six months later, no tuning has happened. What predictable outcomes do you expect, and what process change would prevent them?

Reveal answer

Predictable outcomes: (1) Alert volume has grown as new software deployments and business changes introduce new FP sources — each untuned rule now fires more. (2) Analysts have begun deprioritizing or auto-closing some rule families because they've 'learned' they're noisy — real TPs from those rules are being missed. (3) Exclusion debt has accumulated: some rules have well-known FP sources that analysts work around manually every time rather than suppressing. (4) MTTD has increased as queue depth grew. Process change: establish a Detection SLA requiring every rule to have a tuning review within 30 days of deployment, then quarterly thereafter. Block calendar time (2–4 hours/week per engineer) specifically for tuning reviews. Track precision per rule in a dashboard — any rule below 70% precision goes on the sprint backlog for the next cycle.