The System Design Interview — Designing Detection at Scale

A system design question in a detection engineering interview is not asking you to build a SIEM. It’s asking whether you can think about detection as an interconnected system — telemetry, analytics, pipeline, tuning, testing, and metrics — rather than a collection of rules.

The Standard System Design Structure

1. CLARIFY CONSTRAINTS (5 min)
   Ask before designing.

2. TELEMETRY LAYER (7 min)
   What data feeds the detections?

3. DETECTION LAYER (10 min)
   What gets detected and how?

4. ALERT PIPELINE (5 min)
   How do alerts become actions?

5. TUNING STRATEGY (5 min)
   How does the system handle FPs?

6. TESTING + VALIDATION (5 min)
   How do you know the rules work?

7. METRICS (3 min)
   How do you measure success?

Total: ~40 minutes. Spend proportionally — don’t spend 30 minutes on detection rules and skip testing and metrics.

Example: “Design a detection system for insider threat”

Step 1: Clarify Constraints

“Before I start — a few clarifying questions. Is this a cloud-first environment or on-prem? How many employees? Are we primarily concerned with data exfiltration or sabotage? Is there an existing DLP solution? What’s the budget — are we adding to a Splunk/Elastic stack or designing from scratch?”

Assume answer: 2,000 employees, hybrid (Azure + on-prem Windows), concerned primarily with data exfiltration, Splunk already deployed, no DLP.

Step 2: Telemetry Layer

For insider threat detection, the relevant log sources are:

Coverage verification: “I’d confirm Sysmon is deployed to 100% of Windows endpoints and that M365 audit logging is active and flowing to Splunk before writing any rules.”

Step 3: Detection Layer

User behavior analytics (UBA) approach:

Behavioral baseline for each user:
  - Normal hours of activity
  - Normal data access volume
  - Normal cloud storage upload volume
  - Normal peer group (which departments they interact with)

Anomaly alerts:
  - Access volume 3σ above personal baseline (not just above average)
  - Data access outside normal hours
  - New external sharing not matching peer group behavior
  - Email forwarding rule to external domain (M365 UAL)

Rule-based detections:

# Insider threat — Email forwarding to external address
Detection: New-InboxRule with ForwardTo external domain → Level: High

# Insider threat — Bulk SharePoint download
Detection: >500 SharePoint GetObject events from one user in 1 hour → Level: Medium

# Insider threat — USB data staging
Detection: Sysmon File creation to drive letter != C: by user with no documented business need

Detection-in-depth layers:

DML-4: Specific exfil tools (Rclone.exe, azcopy.exe, winscp.exe by non-IT users)
DML-5: Web upload to known file sharing domains (Box, Dropbox, personal OneDrive)
DML-6: Large outbound data volume from process categories outside IT tools
DML-7: Statistical deviation from personal baseline in data access patterns

Step 4: Alert Pipeline

Alert routing:
  High severity (forwarding rule, bulk download) → IR team within 15 min
  Medium severity (anomalous access) → SOC analyst review within 4 hr
  Low severity (unusual hour access) → Weekly review, aggregate for trend analysis

Enrichment on every alert:
  - Employee status (active/on PIP/resignation notice)
  - Recent HR events (performance review, denied promotion)
  - Manager chain (who to notify)
  - Asset classification of accessed data (public/internal/confidential/restricted)

Step 5: Tuning Strategy

Insider threat has unique tuning challenges:

• No external threat intel — insider behavior is unique to your org’s baseline
• Long baseline establishment — 90+ days of user history needed before anomaly detection is meaningful
• High sensitivity to env changes — project assignments, department transfers, new tools all shift baselines

FP management: “I’d tier users by risk factors (access to sensitive data, recent HR events, network access breadth) and apply tighter thresholds to higher-risk users, looser thresholds to lower-risk users, rather than one global threshold.”

Step 6: Testing

“For insider threat, there’s no Atomic Red Team test — the behaviors are human-generated. I’d validate by:

1. Synthetic user simulation: create a test account, perform the exact behaviors the rules target, verify alerts fire
2. Red team insider scenario: have a red teamer emulate an insider using legitimate credentials
3. Historical incident replay: if we have past insider incidents, replay the log artifacts and verify the rules would have caught them”

Step 7: Metrics

“Key metrics for this system:

• MTTD for insider incidents (target: detect within 24 hours of pattern start)
• FP rate per detection category (UBA tends to have higher FP rates — monitor separately)
• Coverage of exfil paths tested (email / USB / web upload / cloud sync) — verify each path quarterly via synthetic test”