What Is a Log? Structure, Timestamps, Canonical Fields

Every detection you write depends on a log that someone, somewhere, decided to generate. Understanding what a log is — and what distinguishes a useful one from a useless one — is the foundation of everything that follows.

Events vs Logs

These terms are often used interchangeably, but the distinction matters:

• An event is the occurrence — “a process started,” “a user authenticated,” “a file was modified.”
• A log is the record — the bytes written to disk or sent over the wire that describe the event.

You detect adversary behavior by analyzing logs. But the log’s quality determines whether detection is even possible.

Structured vs Unstructured Logs

Unstructured: Classic Apache Access Log

192.168.1.100 - frank [10/Oct/2023:13:55:36 -0700] "GET /malware.exe HTTP/1.1" 200 2326

To extract the IP address, you need a regex: ^(\S+). To extract the URL, another regex. If Apache changes its format across versions, your regex breaks silently.

Structured: Same Event in JSON

{
  "timestamp": "2023-10-10T13:55:36-07:00",
  "src_ip": "192.168.1.100",
  "user": "frank",
  "method": "GET",
  "uri": "/malware.exe",
  "protocol": "HTTP/1.1",
  "status": 200,
  "bytes": 2326
}

Now src_ip is a typed field. Querying src_ip == "192.168.1.100" is exact. Adding an index on uri is trivial. Joining with a threat intel feed on src_ip is a lookup, not a string parse.

The Five Canonical Fields

Every useful security log has five fields. Missing any one degrades your detection capability:

Timestamps: The Hardest Canonical Field

Timestamps sound simple. They aren’t.

Three timestamp problems you will encounter:

1. Missing Timezone

2024-03-15T14:00:00          ← Which timezone? UTC? Local? EST? PST?
2024-03-15T14:00:00Z         ← UTC (ISO 8601 with Z suffix) ✓
2024-03-15T14:00:00-05:00    ← EST (explicit offset) ✓

Always require UTC or explicit offset. In a global incident, a missing timezone means a 12-hour window of ambiguity.

2. Clock Skew

Log collectors and endpoints have clocks. If a Windows workstation is 3 minutes ahead of your SIEM, the process creation event can arrive before the authentication event that caused it. Your correlation query sees the process but no preceding login — and the alert fires or fails to fire incorrectly.

3. Ingestion Time vs Event Time

SIEMs record two timestamps: when the event occurred (from the log) and when the SIEM received it. A log forwarder that buffers 24 hours before shipping will show events at the correct event time but with a 24-hour ingestion delay. Alerts based on “received time” will be 24 hours late.

Why This Matters Before Writing Your First Detection

The rule process.name == "mimikatz.exe" AND parent.name == "cmd.exe" requires:

• process.name — structured, named field ✓
• parent.name — structured, named field ✓
• Both events from the same host, correlatable by time ✓

If your endpoint telemetry doesn’t give you parent process context (parent.name), this rule is impossible. If timestamps aren’t in UTC, your cross-host correlation joins can produce phantom sequences.

Every detection requirement traces back to: “do my logs have the fields and quality to support this query?”