Kubernetes Audit Logs — verb, resource, user, sourceIPs

Kubernetes audit logs are the control-plane telemetry for container orchestration — they record every API call to the kube-apiserver. In a Kubernetes environment, the API server is the single chokepoint for all cluster operations, making it the highest-value telemetry source.

🔍 Check your cluster's audit policy

If you have Kubernetes access:

# Find the kube-apiserver manifest
cat /etc/kubernetes/manifests/kube-apiserver.yaml | grep -A5 "audit"

# Or for managed clusters (EKS, GKE, AKS) — check the audit log configuration
# EKS: aws eks describe-cluster --name my-cluster | jq '.cluster.logging'
# GKE: gcloud container clusters describe my-cluster --format='json' | jq '.loggingConfig'

Questions: Is audit logging enabled? What’s the audit policy path? Are secret reads being logged?

Attempt this before reading on. Getting stuck is the point.

Kubernetes Audit Log Structure

{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",
  "level": "RequestResponse",
  "auditID": "3da92f8d-8f31-4497-b6c2-54ff16948f7a",
  "stage": "ResponseComplete",
  "requestURI": "/api/v1/namespaces/production/pods/webapi-7d9b8c-xk2np/exec",
  "verb": "create",
  "user": {
    "username": "attacker@victim.com",
    "uid": "12345678-abcd-...",
    "groups": ["system:authenticated"],
    "extra": {}
  },
  "impersonatedUser": null,
  "sourceIPs": ["203.0.113.55"],
  "userAgent": "kubectl/v1.28.0 (linux/amd64) kubernetes/abc123",
  "objectRef": {
    "resource": "pods",
    "namespace": "production",
    "name": "webapi-7d9b8c-xk2np",
    "apiVersion": "v1",
    "subresource": "exec"
  },
  "responseStatus": {
    "code": 101
  },
  "requestObject": {
    "stdin": true,
    "stdout": true,
    "tty": true,
    "container": "webapi",
    "command": ["/bin/bash"]
  },
  "timestamp": "2024-03-15T09:23:14.882Z"
}

Key fields for detection

Field	Detection use
`verb`	What type of operation — exec, create, delete
`user.username`	Who made the request
`user.groups`	Group membership (system:masters = cluster-admin)
`sourceIPs`	Where the request came from
`objectRef.resource`	What type of resource (pods, secrets, configmaps)
`objectRef.namespace`	Which namespace
`objectRef.name`	Which specific resource
`objectRef.subresource`	exec, log, portforward — high-interest operations
`responseStatus.code`	200/201=success, 401=unauth, 403=denied, 404=not found
`requestObject`	What was requested (command in exec, spec in create)

High-Priority Detection Patterns

1. Interactive Shell Execution (`exec`)

{
  "verb": "create",
  "objectRef": {"resource": "pods", "subresource": "exec"},
  "requestObject": {"command": ["/bin/bash"], "tty": true}
}

kubectl exec -it <pod> -- /bin/bash generates this. In production, there is almost no legitimate reason to exec an interactive shell into a running pod — operations should be automated, not interactive. This is a critical alert.

2. Secret Access

{
  "verb": "get",
  "objectRef": {"resource": "secrets", "namespace": "production", "name": "aws-credentials"},
  "user": {"username": "system:serviceaccount:default:compromised-sa"}
}

A service account in the default namespace reading a secret named aws-credentials in the production namespace — cross-namespace secret access is unusual and warrants investigation.

3. ClusterRoleBinding Creation (Privilege Escalation)

{
  "verb": "create",
  "objectRef": {"resource": "clusterrolebindings"},
  "requestObject": {
    "metadata": {"name": "backdoor-admin"},
    "roleRef": {"name": "cluster-admin"},
    "subjects": [{"kind": "User", "name": "attacker@victim.com"}]
  }
}

Creating a ClusterRoleBinding that grants cluster-admin to an external user is the Kubernetes equivalent of adding someone to the Domain Admins group. Immediate alert.

Worked Example

Writing a Sigma rule for Kubernetes exec

title: Kubernetes Interactive Pod Shell
id: 7e9a2b1c-4d5f-6e7a-8b9c-0d1e2f3a4b5c
status: stable
logsource:
  product: kubernetes
  service: audit
detection:
  selection:
    verb: create
    objectRef.subresource: exec
    requestObject.tty: true
  filter_ci:
    user.username|contains: 'ci-runner'
    objectRef.namespace: 'ci'
  condition: selection and not filter_ci
falsepositives:
  - Development environments where developers debug interactively
  - Break-glass procedures for production incidents (should be documented)
level: high
tags:
  - attack.execution
  - attack.t1609

The filter excludes CI runners in the CI namespace — those legitimately exec into pods to run test commands. Production namespaces get no exclusions.

Configuring an Audit Policy

Without a policy, Kubernetes logs nothing. A production-grade minimal policy:

# /etc/kubernetes/audit-policy.yaml
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  # Don't log reads to health endpoints
  - level: None
    nonResourceURLs: ["/healthz", "/readyz", "/livez"]
  
  # Don't log kube-system service account activity (too noisy)
  - level: None
    userGroups: ["system:nodes"]
    verbs: ["get", "list", "watch"]
  
  # Log ALL exec and portforward at full detail
  - level: RequestResponse
    resources:
      - group: ""
        resources: ["pods/exec", "pods/portforward", "pods/log"]
  
  # Log secret access metadata (not content)
  - level: Metadata
    resources:
      - group: ""
        resources: ["secrets", "configmaps"]
  
  # Log all other writes
  - level: Metadata
    verbs: ["create", "update", "patch", "delete"]
  
  # Don't log read-only operations on non-sensitive resources
  - level: None
    verbs: ["get", "list", "watch"]

Kubernetes Auditing— Kubernetes Docs Falco Default Rules— CNCF Falco

Retrieval Practice

You see a Kubernetes audit event: verb=create, resource=pods, namespace=kube-system, responseStatus.code=201. The pod spec includes hostNetwork=true and privileged=true. Why is this critical, and what ATT&CK technique does it represent?

Reveal answer

Creating a privileged pod with hostNetwork=true in kube-system is a container escape — the pod has full network access to the host's network stack and all capabilities of a privileged container, which can be used to escape to the underlying node (T1610 — Deploy Container). kube-system pods running with host networking and privileged mode can: mount the host filesystem, access the kubelet credentials at /etc/kubernetes/pki/, and pivot to the cloud provider metadata service (169.254.169.254) to steal instance role credentials. This is one of the most critical Kubernetes alerts — a privileged pod in kube-system represents near-complete cluster compromise.

Retrieval Practice

Your Kubernetes audit logs show: user.username='system:serviceaccount:default:default', verb=list, resource=secrets, namespace=production, responseStatus.code=403. Is this a successful attack? What should you investigate?

Reveal answer

The 403 means the access was DENIED — the default service account in the default namespace doesn't have permission to list secrets in production. This is NOT a successful attack, but it IS a detection signal for: (1) an attacker probing permissions — they compromised a pod with the default service account and are testing what they can access; (2) a misconfigured application — though legitimate apps rarely need to list secrets via API. Investigate: which pod is running with the default service account? (kubectl get pods -n default -o json | jq '.[].spec.serviceAccountName'). Why is a pod from the default namespace calling the production namespace? Is the default service account used by many pods (making it a high-value target)?

Kubernetes Audit Log Structure

Key fields for detection

High-Priority Detection Patterns

1. Interactive Shell Execution (exec)

2. Secret Access

3. ClusterRoleBinding Creation (Privilege Escalation)

Configuring an Audit Policy

1. Interactive Shell Execution (`exec`)