Linux Forensic Artifacts Reference
Where the evidence lives on a compromised Linux host: filesystem timestamps, authentication logs, persistence, process evidence, and the anti-forensics tells. Paths and commands for RHEL and Ubuntu, with what each one reveals. No account needed.
Paths differ across distributions: Debian and Ubuntu use /var/log/auth.log, RHEL and its derivatives use /var/log/secure. Where a path or behaviour diverges, both are noted. Capture volatile state before touching disk, and prefer reading from a mounted image where the investigation allows it.
Filesystem and timestamps
Every file carries four inode timestamps, and the relationships between them are where tampering shows. On ext4 the inode also records a creation time (crtime); xfs records it too, but classic tools do not all surface it. The key tell: a modification time earlier than the creation time is impossible in normal operation and indicates a backdated (timestomped) file.
# Read all timestamps on a file (atime, mtime, ctime)
stat /path/to/file
# Read the inode directly, including crtime (ext4), via debugfs on the raw device
debugfs -R "stat <inode>" /dev/sdaN
debugfs -R "stat /path/to/file" /dev/sdaN
# Find files modified in a window (e.g. the incident hour)
find / -newerct "2026-03-15 02:00:00" ! -newerct "2026-03-15 03:00:00" -type f 2>/dev/null| Timestamp | Meaning / investigation use |
|---|---|
| atime | Last access. Often unreliable (relatime/noatime mounts suppress updates). |
| mtime | Last content modification. The headline "when was this changed". |
| ctime | Last inode change (permissions, ownership, links). Harder for an attacker to forge than mtime. |
| crtime | Creation time (ext4, via debugfs). mtime earlier than crtime = timestomping. |
debugfs reads crtime and the journal can hold a before-image of a recently changed inode. On xfs the equivalent is xfs_db, and recovery relies more on carving than journal replay. Confirm the filesystem with findmnt before choosing your approach.A web shell .cache.php shows an mtime of 2024 (looks old, blends in), but debugfs reads its crtime as 02:09 on the incident date. The crtime, which the attacker did not think to forge, places the file's true creation inside the compromise window. The 2024 mtime is the backdate.
Who created it, or that it executed. mtime alone would have hidden it entirely.
Web access log for the request that dropped it; the journal before-image for the inode; other files with the same crtime minute.
auditd: syscall-level evidence
Where syslog records what services chose to report, auditd records what the kernel actually did: the syscalls, the file accesses, the executions. If it was configured before the incident, it is the highest-fidelity log on the host, and unlike the text logs it is keyed and queryable. Its absence is itself worth noting, because most compromised hosts never had it enabled.
# Search the audit log by key, time, or syscall
ausearch -k persistence # events tagged by a watch rule
ausearch -ts 02:00 -te 03:00 # by time window
ausearch -f /etc/passwd # every access to a watched file
ausearch -m EXECVE # every command execution
# Summary report of activity
aureport --summary ; aureport -x --summary # by executable
# The rules that define what is captured
cat /etc/audit/audit.rules ; ls /etc/audit/rules.d/| Artifact | What it tells you |
|---|---|
| ausearch -m EXECVE | Every command executed, with arguments. The execution record syslog does not keep. |
| ausearch -f /etc/shadow | Every read/write of a watched sensitive file: credential theft, account tampering. |
| /etc/audit/rules.d/ | The watch rules in force. An attacker who disabled auditd leaves a gap here. |
Authentication and access logs
Authentication is the first thing to reconstruct: who logged in, from where, and when. The text logs record the SSH and sudo story; the binary accounting files record the session story. Read both, because an attacker who edits one often forgets the other.
# Text auth log (Debian/Ubuntu vs RHEL)
grep -E "Accepted|Failed|sudo" /var/log/auth.log # Debian/Ubuntu
grep -E "Accepted|Failed|sudo" /var/log/secure # RHEL/derivatives
# Binary accounting: successful logins, failed logins, current sessions
last -f /var/log/wtmp # login/logout history
lastb -f /var/log/btmp # failed login attempts
who -a /var/run/utmp # currently logged-in sessions
# Same events through journald (systemd hosts)
journalctl _COMM=sshd
journalctl _COMM=sudo --since "2026-03-15 02:00" --until "2026-03-15 03:00"| Artifact | What it tells you |
|---|---|
| /var/log/auth.log | (Debian/Ubuntu) SSH accepts/failures, sudo use, session open/close. |
| /var/log/secure | (RHEL) the same authentication record. |
| /var/log/wtmp | Login/logout history. Read with last. |
| /var/log/btmp | Failed login attempts. Read with lastb. Brute-force evidence. |
| /var/run/utmp | Currently active sessions. Read with who. |
Persistence locations
Persistence is the highest-value thing to enumerate: finding it often means finding a foothold you did not know about. Linux offers many mechanisms, and a thorough responder checks all of them, because attackers frequently plant more than one and expect you to find only the obvious.
# Accounts: UID 0 duplicates, new accounts, login shells where there should be none
awk -F: '($3==0){print}' /etc/passwd # any non-root UID 0 = backdoor
grep -vE "/nologin|/false" /etc/passwd # accounts with a real shell
# SSH keys (per-user and root)
cat /root/.ssh/authorized_keys
find /home -name authorized_keys -exec ls -la {} \;
# Scheduled tasks: cron and at
cat /etc/crontab /etc/cron.d/* ; ls -la /etc/cron.daily /etc/cron.hourly
ls -la /var/spool/cron/ /var/spool/cron/crontabs/ # per-user crontabs
ls -la /var/spool/at/ # one-off at jobs
# systemd services and timers
ls -la /etc/systemd/system/ ; systemctl list-timers --all
systemctl list-units --type=service --state=running
# Shell init (runs on every login/shell)
cat /etc/profile /etc/bash.bashrc ; ls -la /etc/profile.d/
cat ~/.bashrc ~/.bash_profile ~/.profile ~/.bash_login
# Library, PAM, and kernel-level persistence (deeper, often missed)
cat /etc/ld.so.preload # preloaded shared objects = hooking
ls -la /etc/pam.d/ ; ls -la /usr/lib64/security/
cat /proc/modules ; ls /sys/module/ # loaded kernel modules / rootkits| Location | Persistence mechanism |
|---|---|
| /etc/passwd, /etc/shadow | Backdoor accounts, duplicate UID 0, unexpected login shells. |
| ~/.ssh/authorized_keys | Attacker public key grants password-free re-entry. |
| /etc/crontab, /etc/cron.d/ | Scheduled execution. A misnamed job (e.g. ntp-sync) hiding a payload. |
| /var/spool/cron/ | Per-user crontabs, often missed when only /etc/cron* is checked. |
| /etc/systemd/system/ | Malicious service or timer units with innocuous names. |
| /etc/ld.so.preload | Library preloading: userland rootkit hooking every process. |
| /etc/pam.d/, /usr/lib64/security/ | Trojaned PAM module capturing credentials or granting access. |
| /proc/modules, /sys/module/ | Loaded kernel modules. A hidden module is a kernel rootkit. |
/var/spool/cron/ and a user's ~/.bashrc, because responders check the system-wide locations and stop. An attacker who owns one account hides there precisely for that reason.A host has a cron job /etc/cron.d/ntp-sync (plausible name) calling a script in /usr/local/bin, an attacker key in backup@ops's authorized_keys, and a SUID .upd binary. Each alone might be dismissed; together they are three independent footholds, the attacker expecting you to find one and stop.
Which came first, or that they are the same actor, until you correlate timestamps.
ctime on all three to order them; auth log for the backup@ops key's first use; remove all three, not the first one found.
Execution and process evidence
What ran, what is running, and what is hiding. The live process tree and /proc are ground truth for a running host; on disk, command history and SUID binaries carry the execution story. Staging directories are where attackers drop tooling, and they favour world-writable and dot-prefixed paths to stay out of casual view.
# Live process state from /proc (the running truth)
ls -la /proc/<pid>/exe # real binary path (even if deleted: "(deleted)")
cat /proc/<pid>/cmdline # full command line
ls -la /proc/<pid>/cwd # working directory
# SUID/SGID binaries (privilege-escalation surface)
find / -perm -4000 -type f 2>/dev/null # SUID; flag any outside standard paths
find / -perm -2000 -type f 2>/dev/null # SGID
# Shell history (per user)
cat ~/.bash_history ; cat /root/.bash_history
# Common attacker staging areas (world-writable + hidden)
ls -la /tmp /var/tmp /dev/shm
ls -la /var/tmp/.ICE-unix/ /opt/.systemd-private/ /usr/local/.cache/ # dot-prefixed = hiding| Artifact | What it tells you |
|---|---|
| /proc/<pid>/exe | The real binary behind a running process. A "(deleted)" target = malware that unlinked itself. |
| find -perm -4000 | SUID binaries. A SUID file in /usr/local/bin or a dot-name (e.g. .upd) is a planted escalation. |
| ~/.bash_history | Commands the attacker typed, unless wiped. A truncated or missing history is itself a tell. |
| /tmp, /var/tmp, /dev/shm | World-writable staging. /dev/shm (RAM-backed) leaves nothing on disk after reboot. |
Network and volatile state
Network state is volatile: it lives in kernel memory and is gone at shutdown, so it must be captured live, before containment. Active connections expose the C2 channel and lateral pivots; the listening sockets expose backdoors waiting for a connection. Map every connection back to the process and the binary behind it.
# Active and listening sockets, with the owning process (-p)
ss -tunap # TCP/UDP, numeric, all states, with PID/process
ss -tlnp # listening TCP sockets (backdoor listeners)
# Open files and sockets per process
lsof -i # every network connection to a process
lsof -p <pid> # everything a suspect process has open
# Map a connection to its binary (even if the binary was deleted)
ls -la /proc/<pid>/exe # "(deleted)" = process running from an unlinked file
cat /proc/net/tcp # raw kernel connection table (hex, survives ss tampering)| Artifact | What it tells you |
|---|---|
| ss -tunap | Every active connection with its process. The established connection to an external IP is the C2 or exfil channel. |
| ss -tlnp | Listening sockets. A listener on an odd port owned by a non-service process is a backdoor. |
| /proc/<pid>/exe | The binary behind a connection. A "(deleted)" target is malware that unlinked itself to evade disk forensics. |
| /proc/net/tcp | Raw kernel connection table. Read it directly when a trojaned ss/netstat may be lying. |
Memory: ground truth
Memory is the one source a rootkit cannot fully lie to: the processes, network connections, and injected code that userland tools are tricked into hiding are still present in RAM. It is volatile, so it is captured first, with a tool that leaves the smallest footprint. Analysis is offline, against the captured image.
# Acquire RAM to an image. LiME (loadable kernel module) is the common choice;
# AVML is a static binary that needs no matching kernel headers.
insmod lime.ko "path=/mnt/evidence/mem.lime format=lime"
avml /mnt/evidence/mem.raw
# Analyse offline with Volatility 3 (no profile needed; uses symbol tables)
vol.py -f mem.lime linux.pslist # processes from kernel structures
vol.py -f mem.lime linux.pstree # parent/child tree
vol.py -f mem.lime linux.psscan # scan for processes hidden from pslist
vol.py -f mem.lime linux.check_syscall # hooked syscalls (rootkit tell)
vol.py -f mem.lime linux.bash # recover bash history from memory| Volatility 3 plugin | What it recovers |
|---|---|
| linux.pslist / pstree | Processes from kernel task structures, and their parentage. |
| linux.psscan | Processes found by scanning memory: a process here but not in pslist is hidden by a rootkit. |
| linux.check_syscall | Syscall table hooks: the kernel-rootkit signature. |
| linux.bash | Shell history from RAM, recoverable even when ~/.bash_history was wiped on disk. |
On the live host, ps and ss show nothing unusual, but a connection to the C2 IP keeps appearing in the firewall logs. linux.psscan against the memory image surfaces a process that linux.pslist (and therefore ps) does not: the rootkit unlinked it from the kernel task list. Its exe points at a deleted file.
What the process did, until you carve its memory and recover the binary.
linux.check_syscall for the hook that hid it; dump the process memory; recover the deleted binary from the image for static analysis.
Web and application logs
On a server compromise, the web logs are often the breach point. The access log records the request that dropped or triggered the web shell; the error log frequently captures the stack traces and file paths that the access log alone does not.
# Nginx
/var/log/nginx/access.log /var/log/nginx/error.log
# Apache (Debian/Ubuntu)
/var/log/apache2/access.log /var/log/apache2/error.log
# Apache (RHEL)
/var/log/httpd/access_log /var/log/httpd/error_log
# Hunt for web-shell access patterns: POSTs to odd PHP, suspicious user agents
grep -E "POST .*\.php" /var/log/nginx/access.log | grep -vE "wp-admin|index\.php"
grep -E "cmd=|exec=|eval\(|base64_decode" /var/log/nginx/access.log| Artifact | What it tells you |
|---|---|
| access.log | Every request, with source IP, method, path, and user agent. The web-shell request lives here. |
| error.log | Stack traces and file paths from failed/exploited requests, often the clearer record. |
Container and cloud
A compromised Linux host is increasingly a container or a cloud instance, and the evidence extends beyond the box. Containers are ephemeral, so the runtime layer and the host's view of them matter; cloud instances carry an extra entry vector in the metadata service, which an SSRF or a foothold can turn into stolen credentials.
# Cloud instance metadata service: the SSRF/credential-theft target
curl http://169.254.169.254/latest/meta-data/iam/security-credentials/ # AWS role creds
# Access to this endpoint from a web-app process = likely SSRF credential theft
# Cloud-init: what configured the instance at first boot (attacker user-data?)
cat /var/log/cloud-init.log /var/log/cloud-init-output.log
# Container runtime: running containers, images, and the writable layer
docker ps -a ; docker images
ls -la /var/lib/docker/containers/ # per-container logs + config
ls -la /var/lib/docker/overlay2/ # the writable filesystem layers
# Kubernetes audit + pod state
kubectl get pods -A ; kubectl logs <pod>
ls /var/log/containers/ /var/log/pods/| Artifact | What it tells you |
|---|---|
| 169.254.169.254/... | Instance metadata. Access from a web process is the SSRF-to-credential-theft signature. |
| /var/log/cloud-init.log | First-boot configuration. Malicious user-data or an unexpected provisioning step shows here. |
| /var/lib/docker/overlay2/ | A container's writable layer: files the attacker wrote inside the container persist here on the host. |
| /var/log/containers/ | Per-pod stdout/stderr on a Kubernetes node, often the only record after a pod is deleted. |
kubectl/docker ps, but its overlay layer, logs, and the node's auth/audit records survive on the host. When the container is gone, pivot to the host's view of it.Anti-forensics indicators
A capable attacker cleans up, and the cleanup is itself evidence. Truncated logs, wiped history, securely deleted files, and backdated timestamps are not the absence of evidence, they are the presence of a specific kind of evidence: someone who knew what to remove.
| Indicator | What it means |
|---|---|
| Truncated /var/log/auth.log | A log that suddenly starts (or ends) mid-incident was edited. Cross-check against journald and wtmp. |
| Missing/short ~/.bash_history | History wiped or redirected to /dev/null. Expected size for an active account is large. |
| shred / secure-delete traces | Files deliberately overwritten, not just removed. The intent to destroy is the finding. |
| mtime earlier than crtime | Backdated file (timestomping). Compare stat output against debugfs crtime. |
| Gaps in journald sequence | journald records a monotonic sequence; a gap indicates removed entries. |
| chattr +i on attacker files | Immutable flag set to resist deletion. Check with lsattr. |
Quick lookup
| Question | Where to look |
|---|---|
| Who logged in, from where? | auth.log / secure, wtmp (last), journalctl _COMM=sshd |
| What failed to log in (brute force)? | btmp (lastb), auth.log "Failed password" |
| How did they persist? | crontab + /var/spool/cron, systemd units, authorized_keys, ld.so.preload, /proc/modules |
| How did they escalate? | SUID/SGID (find -perm -4000), sudo entries in auth log, ctime cluster |
| What ran / is running? | /proc/<pid>/exe + cmdline, bash_history, Prefetch-equivalent: none, use process + history |
| Where did they stage tooling? | /tmp, /var/tmp, /dev/shm, dot-prefixed dirs under /opt /usr/local |
| When did it happen? | find -newerct window, stat/debugfs timestamps, log timestamps |
| Did they cover tracks? | Truncated logs, short history, shred traces, mtime < crtime, chattr +i |
| What is talking to the network? | ss -tunap, lsof -i, /proc/net/tcp, map to /proc/<pid>/exe |
| Is something hidden from ps? | Memory image + Volatility linux.psscan vs pslist; linux.check_syscall |
| Was this a cloud/container compromise? | 169.254.169.254 access, cloud-init logs, /var/lib/docker/overlay2, /var/log/containers |
| What syscalls/executions ran? | auditd: ausearch -m EXECVE, ausearch -f <file> |
From artifacts to a defensible Linux investigation
This reference shows where the evidence lives. Linux Endpoint Investigation teaches the method: reconstructing a server compromise end to end across the filesystem, logs, and memory, and proving what happened. Built by practitioners who still do the job.
Explore the course