Reading width
Wide uses the full column for everything, text, diagrams, code, and exercises. Narrow keeps the standard reading width.
Text size
Scales the body text. Headings and code blocks keep their size.
In this section
Linux Log Analysis: auth.log, the Journal and Audit Logs Read Against Each Other
Introduction
A Linux host does not have one log. It has several, written by different processes, kept in different formats and removed in different ways. The authentication log is a text file written by rsyslog; on Ubuntu it is auth.log and on RHEL it is secure.
The journal is a structured binary store kept by systemd's journald. The audit log is written by auditd, and records only what its rules ask for. Every application keeps its own.
An investigation that reads one of them reads one witness. This lesson reads them against each other on three hosts, and asks the question Module 3 asks of every record: what wrote this, and who could have removed it?
Four kinds of log on a Linux host, written by different processes and lost in different ways.
The figure is the map, and every lesson in Module 3 starts by placing its records on it. The most important thing it shows is that the four kinds are independent, so an intruder who empties one has not emptied the others, and an investigator who finds one empty has not found the end of the evidence.
Two Logs, One Sudo
The authentication log and the journal on a database serverSRV-NGE-DB01 in INC-NE-2026-1501 is a production database server whose admin account normally signs in with a key. On 26 March it signed in with a password. The authentication records for that hour:
10:15:40 auth_success dbadmin password from 10.1.10.15
10:20:00 sudo_success dbadmin as postgres: /usr/bin/pg_dump -Fc crm_productio
A password login at 10:15:40 from 10.1.10.15, and about four minutes later a sudo to the postgres account to run pg_dump on the CRM database. The account's other 9 logins in the case all used a public key, so the password is the anomaly, and it is the reason this alert fired.
The journal for the same hour holds more:
10:20:00 sudo dbadmin : TTY=pts/0 ; PWD=/tmp ; USER=postgres ; COMMAND
10:20:05 postgresql-15. connection authorized: user=postgres database=crm_produc
10:20:10 postgresql-15. statement: COPY public.clients TO stdout
10:22:00 postgresql-15. statement: COPY public.invoices TO stdout
10:24:00 postgresql-15. statement: COPY public.payment_details TO stdout
10:27:00 postgresql-15. disconnection: session time: 0:07:00 user=postgres datab
10:30:00 sudo dbadmin : TTY=pts/0 ; PWD=/tmp ; USER=root ; COMMAND=/us
The journal has the same sudo with the same command line, because sudo's record reaches both. It also has what PostgreSQL said: a connection authorized for pg_dump, then 3 COPY statements naming the tables exported, then the disconnection seven minutes later. And at 10:30:00 it has a second sudo that the authentication records do not: scp, run as root, copying the compressed export to another host.
So the two logs overlap and differ. Where they overlap, each confirms the other. Where they differ, the journal is the fuller record here, because it holds every unit's output, while the authentication records hold only what PAM and sshd report.
That is not a universal rule, and it is worth knowing why before relying on the journal anywhere; on a host where the journal is not kept on disk it is lost at every reboot, and the text log is the survivor. Module 3's first lessons on the logging architecture and the journal are about knowing which is which before trusting either.
The table names are what the business needs. The export included tables called clients, invoices and payment_details, and the only record of that is a database server's own log of its statements, collected through the journal. Without it the finding would be "a dump was taken"; with it the finding says what was in it.
Notice also where the password login came from: 10.1.10.15, the same address the account's ordinary key logins come from every morning. The source address did not change; the authentication method did.
A detection that watched only for new addresses would have missed this entirely, and the reason the alert fired at all is that someone wrote a rule about the method. Module 3's authentication lesson is largely about which fields in these records carry that kind of signal, and Module 7 follows the address back to the machine it belongs to.
What the Journal Held
A malicious package and the commands its payload ranThe second host is SRV-NGE-DEV03 in INC-NE-2026-0491, a development server. Its developer ran npm install on 19 March and noticed nothing. The journal from that moment:
14:31:05 user@1001.service > lodassh@1.0.3 postinstall /opt/apps/nge-portal
14:32:00 systemd Created symlink /etc/systemd/system/timers.targe
14:32:05 systemd Started node-telemetry.timer - Node.js Telemetry
14:33:00 node-telemetry.service Started Node.js Telemetry Collection
14:33:02 node-telemetry.service Connecting to telemetry endpoint...
14:35:10 node-telemetry.service exec: id; uname -a; cat /etc/hostname
14:36:00 node-telemetry.service exec: find / -name ".env" -type f 2>/dev/null
14:37:00 node-telemetry.service exec: cat /opt/apps/nge-portal/.env
14:38:00 node-telemetry.service exec: cat /opt/apps/nge-api/.env
At 14:31:05 npm ran the postinstall script of a package called lodassh, in the developer's user session. A minute later systemd created a timer, node-telemetry.timer, and started it. The service it runs started at 14:33 and printed that it was connecting. Then it printed every command it was given: id, uname and hostname, a search of the whole disk for .env files, and the contents of two of them.
Those last lines are in the journal for one reason: the payload wrote them to its standard output, and systemd sends every service's output to the journal. A more careful attacker would have printed nothing, and the journal would have shown a service starting and nothing else.
The lesson is not that the journal records commands; it is that it records whatever a unit writes, and here that happened to include the attacker's own working.
The package's name is the other finding. lodassh is one letter from lodash, one of the most widely installed JavaScript libraries, and the files the package dropped sit beside the application's own with nothing but the name to mark them. Module 3's lesson on system, service and application logs is where an install like this is traced from the package manager's records as well as the journal.
The journal also attributes each record to a unit, which is worth more than it looks. The postinstall line belongs to the developer's user session, the timer lines to systemd itself, and the exec lines to node-telemetry.service.
That attribution is set by journald from the process that wrote the line, not by the line's text, so a payload can print anything it likes but cannot easily make its output appear under another unit. Filtering by unit is how an investigator separates a compromise from the ordinary noise around it.
The timer file itself was written at 14:32:00 and owned by root, so the install that the developer ran as an ordinary user ended in a root-owned unit. How a postinstall script got root is a privilege question for Module 5; the journal only says that it did.
A Timer Read From Its Starts
Cadence as evidenceThe journal records every start of node-telemetry.service. Counting them by hour on 19 March:
14:00 to 14:59 #
15:00 to 15:59 ####
16:00 to 16:59 ####
17:00 to 17:59 ####
18:00 to 18:59 ####
19:00 to 19:59 ###
20 starts, fifteen minutes apart after the first, from 14:33 until the last record at 19:33. That regularity is a timer's signature, and it matches the timer the journal saw being created. Without the unit file, the cadence alone would have said "something scheduled every fifteen minutes", which is enough to know where to look.
The record also says where it stops. The last start in the case is at 19:33 on 19 March, and the journal continues to 2026-03-22 with the application's own daily line. Either the timer stopped or the collection recorded only part of its runs; the records here cannot say which, and a finding that claimed the payload ran until the collection would be claiming more than the journal shows.
Cadence is useful far beyond timers. A beacon, a cron job, a brute-force tool and a backup all produce events at their own rhythm, and counting events per hour or per minute is the quickest way to see a rhythm that should not be there. Module 3's correlation lesson builds on it, and Module 6 uses it to tell a persistence mechanism that runs from one that was only installed.
There is one more thing in this journal: the daily pm2 line at 09:30 shows the developer's application was online every day before and after. Nothing about the compromise disturbed it. Most persistence is designed that way, and the journal's job is to show the second rhythm running beside the first.
Cadence also helps with something investigators are often asked: how long did this run? Each start is a moment the payload was alive and, here, a moment it could reach its controller. Counting starts gives a lower bound on the number of times it ran, and the first and last give the window.
A lower bound, because a start that the journal lost, or that fell outside what was collected, is simply not counted; the true number can only be higher. Those are the kind of figures a report needs, and they come straight from a count over the journal.
What No Log Records
The silence an investigator has to plan forThe first three sections showed logs recording more than one might expect. This one is about what they do not record by default, on either distribution.
What the default logs do not record
Silence by defaultThe authentication log records logins, sudo, su and account changes. It does not record what anybody did after logging in. A shell does not log its commands anywhere but the user's history file, which the user can edit or clear, as lesson 0.1 showed.
Reading a file leaves no log at all unless an audit rule watches it, so a database export copied with cat or a key file read with less leaves nothing behind. And an outbound connection is recorded only if a firewall or an audit rule records it.
That is why the case on SRV-NGE-DEV03 had the commands only because the payload printed them, and why the database export on SRV-NGE-DB01 was visible only because PostgreSQL logs statements. In both cases an application happened to keep a record the operating system did not. Most investigations are not that lucky.
Module 3's audit lesson is the answer: auditd can record process execution, file access and network calls, but only for the rules it is given, and only from the moment they are loaded. An estate that turns it on after an incident has evidence about the next one. Module 11's readiness lessons are about deciding, before an incident, which of these gaps an estate is willing to leave open.
The practical habit is to write down, at the start of any investigation, which of these sources exist on the host in front of you. A finding that says nothing was read from a directory is only as good as the record that would have shown it, and on most default hosts there is no such record.
The same limit applies in the other direction. When a case does hold a record of a file read or a command run, ask which source produced it, because the answer decides how much weight it carries.
An application's own log is written by software the attacker may control; an audit record is written by the kernel's audit subsystem; a shell history is written by the user. They are not equally trustworthy, and Module 3 ranks them.
What Root Can Remove
An audit log emptied, an authentication log keptThe third host is SRV-NGE-BRS-DB01 in INC-NE-2026-0612, a RHEL database server where an intruder gained root. The last two lines of the account's history are its attempt to leave nothing behind:
02:35:15 build-agent > /var/log/audit/audit.log
02:35:30 build-agent history -c
The first line truncates the audit log to nothing; the second clears the shell's history. Both are things root can do, and both remove evidence. But the intruder emptied one log out of several. The authentication records that the collection kept still hold the entry:
02:14:01 to 02:17:41 auth_failure x45 from 45.33.32.156
03-16 02:18:22 auth_success sshd build-agent from 45.33.32.156
03-16 02:19:30 sudo_failure sudo build-agent
03-16 02:24:45 auth_success su root from local
03-16 02:26:10 useradd useradd sysadm
03-16 02:28:30 auth_success sshd sysadm from 45.33.32.156
45 failed password attempts from 45.33.32.156 between 02:14:01 and 02:17:41, then a successful login, a refused sudo, a root session, an account created with UID 0, and that account logging in by key from the same address. The audit log is gone, and the authentication log tells the story anyway, from the first attempt to the backdoor's first use.
Read the failed attempts for their targets as well as their number. They tried deploy, ci, automation, admin, jenkins, root, build and build-agent, the names of service and automation accounts an attacker expects on a build or database server, and the one that worked was build-agent.
Module 4 reads that pattern as the signature of a guessing tool working through a list, and Module 5 starts from the account it found.
Two lessons follow. An intruder with root can remove any log on the host, so the logs that survive are the ones they did not think of or did not have time for, and checking every one is never wasted.
And a log that is empty, or shorter than its neighbors, is itself a finding: Module 3's integrity lesson covers how to tell a log that was emptied from one that was never written, using sizes, rotation and the records that refer to it.
The strongest defense is a copy the host cannot reach. Logs forwarded to a central collector as they are written are out of an intruder's hands before they gain root, and Module 3 and Module 11 both return to that as the one control that makes log tampering on the host irrelevant.
The order of the two lines is worth a moment too. The audit log went first and the history second, and the authentication log was never touched.
That is a person working from a checklist they half remembered, not a tool designed to clean up, and it is common. Most intruders remove what they think records them and miss the rest, which is why the rest is where an investigation looks first.
What Module 3 Builds On It
Log analysisModule 3 starts with the logging architecture, so that every record can be traced to the process that wrote it, then takes authentication logs, the journal and audit records in turn. It reads web server and application logs, correlates them, tests their integrity, and asks what a timestamp actually means before ending with a workflow.
What Module 3 builds
Module 3Each section here is a short version of one of those. The database server's two logs are the architecture and correlation lessons. The development server's journal is the journal and application log lessons. The timer's cadence is correlation again. The silent defaults are the audit lesson's reason for existing, and the emptied audit log is the integrity lesson.
The habits to carry into it are the ones this lesson used, and each one costs a few minutes at the start of an investigation rather than hours at the end. Name the process that wrote each log.
Read every log on the host, because they overlap and they fail separately. Count events over time to find rhythms. List the sources a host does not have before claiming anything did not happen. And treat an empty log as a fact to be explained.
Module 3's material is collected logs from lab hosts where you control what happens, so that each record can be matched to the action that produced it. That ground truth matters most for the journal and for auditd, whose output depends heavily on configuration, and where a default lab host and a production server can produce very different records for the same action.
It also prepares for the modules after it. Module 4 reads the web and SSH logs for the entry point, Module 5 reads sudo and su records for the escalation, Module 6 reads the journal for units that start, and Module 7 reads authentication records across hosts. Each of them assumes the reading this module teaches.
Module 3 also takes the distribution split seriously, because this is where it matters most. The file names, the default journal settings and the audit configuration all differ between Ubuntu and RHEL, and every lesson shows both.
Practice
Decide what the journal says prepared evidenceThe prepared evidence below is the journal on SRV-NGE-DEV03 from 17 March to 15:00 on 19 March. Decide which record is the first sign of the compromise, which records show commands the intruder ran, and why the developer saw nothing. Each answer comes with its reasoning, so check it against the rows rather than only the choice.
Read the unit column first. Before 19 March the only unit is the developer's application; anything that appears after the install under a new unit name needs to be explained before its messages are read.
Then read the times. The ordinary records arrive once a day at the same minute. The records you are looking for arrive in a burst over a few minutes on one afternoon, which is the shape most intrusions have in any log. Once the burst is found, the record just before it is usually the cause and the records just after are usually the consequences.
Then, on a lab host, run journalctl with the unit filter for one service and again without it, and see how much of what a service prints lands in the journal. Then check whether your journal is kept on disk or only in memory, because that one setting decides whether any of it survives a reboot.
Finally, run sudo once and find its record in both auth.log or secure and the journal, and compare the fields each one keeps.
A Linux host keeps several logs that overlap and fail separately. On the database server the journal held the tables taken and a transfer the authentication records missed; on the development server it held a malicious install and the payload's own commands; on the third server an intruder emptied the audit log and the authentication log kept the whole entry.
No default log records what a user does in a shell, and every finding has to say which of these sources it rests on and which it could not have had.
The next lesson takes the first question of every investigation, how the intruder got in, and reads both doors: the web application and SSH.