In this section

Linux Live Response and Memory Forensics: Hidden Processes, Hooked System Calls and Volatile Evidence

Module 0

Introduction

Every live collection in this module was taken hours after the intrusion it describes, from a host the intruder still controlled. That is the normal situation for live response, and it has two consequences. First, much of what the intruder did had already left the live state. Second, the tools used to collect ran on a system that could lie to them.

This lesson takes both in turn. It starts with how late the collections in this module's cases were, then reads two memory captures of one server to show what a live collection on a compromised host can miss, what memory keeps that the disk has lost, and what the responder's own work leaves behind.

1. Memoryprocesses, modules, keys, history: lost at power-off2. Live stateprocess list and sockets: change with every command3. Logs and disksurvive, but root can change them4. Your own stepsevery command a responder runs is evidence

An order of collection for a host that cannot be switched off.

The figure is an order of collection, from most volatile to least, and Module 9 measures it rather than taking it on trust. Its last line is the one most often skipped: the responder is on the host too, and every command they run changes it and becomes part of the evidence.

01

How Late the Collections Were

Hours between the intrusion and the first look

Two of the cases this module has read recorded both the start of the intrusion and the moment of collection:

INC-NE-2026-0310  first request 03-10 02:10   collection 03-10 15:00   12h 49m later
INC-NE-2026-0422  first injection 04-22 02:48 collection 04-22 04:00   1h 11m later

On the web server, the collection ran nearly thirteen hours after the intruder's first request. On the Kubernetes node, a little over an hour after the template injection.

In both, the live records showed only what was still running at that moment: the reverse shells, the persistence, the established connections. Everything the intruders ran and finished in between was visible only through what it wrote to disk, as lessons 0.1 and 0.8 found.

That gap is not a failure of the people collecting. Intrusions are noticed late, and the first person to look is rarely the first person to arrive. What matters is that the collection says when it ran, so that every absence in it can be read as a fact about that moment. Both of these did, with a capture time on every process record.

The gap also changes what a live collection is for. Thirteen hours after the entry, the live state cannot show the entry; it can show what the intruder left running and what they are doing now.

The live collection is therefore mostly about the present and the persistence, and the past has to come from the disk and the logs. Module 9 is explicit about that division, because a responder who expects the live state to explain the intrusion will be disappointed, and one who skips it will miss what is still running.

The other consequence is less obvious. By the time of collection, both intruders had root. Every tool the responder ran, ps, netstat, ls, cat, ran on a kernel the intruder could have modified. In these two cases nothing suggests they had, and the live records agreed with the disk. The rest of this lesson reads a server where that was not true.

Module 9 starts from exactly this position: a host that cannot be switched off, an intruder who may still be present, and a collection that has to be taken anyway. Its first lessons are about what the responder's own commands change, and about measuring the order of volatility rather than reciting it.

02

What ps Could Not See

A responder's check, and the task list in memory

The Memory Range holds two captures of an application server, app01, taken about forty minutes apart on 24 March. Like every capture in the range they are simulated from a written scenario, with each plugin's real output format. At 15:01, a responder logged in to the live host. The shell history that bash kept in memory for that session:

$ vol -f NE-APP01-1505.lime linux.bash --pid 4993 | grep UTC | cut -f3,4 \
    | cut -c12-19,31-
Progress:  100.00		Stacking attempts finished
15:02:30	ps aux | grep -i beacon
15:03:12	lsmod | head -5
15:05:51	sudo insmod /media/ir/lime-6.8.0-52-generic.ko path=tcp:4444 format=lime

At 15:02 the responder ran ps aux and filtered it for beacon, because the host was known to be connecting out. It found nothing. At 15:03 they listed the most recent kernel modules. At 15:05 they loaded LiME, a kernel module that writes the whole of memory out over the network, and the second capture is the image it took.

The task list in that image, filtered to the processes that matter:

$ vol -f NE-APP01-1505.lime linux.pslist | grep -e 2092 -e 4820 -e 4993 -e 5012 \
    | cut -f2,4-6,10
Progress:  100.00		Stacking attempts finished
2092	2090	bash	0	2026-03-24 14:01:58.630922 UTC
4820	2092	evil_beacon	0	2026-03-24 14:33:05.002771 UTC
4993	4992	bash	1002	2026-03-24 15:01:47.611340 UTC
5010	4993	sudo	1002	2026-03-24 15:05:51.404217 UTC
5012	5011	insmod	0	2026-03-24 15:05:51.442611 UTC

PID 4820, evil_beacon, running as root since 14:33, a child of the attacker's root shell, PID 2092. Its entry is in the kernel's task list, which Volatility reads directly from memory. It was not in the output of ps aux on the live host. And the other two lines are the responder: bash 4993, logged in as user 1002, and the insmod that loaded LiME.

So the live check was not careless; it was answered by a system that had been told to hide this process.

There were other signs available on the live host, and they are worth listing because they are what a responder can check without memory. The host's own sockets would have shown an established connection to 203.0.113.66 with no process the responder could see behind it, because the hook hid the process's /proc entry and not the socket in the kernel's tables.

The process count from the kernel would have been one higher than the number of /proc directories ps could list. Each of those is a disagreement between two views of the same kernel, and Module 9 teaches looking for exactly that kind of disagreement before trusting any single view.

Module 9's lesson on concealment, live, is about checks that do not depend on the same mechanism the intruder could have tampered with, and Module 10's lesson on recovering what was removed from the list is about finding hidden processes in memory.

03

Why the Live Tools Lied

A module, and one system call that pointed into it

How the process was hidden is in the same image. The most recent kernel modules, then the system call table entries that do not point into the kernel's own code, with one ordinary entry for comparison:

$ vol -f NE-APP01-1505.lime linux.lsmod | head -6 | cut -f2-4
Progress:  100.00		Stacking attempts finished
Volatility 3 Framework 2.28.2

Module Name	Code Size	Taints

lime	0x5000	OOT_MODULE,UNSIGNED_MODULE
hideproc	0x4000	OOT_MODULE,UNSIGNED_MODULE
$ vol -f NE-APP01-1505.lime linux.malware.check_syscall \
    | grep -e UNKNOWN -e "	78	0xffffffff9a302080" | cut -f2-5
Progress:  100.00		Stacking attempts finished
64bit	78	0xffffffff9a302080	__x64_sys_getdents
64bit	217	0xffffffffc0a81180	UNKNOWN

The two most recently loaded modules are lime, the responder's, and hideproc, which the attacker's history shows was loaded with insmod at 14:32. Both are tainted as out of tree and unsigned, meaning neither came from the distribution's kernel.

And one entry in the 64-bit system call table, number 217, has a handler address in the area where modules are loaded rather than in the kernel's text, with no kernel symbol to name it.

Entry 217 on this architecture is getdents64, the call that lists the entries in a directory. ps finds processes by listing /proc, so a getdents64 that leaves out one directory makes one process vanish from ps, top and anything else that lists /proc, while the process keeps running. The attacker even checked it worked: their own history ends with ps aux | grep -c evil_beacon.

This is the central problem of live response, made concrete. Every live tool is a program asking the running kernel questions, and a kernel the intruder has modified answers them however the intruder chose. Memory acquisition sidesteps it by taking the kernel's own structures and reading them somewhere else, with a tool the intruder never touched.

The acquisition itself still runs on the compromised kernel, which is worth being honest about. LiME is loaded into that kernel and reads its memory; a rootkit written to defeat acquisition could interfere.

Nothing in this image suggests that happened, and the image is internally consistent, with the hook visible in it. But Module 10's lesson on acquiring memory without destroying it covers how to check an image for signs that the acquisition was tampered with, because an image is only as good as the kernel it was taken through.

Two cautions follow. A memory image is not a cure-all: Module 10 covers rootkits that leave the system call table clean. And the module list in the live host's lsmod would have shown hideproc, because it hid processes, not itself; the responder ran lsmod at 15:03, and whether the name meant anything to them at the time is not in the records.

04

What Memory Kept That the Disk Lost

History after history -c, and a password in the arguments

The first capture, taken earlier in the same boot, shows what memory holds for the attacker's root shell:

$ vol -f NE-APP01-1421.lime linux.bash --pid 2092 | cut -f3,4
Progress:  100.00		Stacking attempts finished
Volatility 3 Framework 2.28.2

CommandTime	Command

2026-03-24 14:02:11.000000 UTC	id
2026-03-24 14:02:19.000000 UTC	cat /etc/shadow
2026-03-24 14:03:31.000000 UTC	history -c

At 14:02 the root shell ran id and read /etc/shadow, and at 14:03 it ran history -c. That command clears the shell's history list, and the history file on disk would hold none of it. The commands are still in this image because bash keeps its history as structures in the process's memory, and Volatility recovered them from there, with their times.

That is a case lesson 0.1 and lesson 0.3 could only hint at: on those hosts the history lines survived on disk, which an attacker could not count on. Here the disk would have shown nothing, and the only record of the shadow file being read is in memory. Module 10's lesson on pulling the session out of memory is about exactly this kind of recovery.

The times on the recovered lines matter as much as the commands. bash records the time each command was entered, and Volatility recovers those times with the lines. So the image dates the read of /etc/shadow to 14:02:19, eight seconds after the shell confirmed it was root, and the clearing of the history to a minute later.

That is a sequence a report can state with confidence, from a source the attacker believed they had erased.

Memory also holds the arguments every process was started with:

$ vol -f NE-APP01-1421.lime linux.psaux | grep psql | cut -c1-80
Progress:  100.00		Stacking attempts finished
4102	4101	psql	psql postgresql://orders:S3cr3t!@db01/orders -c \copy (SELECT * F

A psql client was started with a connection string on its command line, and the string includes the database user's password. Any account on the server could read it from /proc at that moment, unless /proc was mounted to hide other users' processes; the image keeps it for the investigator.

That is a credential the intruder may have, and the investigation has to treat it as exposed whether or not any record shows it being used.

Neither finding could have come from a collection taken after a reboot, and neither is in any log. They exist only in memory, and only for as long as the processes that hold them keep running.

That is the argument for acquiring memory early on a host like this. Every minute the server runs, shells exit, processes end and pages are reused, and the history and arguments go with them.

The second capture, forty minutes later, still had the root shell's history because the shell was still running; had the attacker closed it, that history would have been gone from both disk and memory. Module 9's lesson on the decision to acquire memory is about weighing that loss against the cost and risk of acquiring.

05

Your Commands Are Evidence Too

The responder's session and the acquisition module

The same image also holds the responder's session, from login to the acquisition:

$ vol -f NE-APP01-1505.lime linux.pstree | grep -A4 "	4990	" | cut -f2-5
Progress:  100.00		Stacking attempts finished
4990	4990	597	sshd
4992	4992	4990	sshd
4993	4993	4992	bash
5010	5010	4993	sudo
5011	5011	5010	sudo
5012	5012	5011	insmod
$ vol -f NE-APP01-1505.lime linux.lsmod | grep lime | cut -f5 | cut -d, -f1-2
Progress:  100.00		Stacking attempts finished
path=tcp:4444, format=lime

An SSH session started at 15:01, its shell as user 1002, a sudo, and insmod running as root at 15:05. The lime module sits in the module list with the arguments it was loaded with: send the image over TCP on port 4444. Every one of those is a trace of the collection, in the evidence the collection produced.

Two people on app01 at 15:05

Everyone leaves traces

The attacker

root shell 2092, evil_beacon 4820, hideproc loaded at 14:32

The responder

ssh session 4990 to 4993 at 15:01, ps, lsmod, then sudo insmod of LiME

Telling them apart

parentage, user, time, and the responder's own notes

Both in the report

the acquisition module is evidence of the collection, not of the intrusion

None of that is a mistake. Acquiring memory from a running Linux host means loading a module or reading a device, and either changes the system. What matters is that the responder's actions are recorded and kept apart from the attacker's in the report. A reader who finds an unsigned out-of-tree module loaded by insmod as root, with no explanation, will reasonably count it as a second rootkit.

That is why Module 9's lessons on what your commands change and on custody of a live collection come before its lessons on tools. A responder who notes every command, with its time, before running it, gives the report a list to set against the image. Here the history kept that list for them; on another host it might not.

The order also matters. The responder ran ps and lsmod before taking memory, and both changed the state slightly: two short-lived processes, a few pages of memory.

On this host that cost nothing, but on a host where the attacker had set a trap for exactly those commands, it could have. Module 9's order of volatility lesson measures what each step costs, so that the order can be chosen rather than inherited.

The acquisition also left a network trace: LiME sent the image over TCP on port 4444 to the responder's machine. On a host being watched by the attacker, that connection is visible to them too, and on a network being monitored it looks like a large transfer to an internal address.

Module 9's lesson on collecting without writing covers the choices, network or local disk or attached storage, and what each leaves behind.

06

What Modules 9 and 10 Build On It

Live response, memory and malware

Module 9 is live response: the host that cannot stop, what a responder's commands change, the order of volatility measured, reading /proc under load, concealment on a live host, the decision to acquire memory, collecting without writing, custody, the handoff and a workflow.

Module 10 is memory and malware: why memory is ground truth, acquiring and opening it, recovering what was hidden, pulling the session out, placing findings in time, triaging and running a binary, and detection that outlives the incident.

What Modules 9 and 10 build

Modules 9 and 10

9.1 to 9.5

the host that cannot stop, what your commands change, order of volatility, /proc under load, concealment

9.6 to 9.10

the decision to acquire memory, collecting without writing, custody, the handoff, the workflow

10.1 to 10.5

memory as ground truth, acquiring it, opening it, recovering the hidden, the session from memory

10.6 to 10.10

placing findings in time, triaging and running the binary, detection, the campaign

This lesson's sections map onto both, and they also show why the two belong in one course: the live response decisions in Module 9 decide what Module 10 has to work with. How late the collections were is Module 9's starting point.

The ps that saw nothing and the hook behind it are concealment, live, and the decision to acquire memory. The recovered history and the password are Module 10's session and ground-truth lessons. The responder's traces are what your commands change, and custody.

The habits to carry into them are the ones this lesson used. Record when every collection ran, and read every absence against that time. Assume the live tools can be lied to on a host where the intruder has root, and plan at least one check that does not depend on them.

Take memory early when a hook is possible, which on a host where the intruder had root is always. Read memory for what no disk or log holds: hidden processes, cleared histories, arguments and keys. And record every command you run, so your traces can be told from the intruder's.

The two modules are deliberately consecutive. Module 9 ends at the moment the memory image and the live collection are handed over, with custody recorded; Module 10 starts with that handover and opens the image. A responder who knows what the analyst will need collects differently, and an analyst who knows what the responder could and could not do reads the collection with the right expectations.

Module 10 also goes past this image's rootkit to the binary itself: triaging an unknown executable without running it, then running it in a controlled lab to watch what it does, and turning what is learned into detection that will catch it on the next host. The beacon here would be the first sample on that path.

Both modules work on lab hosts and captures where the ground truth is known, including the course lab version of this exact scenario, so that every hidden process and every recovered line can be checked against what actually happened.

Do

Practice

Decide whose processes these are
prepared evidence

The prepared evidence below is the task list from NE-APP01-1505, from the attacker's SSH session onward. Decide which processes are the responder's, which one their ps could not see, and how the report should treat the acquisition. Each answer comes with its reasoning, so check it against the rows rather than only the choice.

Follow the PPID column to group processes into sessions, then read the UID and creation time of each session's first process. Two people logged in about an hour apart, as different users, and each left a chain.

Then look closely at the EUID column. A process whose UID is an ordinary user but whose EUID is 0 is running with root's effective rights; in this list those are the sudo processes, one pair in each session. Reading UID and EUID together is how a task list shows an escalation without any log at all, in either session.

Then open the Memory Range and run linux.pslist and linux.pstree on NE-APP01-1505 yourself, and compare them with linux.psscan. Note which plugin finds evil_beacon and why each one does or does not. Then run linux.sockstat on the same capture and find the beacon's connection, which no hook on getdents64 could hide.

Live response happens late, on a host the intruder controls. On app01 the responder's ps missed a running beacon because a module had hooked the system call ps relies on, and only the memory image showed it; memory also kept a root shell's history after it was cleared and a database password from a process's arguments, along with every trace of the responder's own collection.

The beacon was invisible to the live tools, the history was gone from disk, and all of it was in the image.

The last lesson of the module puts everything together on one host, from the alert to a written finding.