In this section

Linux Memory Forensics: Banners, pstree, psscan, bash History and a getdents64 Rootkit in Volatility 3

Module 0

Introduction

Every capture so far has been Windows. Most of the servers an attacker wants are not: web and application servers, databases, build machines and cloud workloads run Linux, and an intrusion that starts on a Windows workstation often ends on one.

The questions an analyst asks of a Linux memory image are the same ones this module has been asking: what was running, what ran and is gone, what the attacker typed, what the machine was talking to, and what is running in the kernel that should not be.

The plugins that answer them have different names, and one thing about Linux changes how an image is read before any plugin runs at all.

Bannerthe kernel build,to the patch levelselectsSymbolsan ISF made forexactly that builddecodesTask listpslist, pstree, psaux,bash, sockstatthenKernellsmod, check_syscall,hidden_modules

Every Linux plugin reads the kernel through a symbol table built for that exact kernel; nothing runs without it.

This lesson reads two captures of the same Ubuntu 24.04 application server, app01, on the same boot. The first, NE-APP01-1421, was taken after an SSH login nobody expected.

The second, NE-APP01-1505, was taken forty minutes later, after the attacker loaded a kernel module, and while a process the live host would not list was still connecting out. Module 7 builds each part of this into its own lesson, with the kernel structures behind every plugin.

A note on what these captures are. Like every image in the range, they are simulated from a written scenario rather than taken from a machine, with each plugin's real columns and output format.

The scenario follows the attack Module 7 runs in the course lab, so what you read here is what that lab produces, on a host whose details were chosen for teaching: one server, one attacker address, one boot, and a responder whose every step is in the image too.

That last point is worth holding on to through the whole lesson. Two people worked on this host before you opened the image: an attacker and a responder. Both left processes, connections and modules behind, and part of reading a Linux image, as with the Windows ones, is telling their traces apart and saying in the report which is which.

Windows to Linux, for this lesson

Same questions

pslist, psscan

linux.pslist walks the task list; linux.psscan scans for task structures

Exit time

none on Linux: psscan gives an exit state, so order comes from other evidence

cmdline, consoles

linux.psaux and linux.bash

netscan

linux.sockstat, from each process's open sockets

modules, modscan, ssdt

linux.lsmod, hidden_modules, check_syscall

01

A Linux Image Needs Its Kernel

The banner, and symbols for that exact build

A Windows image tells Volatility which symbols it needs, and Volatility can fetch them from Microsoft. A Linux image does not work that way.

The structures every plugin reads, the task list, the module list, the socket tables, change shape with the kernel version and the options it was built with, so Volatility needs a symbol table, an ISF file, made for exactly the kernel that was running. The first thing to read from a Linux image is which kernel that was:

$ vol -f NE-APP01-1421.lime banners.Banners | grep Linux | cut -f2 | cut -d" " -f1-3
Progress:  100.00		Stacking attempts finished
Linux version 6.8.0-52-generic
Linux version 6.8.0-52-generic

Two copies of the banner, both 6.8.0-52-generic: Ubuntu's build of the 6.8 kernel. That string is the key to everything else.

The symbol table has to be built from the debug symbols of that exact package, and one built for 6.8.0-51 or for another distribution's 6.8 will not do: structure offsets move between builds, and a symbol table that is close but wrong produces output that looks plausible and is not.

The captures in this range come with their symbols, so every Linux plugin here resolves. Pointing a Windows plugin at the same image shows what an unresolved image looks like:

$ vol -f NE-APP01-1421.lime windows.pslist 2>/dev/null | head -4
Volatility 3 Framework 2.28.2

Unsatisfied requirement plugins.PsList.kernel.layer_name: 
Unsatisfied requirement plugins.PsList.kernel.symbol_table_name: 

Both kernel requirements unsatisfied, and no output. This is the same failure lesson 0.1 warned about for a Windows image with no symbol table, and the same message you would see from a Linux plugin with the wrong ISF.

On a real case it is the most common reason a Linux analysis stalls, which is why Module 7 captures the kernel package details at acquisition time, while the machine is still there to ask.

Notice what the banner gives beyond the version. It names the compiler and the build date, so two images that both say 6.8 but came from different builds are distinguishable before any plugin runs, and it is found by scanning raw memory, so it works on an image whose symbols you do not yet have.

That makes it the first command on every Linux image and the one whose output goes into the case notes: the exact string is what the symbol table is matched against, and what a reviewer checks the analysis against later.

The acquisition itself left a mark worth knowing. These images were taken with LiME, a kernel module the responder loads to read memory, and it is in the module list of every capture, tainted as out of tree and unsigned. That is the Linux counterpart of the winpmem process in lesson 0.1: the investigation's own footprint, recorded so that nobody later mistakes it for the attacker's.

02

The Escalation in the Task Tree

Parents, users and the shell that should not exist

linux.pstree walks the kernel's task list and arranges it by parent. Below the SSH listener, PID 597, the first capture has this:

$ vol -f NE-APP01-1421.lime linux.pstree | grep -A7 "	597	1	sshd" | cut -f2-5
Progress:  100.00		Stacking attempts finished
597	597	1	sshd
1442	1442	597	sshd
1501	1501	1442	sshd
1502	1502	1501	bash
2088	2088	1502	sudo
2089	2089	2088	sudo
2090	2090	2089	vim
2092	2092	2090	bash

Read down the tree: an SSH session, two sshd processes as OpenSSH runs each login, a bash shell, then sudo twice, vim, and a second bash under vim. A text editor with a shell for a child is the shape to notice. Editors can run commands, and an editor started through sudo runs them as root. The user IDs settle what the shape suggests:

$ vol -f NE-APP01-1421.lime linux.pslist | grep -e PID -e 1502 -e 2088 -e 2092 \
    | cut -f2-9
Progress:  100.00		Stacking attempts finished
PID	TID	PPID	COMM	UID	GID	EUID	EGID
1502	1502	1501	bash	1000	1000	1000	1000
2088	2088	1502	sudo	1000	1000	0	1000
2089	2089	2088	sudo	1000	1000	0	1000
2092	2092	2090	bash	0	0	0	0

bash 1502 runs as user 1000, svc-backup. sudo 2088 keeps the real user ID 1000 with an effective user ID of 0, which is exactly what sudo is, and the bash at 2092 runs entirely as root. The environment of that root shell records how it got there:

$ vol -f NE-APP01-1421.lime linux.envars --pid 2092 | grep SUDO | cut -f4,5
Progress:  100.00		Stacking attempts finished
SUDO_GID	1000
SUDO_COMMAND	/usr/bin/vim
SUDO_USER	svc-backup
SUDO_UID	1000

SUDO_USER svc-backup and SUDO_COMMAND /usr/bin/vim, inherited from the sudo that started the editor. A service account allowed to run vim as root without a password is a root shell for anyone who has its password, and the tree, the user IDs and the environment show that it was used as one.

The order of reading matters here as much as anything in the output. Names alone would have cleared every process in that tree: sshd, bash, sudo and vim are on every Linux server.

The parent relationships, then the user IDs, then the environment, each narrowed what the one before suggested, and none of them needed knowledge of the attacker's tools. It is lesson 0.2's habit on a different operating system: read the shape of the tree before the names in it, and let the columns confirm what the shape suggests.

1. sshd 1442 and 1501the session from 203.0.113.66:52210, user svc-backup2. bash 1502, uid 1000sudo -l, then sudo vim3. sudo 2088 and 2089euid 0: sudo running vim as root4. vim 2090, uid 0an editor that can start a shell5. bash 2092, uid 0id, cat /etc/shadow, history -c

The escalation, one parent at a time, in NE-APP01-1421.

None of this needed a rootkit or an exploit. Module 7 calls this the SSH-to-root chain and follows each step to the memory structure it writes, so that the analysis goes to known places rather than searching the image cold.

03

What Only the Scan Keeps

Dead tasks, and no exit time

linux.psscan scans memory for task structures rather than walking the list, exactly as psscan does on Windows. The tasks it finds that the list no longer holds tell the part of the story that finished before the capture:

$ vol -f NE-APP01-1421.lime linux.psscan | grep EXIT_DEAD | cut -f2-6 | sort -n
Progress:  100.00		Stacking attempts finished
1388	1388	597	sshd	EXIT_DEAD
1395	1395	597	sshd	EXIT_DEAD
1404	1404	597	sshd	EXIT_DEAD
1417	1417	597	sshd	EXIT_DEAD
1428	1428	597	sshd	EXIT_DEAD
1436	1436	597	sshd	EXIT_DEAD
2061	2061	1502	sudo	EXIT_DEAD
2101	2101	2092	id	EXIT_DEAD
2103	2103	2092	cat	EXIT_DEAD

Six dead sshd tasks under the listener, PIDs 1388 to 1436, every one lower than 1442, the sshd of the session that got in. That is password guessing as memory records it: each failed attempt was a short-lived child of the SSH listener, and the scan found what was left of six of them.

Then sudo -l from the attacker's shell, and id and cat from the root shell, all finished and all still in memory. Each is a task structure the kernel has finished with but has not yet reused, which is why the scan finds some short-lived tasks and not others: memory that nothing has needed again keeps what was last written to it.

There is a difference from Windows to keep in mind. A Windows process carries its exit time, which lesson 0.2 used to tell a finished process from a hidden one.

A Linux task carries an exit state, EXIT_DEAD here, and no exit time. The scan proves these tasks ran and ended; when they ended has to come from somewhere else, and on this host the somewhere else is the shell history.

The scan also finds what the list does not show for a less innocent reason, which is why it is run on every Linux image whether or not anything looks wrong. On this capture, every running task the scan found is also on the list. That is a result to record, and the second capture tests it.

The PIDs carry information too. Linux hands out process IDs in increasing order until they wrap, so on a server that has not been up long, a lower PID was created earlier.

The six dead sshd tasks all sit below 1442, the sshd of the session that got in, and above the services started at boot, which fits a run of attempts shortly before the successful login. It is an inference from numbering, not a timestamp, and it is stated as one; the shell history and the authentication log on disk are where the times come from.

04

What the Root Shell Typed

History cleared, history recovered

bash keeps its command history in the shell's own memory, with a timestamp for each command, and writes it to disk only when the shell exits. linux.bash reads it from memory, for every bash process in the image:

$ vol -f NE-APP01-1421.lime linux.bash --pid 1502 2092 | cut -f1,3,4
Progress:  100.00		Stacking attempts finished
Volatility 3 Framework 2.28.2

PID	CommandTime	Command

1502	2026-03-24 13:59:02.000000 UTC	sudo -l
1502	2026-03-24 14:01:30.000000 UTC	sudo vim
2092	2026-03-24 14:02:11.000000 UTC	id
2092	2026-03-24 14:02:19.000000 UTC	cat /etc/shadow
2092	2026-03-24 14:03:31.000000 UTC	history -c

From svc-backup's shell, sudo -l at 13:59:02, checking what the account may run as root, then sudo vim at 14:01:30.

From the root shell, id at 14:02:11 to confirm it was root, cat /etc/shadow eight seconds later, which is the objective, and history -c at 14:03:31. history -c clears the shell's history list, and the commands are still here: clearing the list does not wipe the memory the entries occupied, and the shell has not exited, so nothing was written to disk either. The cover-up is itself evidence.

Where the session came from is in the sockets the sshd processes hold:

$ vol -f NE-APP01-1421.lime linux.sockstat | grep 203.0.113 | cut -f2,3,10-14
Progress:  100.00		Stacking attempts finished
sshd	1442	10.40.12.21	22	203.0.113.66	52210	ESTABLISHED
sshd	1501	10.40.12.21	22	203.0.113.66	52210	ESTABLISHED

Both sshd processes of the session hold the same connection, from 203.0.113.66 port 52210 to port 22. That is the source of the login, and the address the password guessing came from.

One more thing in this image matters even though the attacker did not create it. A scheduled export started at 14:00, and its database client carries a password in its command line:

$ vol -f NE-APP01-1421.lime linux.psaux | grep psql | cut -c1-80
Progress:  100.00		Stacking attempts finished
4102	4101	psql	psql postgresql://orders:S3cr3t!@db01/orders -c \copy (SELECT * F

Any user on the host could read that from the process list while it ran, and the attacker was root. Module 7 makes this part of the credential exposure assessment: everything the attacker could reach, which here includes the database password as well as the hashes in /etc/shadow.

Put together, the first capture answers the questions the case started with. The attacker guessed svc-backup's password from 203.0.113.66, logged in at 13:58, found that the account could run vim as root, used it to get a root shell, read the password hashes and tried to cover the shell's history.

Every part of that sentence rests on a row above, and the parts no row supports, how long the guessing ran and whether the hashes left the host, are what the report says it does not know.

05

The Rootkit ps Could Not See

Task list, module list, system call table

The second capture is the same boot forty minutes later. Before taking it, the responder checked the live host:

$ vol -f NE-APP01-1505.lime linux.bash --pid 4993 | grep 2026 | cut -f4
Progress:  100.00		Stacking attempts finished
ps aux | grep -i beacon
lsmod | head -5
sudo insmod /media/ir/lime-6.8.0-52-generic.ko path=tcp:4444 format=lime

The responder's own shell, PID 4993, looked for the beacon with ps, checked the top of the module list, and then loaded LiME. ps aux showed no beacon. The task list in memory does:

$ vol -f NE-APP01-1505.lime linux.pstree | grep -B2 evil_beacon | cut -f2-5
Progress:  100.00		Stacking attempts finished
2090	2090	2089	vim
2092	2092	2090	bash
4820	4820	2092	evil_beacon

evil_beacon, PID 4820, a child of the root shell, running. ps finds processes by listing the directories under /proc, and the listing goes through a system call, getdents64. Something on this host was answering that call. The module list shows what had been loaded:

$ vol -f NE-APP01-1505.lime linux.lsmod | head -7 | cut -f1-4
Progress:  100.00		Stacking attempts finished
Volatility 3 Framework 2.28.2

Offset	Module Name	Code Size	Taints

0xffffffffc0a87840	lime	0x5000	OOT_MODULE,UNSIGNED_MODULE
0xffffffffc0a82040	hideproc	0x4000	OOT_MODULE,UNSIGNED_MODULE
0xffffffffc0a6c040	tls	0x26000	

The responder's lime, then hideproc, loaded at 14:32:40 by the root shell's history, both tainted as out of tree and unsigned, then the stock modules, which carry no taint. The system call table shows what hideproc did. check_syscall lists all 924 entries of the 64-bit and 32-bit tables with the kernel symbol each handler resolves to:

$ vol -f NE-APP01-1505.lime linux.malware.check_syscall | grep -c UNKNOWN
Progress:  100.00		Stacking attempts finished
1
$ vol -f NE-APP01-1505.lime linux.malware.check_syscall \
    | grep -e UNKNOWN -e "	78	" | cut -f2-5 | head -3
Progress:  100.00		Stacking attempts finished
64bit	78	0xffffffff9a302080	__x64_sys_getdents
64bit	217	0xffffffffc0a81180	UNKNOWN
32bit	78	0xffffffff9a312630	__ia32_compat_sys_gettimeofday

One entry has no symbol: index 217, getdents64, with a handler at 0xffffffffc0a81180. Every ordinary handler is in the kernel's text, at addresses beginning 0xffffffff9a, like the older getdents at index 78. This one is in the area where modules load.

The getdents64 entry

Attributed by address

Kernel text

every ordinary handler sits at 0xffffffff9a...

Entry 217

0xffffffffc0a81180: the module area, with no kernel symbol

Nearest module

hideproc, structure at 0xffffffffc0a82040, 0x4000 bytes, out of tree and unsigned

Still to prove

that the code at that address is hideproc's: Module 7 extracts the module

The attribution is by address, as it was for the Windows callback in lesson 0.7, and it needs the same care: lsmod's Offset is where the module's structure sits, not where its code starts, so the window is an argument rather than a proof. Module 7 extracts the module and reads the function. Two more results complete the reading:

$ vol -f NE-APP01-1505.lime linux.malware.hidden_modules | tail -2
Progress:  100.00		Stacking attempts finished
Offset	Module Name	Code Size	Taints	Load Arguments	File Output

$ vol -f NE-APP01-1505.lime linux.sockstat --pids 4820 | cut -f2,3,10-14 | tail -1
Progress:  100.00		Stacking attempts finished
evil_beacon	4820	10.40.12.21	47352	203.0.113.66	443	ESTABLISHED

hidden_modules found nothing the list lacks, so this rootkit did not hide itself, and the beacon's connection to 203.0.113.66 on port 443 is in its sockets. The hook hid one process from the tools that ask the kernel; memory analysis does not ask.

That last sentence is the reason this capture was taken with the rootkit still loaded. Removing hideproc first would have restored ps, unhooked getdents64 and taken the module out of the list, leaving the beacon visible and nothing to show how it had been hidden.

Module 7 makes that decision explicit: the rootkit is the evidence, and a responder who cleans before capturing destroys it. The responder here checked the live host, saw nothing, and captured memory anyway, which is the step that found it.

06

What Module 7 Builds On It

Linux compromise in depth

Module 7 starts with the kernel structures Linux memory forensics reads, mapped to the Windows objects this module has already met, and the symbol-table dependency that decides whether a Linux image is evidence or unreadable bytes.

It runs the SSH-to-root chain in the course lab, acquires the host with LiME and proves the image resolves, then reads the user-space intrusion and every credential the attacker reached. It loads the getdents64 rootkit, captures it live, detects it, and ends with an eBPF rootkit that leaves both the module list and the system call table clean, and an unguided hunt.

What Module 7 builds

Module 7

7.1 to 7.3

Linux memory architecture; the SSH-to-root chain; acquiring with LiME and proving the image resolves

7.4 to 7.5

reading the user-space intrusion; every credential the attacker reached

7.6 to 7.7

the kernel rootkit, loaded and detected

7.8 to 7.9

the eBPF rootkit that leaves the table clean; the unguided hunt

The habits carry over from the Windows lessons almost unchanged. Read the tree before the names: an editor with a shell for a child was the finding, and nothing in either name was suspicious.

Run the scan as well as the list, and read what it alone keeps. Read a cleared history as a cleared list, not cleared memory. Attribute an unnamed address by where it points, and say how firm the attribution is.

Two habits are new. The first is the symbol table: confirm the banner and the ISF before reading anything, because a Linux image analyzed with the wrong symbols does not always fail loudly. The second is the limit of the checks in this lesson.

A clean module list and a clean system call table rule out the techniques they check, and Module 7's eBPF rootkit is built to pass both. A report on a Linux host says which checks were run, as lesson 0.7 asked of the Windows kernel.

The second capture also shows why the two captures were worth comparing rather than reading separately. The root shell 2092 is the same process in both, with the same address and the same start time, and its history in the second capture continues from where the first ended.

The forty minutes between them are where the rootkit arrived, and reading the second image against the first is what turns a list of findings into a sequence.

Do

Practice

Find what answers for getdents64
prepared evidence

The prepared evidence below is eight entries of the 64-bit system call table and the five most recently loaded modules in NE-APP01-1505. Find the entry that does not point into the kernel, attribute it to a module, and explain why ps could not see the beacon. Each answer comes with its reasoning, so check it against the rows rather than only the choice.

The two tabs are deliberately small. The full table has hundreds of entries, almost all of them ordinary, and the skill being practiced is the comparison that makes one entry stand out: where every handler should live, and which one does not live there.

In the range, open NE-APP01-1505 and run linux.pslist with grep for 4820, then run linux.malware.check_syscall and count the UNKNOWN rows. Then open the baseline, NE-APP01-baseline, and run the same count: a clean boot of the same build has none, and its module list has lime and the stock set, with no hideproc.

Then run linux.bash on NE-APP01-1505 for PID 2092 and read the commands after history -c: the download, the insmod and the start of the beacon, each with its time, explain the rootkit you just found.

A Linux image is read through a symbol table built for its exact kernel, so the banner comes first. After that, the questions are the Windows ones: the task tree shows an escalation through its parents and user IDs, the scan keeps the tasks that finished but carries no exit time, bash history survives history -c in a running shell, and sockstat names the session's source.

A rootkit that hooks getdents64 hides a process from ps and not from the task list, and its hook is found by a handler address outside the kernel's text and attributed, with stated care, to the module it points into.

The last two lessons read the evidence of the whole module across time and on an image you have not seen.