Reading width
Wide uses the full column for everything, text, diagrams, code, and exercises. Narrow keeps the standard reading width.
Text size
Scales the body text. Headings and code blocks keep their size.
In this section
Linux Memory Forensics: Banners, pstree, psscan, bash History and a getdents64 Rootkit in Volatility 3
Introduction
Every capture so far has been Windows. Most of the servers an attacker wants are not: web and application servers, databases, build machines and cloud workloads run Linux, and an intrusion that starts on a Windows workstation often ends on one.
The questions an analyst asks of a Linux memory image are the same ones this module has been asking: what was running, what ran and is gone, what the attacker typed, what the machine was talking to, and what is running in the kernel that should not be.
The plugins that answer them have different names, and one thing about Linux changes how an image is read before any plugin runs at all.
Every Linux plugin reads the kernel through a symbol table built for that exact kernel; nothing runs without it.
This lesson reads two captures of the same Ubuntu 24.04 application server, app01, on the same boot. The first, NE-APP01-1421, was taken after an SSH login nobody expected.
The second, NE-APP01-1505, was taken forty minutes later, after the attacker loaded a kernel module, and while a process the live host would not list was still connecting out. Module 7 builds each part of this into its own lesson, with the kernel structures behind every plugin.
A note on what these captures are. Like every image in the range, they are simulated from a written scenario rather than taken from a machine, with each plugin's real columns and output format.
The scenario follows the attack Module 7 runs in the course lab, so what you read here is what that lab produces, on a host whose details were chosen for teaching: one server, one attacker address, one boot, and a responder whose every step is in the image too.
That last point is worth holding on to through the whole lesson. Two people worked on this host before you opened the image: an attacker and a responder. Both left processes, connections and modules behind, and part of reading a Linux image, as with the Windows ones, is telling their traces apart and saying in the report which is which.
Windows to Linux, for this lesson
Same questionsA Linux Image Needs Its Kernel
The banner, and symbols for that exact buildA Windows image tells Volatility which symbols it needs, and Volatility can fetch them from Microsoft. A Linux image does not work that way.
The structures every plugin reads, the task list, the module list, the socket tables, change shape with the kernel version and the options it was built with, so Volatility needs a symbol table, an ISF file, made for exactly the kernel that was running. The first thing to read from a Linux image is which kernel that was:
$ vol -f NE-APP01-1421.lime banners.Banners | grep Linux | cut -f2 | cut -d" " -f1-3
Progress: 100.00 Stacking attempts finished
Linux version 6.8.0-52-generic
Linux version 6.8.0-52-generic
Two copies of the banner, both 6.8.0-52-generic: Ubuntu's build of the 6.8 kernel. That string is the key to everything else.
The symbol table has to be built from the debug symbols of that exact package, and one built for 6.8.0-51 or for another distribution's 6.8 will not do: structure offsets move between builds, and a symbol table that is close but wrong produces output that looks plausible and is not.
The captures in this range come with their symbols, so every Linux plugin here resolves. Pointing a Windows plugin at the same image shows what an unresolved image looks like:
$ vol -f NE-APP01-1421.lime windows.pslist 2>/dev/null | head -4
Volatility 3 Framework 2.28.2
Unsatisfied requirement plugins.PsList.kernel.layer_name:
Unsatisfied requirement plugins.PsList.kernel.symbol_table_name:
Both kernel requirements unsatisfied, and no output. This is the same failure lesson 0.1 warned about for a Windows image with no symbol table, and the same message you would see from a Linux plugin with the wrong ISF.
On a real case it is the most common reason a Linux analysis stalls, which is why Module 7 captures the kernel package details at acquisition time, while the machine is still there to ask.
Notice what the banner gives beyond the version. It names the compiler and the build date, so two images that both say 6.8 but came from different builds are distinguishable before any plugin runs, and it is found by scanning raw memory, so it works on an image whose symbols you do not yet have.
That makes it the first command on every Linux image and the one whose output goes into the case notes: the exact string is what the symbol table is matched against, and what a reviewer checks the analysis against later.
The acquisition itself left a mark worth knowing. These images were taken with LiME, a kernel module the responder loads to read memory, and it is in the module list of every capture, tainted as out of tree and unsigned. That is the Linux counterpart of the winpmem process in lesson 0.1: the investigation's own footprint, recorded so that nobody later mistakes it for the attacker's.
The Escalation in the Task Tree
Parents, users and the shell that should not existlinux.pstree walks the kernel's task list and arranges it by parent. Below the SSH listener, PID 597, the first capture has this:
$ vol -f NE-APP01-1421.lime linux.pstree | grep -A7 " 597 1 sshd" | cut -f2-5
Progress: 100.00 Stacking attempts finished
597 597 1 sshd
1442 1442 597 sshd
1501 1501 1442 sshd
1502 1502 1501 bash
2088 2088 1502 sudo
2089 2089 2088 sudo
2090 2090 2089 vim
2092 2092 2090 bash
Read down the tree: an SSH session, two sshd processes as OpenSSH runs each login, a bash shell, then sudo twice, vim, and a second bash under vim. A text editor with a shell for a child is the shape to notice. Editors can run commands, and an editor started through sudo runs them as root. The user IDs settle what the shape suggests:
$ vol -f NE-APP01-1421.lime linux.pslist | grep -e PID -e 1502 -e 2088 -e 2092 \
| cut -f2-9
Progress: 100.00 Stacking attempts finished
PID TID PPID COMM UID GID EUID EGID
1502 1502 1501 bash 1000 1000 1000 1000
2088 2088 1502 sudo 1000 1000 0 1000
2089 2089 2088 sudo 1000 1000 0 1000
2092 2092 2090 bash 0 0 0 0
bash 1502 runs as user 1000, svc-backup. sudo 2088 keeps the real user ID 1000 with an effective user ID of 0, which is exactly what sudo is, and the bash at 2092 runs entirely as root. The environment of that root shell records how it got there:
$ vol -f NE-APP01-1421.lime linux.envars --pid 2092 | grep SUDO | cut -f4,5
Progress: 100.00 Stacking attempts finished
SUDO_GID 1000
SUDO_COMMAND /usr/bin/vim
SUDO_USER svc-backup
SUDO_UID 1000
SUDO_USER svc-backup and SUDO_COMMAND /usr/bin/vim, inherited from the sudo that started the editor. A service account allowed to run vim as root without a password is a root shell for anyone who has its password, and the tree, the user IDs and the environment show that it was used as one.
The order of reading matters here as much as anything in the output. Names alone would have cleared every process in that tree: sshd, bash, sudo and vim are on every Linux server.
The parent relationships, then the user IDs, then the environment, each narrowed what the one before suggested, and none of them needed knowledge of the attacker's tools. It is lesson 0.2's habit on a different operating system: read the shape of the tree before the names in it, and let the columns confirm what the shape suggests.
The escalation, one parent at a time, in NE-APP01-1421.
None of this needed a rootkit or an exploit. Module 7 calls this the SSH-to-root chain and follows each step to the memory structure it writes, so that the analysis goes to known places rather than searching the image cold.
What Only the Scan Keeps
Dead tasks, and no exit timelinux.psscan scans memory for task structures rather than walking the list, exactly as psscan does on Windows. The tasks it finds that the list no longer holds tell the part of the story that finished before the capture:
$ vol -f NE-APP01-1421.lime linux.psscan | grep EXIT_DEAD | cut -f2-6 | sort -n
Progress: 100.00 Stacking attempts finished
1388 1388 597 sshd EXIT_DEAD
1395 1395 597 sshd EXIT_DEAD
1404 1404 597 sshd EXIT_DEAD
1417 1417 597 sshd EXIT_DEAD
1428 1428 597 sshd EXIT_DEAD
1436 1436 597 sshd EXIT_DEAD
2061 2061 1502 sudo EXIT_DEAD
2101 2101 2092 id EXIT_DEAD
2103 2103 2092 cat EXIT_DEAD
Six dead sshd tasks under the listener, PIDs 1388 to 1436, every one lower than 1442, the sshd of the session that got in. That is password guessing as memory records it: each failed attempt was a short-lived child of the SSH listener, and the scan found what was left of six of them.
Then sudo -l from the attacker's shell, and id and cat from the root shell, all finished and all still in memory. Each is a task structure the kernel has finished with but has not yet reused, which is why the scan finds some short-lived tasks and not others: memory that nothing has needed again keeps what was last written to it.
There is a difference from Windows to keep in mind. A Windows process carries its exit time, which lesson 0.2 used to tell a finished process from a hidden one.
A Linux task carries an exit state, EXIT_DEAD here, and no exit time. The scan proves these tasks ran and ended; when they ended has to come from somewhere else, and on this host the somewhere else is the shell history.
The scan also finds what the list does not show for a less innocent reason, which is why it is run on every Linux image whether or not anything looks wrong. On this capture, every running task the scan found is also on the list. That is a result to record, and the second capture tests it.
The PIDs carry information too. Linux hands out process IDs in increasing order until they wrap, so on a server that has not been up long, a lower PID was created earlier.
The six dead sshd tasks all sit below 1442, the sshd of the session that got in, and above the services started at boot, which fits a run of attempts shortly before the successful login. It is an inference from numbering, not a timestamp, and it is stated as one; the shell history and the authentication log on disk are where the times come from.
What the Root Shell Typed
History cleared, history recoveredbash keeps its command history in the shell's own memory, with a timestamp for each command, and writes it to disk only when the shell exits. linux.bash reads it from memory, for every bash process in the image:
$ vol -f NE-APP01-1421.lime linux.bash --pid 1502 2092 | cut -f1,3,4
Progress: 100.00 Stacking attempts finished
Volatility 3 Framework 2.28.2
PID CommandTime Command
1502 2026-03-24 13:59:02.000000 UTC sudo -l
1502 2026-03-24 14:01:30.000000 UTC sudo vim
2092 2026-03-24 14:02:11.000000 UTC id
2092 2026-03-24 14:02:19.000000 UTC cat /etc/shadow
2092 2026-03-24 14:03:31.000000 UTC history -c
From svc-backup's shell, sudo -l at 13:59:02, checking what the account may run as root, then sudo vim at 14:01:30.
From the root shell, id at 14:02:11 to confirm it was root, cat /etc/shadow eight seconds later, which is the objective, and history -c at 14:03:31. history -c clears the shell's history list, and the commands are still here: clearing the list does not wipe the memory the entries occupied, and the shell has not exited, so nothing was written to disk either. The cover-up is itself evidence.
Where the session came from is in the sockets the sshd processes hold:
$ vol -f NE-APP01-1421.lime linux.sockstat | grep 203.0.113 | cut -f2,3,10-14
Progress: 100.00 Stacking attempts finished
sshd 1442 10.40.12.21 22 203.0.113.66 52210 ESTABLISHED
sshd 1501 10.40.12.21 22 203.0.113.66 52210 ESTABLISHED
Both sshd processes of the session hold the same connection, from 203.0.113.66 port 52210 to port 22. That is the source of the login, and the address the password guessing came from.
One more thing in this image matters even though the attacker did not create it. A scheduled export started at 14:00, and its database client carries a password in its command line:
$ vol -f NE-APP01-1421.lime linux.psaux | grep psql | cut -c1-80
Progress: 100.00 Stacking attempts finished
4102 4101 psql psql postgresql://orders:S3cr3t!@db01/orders -c \copy (SELECT * F
Any user on the host could read that from the process list while it ran, and the attacker was root. Module 7 makes this part of the credential exposure assessment: everything the attacker could reach, which here includes the database password as well as the hashes in /etc/shadow.
Put together, the first capture answers the questions the case started with. The attacker guessed svc-backup's password from 203.0.113.66, logged in at 13:58, found that the account could run vim as root, used it to get a root shell, read the password hashes and tried to cover the shell's history.
Every part of that sentence rests on a row above, and the parts no row supports, how long the guessing ran and whether the hashes left the host, are what the report says it does not know.
The Rootkit ps Could Not See
Task list, module list, system call tableThe second capture is the same boot forty minutes later. Before taking it, the responder checked the live host:
$ vol -f NE-APP01-1505.lime linux.bash --pid 4993 | grep 2026 | cut -f4
Progress: 100.00 Stacking attempts finished
ps aux | grep -i beacon
lsmod | head -5
sudo insmod /media/ir/lime-6.8.0-52-generic.ko path=tcp:4444 format=lime
The responder's own shell, PID 4993, looked for the beacon with ps, checked the top of the module list, and then loaded LiME. ps aux showed no beacon. The task list in memory does:
$ vol -f NE-APP01-1505.lime linux.pstree | grep -B2 evil_beacon | cut -f2-5
Progress: 100.00 Stacking attempts finished
2090 2090 2089 vim
2092 2092 2090 bash
4820 4820 2092 evil_beacon
evil_beacon, PID 4820, a child of the root shell, running. ps finds processes by listing the directories under /proc, and the listing goes through a system call, getdents64. Something on this host was answering that call. The module list shows what had been loaded:
$ vol -f NE-APP01-1505.lime linux.lsmod | head -7 | cut -f1-4
Progress: 100.00 Stacking attempts finished
Volatility 3 Framework 2.28.2
Offset Module Name Code Size Taints
0xffffffffc0a87840 lime 0x5000 OOT_MODULE,UNSIGNED_MODULE
0xffffffffc0a82040 hideproc 0x4000 OOT_MODULE,UNSIGNED_MODULE
0xffffffffc0a6c040 tls 0x26000
The responder's lime, then hideproc, loaded at 14:32:40 by the root shell's history, both tainted as out of tree and unsigned, then the stock modules, which carry no taint. The system call table shows what hideproc did. check_syscall lists all 924 entries of the 64-bit and 32-bit tables with the kernel symbol each handler resolves to:
$ vol -f NE-APP01-1505.lime linux.malware.check_syscall | grep -c UNKNOWN
Progress: 100.00 Stacking attempts finished
1
$ vol -f NE-APP01-1505.lime linux.malware.check_syscall \
| grep -e UNKNOWN -e " 78 " | cut -f2-5 | head -3
Progress: 100.00 Stacking attempts finished
64bit 78 0xffffffff9a302080 __x64_sys_getdents
64bit 217 0xffffffffc0a81180 UNKNOWN
32bit 78 0xffffffff9a312630 __ia32_compat_sys_gettimeofday
One entry has no symbol: index 217, getdents64, with a handler at 0xffffffffc0a81180. Every ordinary handler is in the kernel's text, at addresses beginning 0xffffffff9a, like the older getdents at index 78. This one is in the area where modules load.
The getdents64 entry
Attributed by addressThe attribution is by address, as it was for the Windows callback in lesson 0.7, and it needs the same care: lsmod's Offset is where the module's structure sits, not where its code starts, so the window is an argument rather than a proof. Module 7 extracts the module and reads the function. Two more results complete the reading:
$ vol -f NE-APP01-1505.lime linux.malware.hidden_modules | tail -2
Progress: 100.00 Stacking attempts finished
Offset Module Name Code Size Taints Load Arguments File Output
$ vol -f NE-APP01-1505.lime linux.sockstat --pids 4820 | cut -f2,3,10-14 | tail -1
Progress: 100.00 Stacking attempts finished
evil_beacon 4820 10.40.12.21 47352 203.0.113.66 443 ESTABLISHED
hidden_modules found nothing the list lacks, so this rootkit did not hide itself, and the beacon's connection to 203.0.113.66 on port 443 is in its sockets. The hook hid one process from the tools that ask the kernel; memory analysis does not ask.
That last sentence is the reason this capture was taken with the rootkit still loaded. Removing hideproc first would have restored ps, unhooked getdents64 and taken the module out of the list, leaving the beacon visible and nothing to show how it had been hidden.
Module 7 makes that decision explicit: the rootkit is the evidence, and a responder who cleans before capturing destroys it. The responder here checked the live host, saw nothing, and captured memory anyway, which is the step that found it.
What Module 7 Builds On It
Linux compromise in depthModule 7 starts with the kernel structures Linux memory forensics reads, mapped to the Windows objects this module has already met, and the symbol-table dependency that decides whether a Linux image is evidence or unreadable bytes.
It runs the SSH-to-root chain in the course lab, acquires the host with LiME and proves the image resolves, then reads the user-space intrusion and every credential the attacker reached. It loads the getdents64 rootkit, captures it live, detects it, and ends with an eBPF rootkit that leaves both the module list and the system call table clean, and an unguided hunt.
What Module 7 builds
Module 7The habits carry over from the Windows lessons almost unchanged. Read the tree before the names: an editor with a shell for a child was the finding, and nothing in either name was suspicious.
Run the scan as well as the list, and read what it alone keeps. Read a cleared history as a cleared list, not cleared memory. Attribute an unnamed address by where it points, and say how firm the attribution is.
Two habits are new. The first is the symbol table: confirm the banner and the ISF before reading anything, because a Linux image analyzed with the wrong symbols does not always fail loudly. The second is the limit of the checks in this lesson.
A clean module list and a clean system call table rule out the techniques they check, and Module 7's eBPF rootkit is built to pass both. A report on a Linux host says which checks were run, as lesson 0.7 asked of the Windows kernel.
The second capture also shows why the two captures were worth comparing rather than reading separately. The root shell 2092 is the same process in both, with the same address and the same start time, and its history in the second capture continues from where the first ended.
The forty minutes between them are where the rootkit arrived, and reading the second image against the first is what turns a list of findings into a sequence.
Practice
Find what answers for getdents64 prepared evidenceThe prepared evidence below is eight entries of the 64-bit system call table and the five most recently loaded modules in NE-APP01-1505. Find the entry that does not point into the kernel, attribute it to a module, and explain why ps could not see the beacon. Each answer comes with its reasoning, so check it against the rows rather than only the choice.
The two tabs are deliberately small. The full table has hundreds of entries, almost all of them ordinary, and the skill being practiced is the comparison that makes one entry stand out: where every handler should live, and which one does not live there.
In the range, open NE-APP01-1505 and run linux.pslist with grep for 4820, then run linux.malware.check_syscall and count the UNKNOWN rows. Then open the baseline, NE-APP01-baseline, and run the same count: a clean boot of the same build has none, and its module list has lime and the stock set, with no hideproc.
Then run linux.bash on NE-APP01-1505 for PID 2092 and read the commands after history -c: the download, the insmod and the start of the beacon, each with its time, explain the rootkit you just found.
A Linux image is read through a symbol table built for its exact kernel, so the banner comes first. After that, the questions are the Windows ones: the task tree shows an escalation through its parents and user IDs, the scan keeps the tasks that finished but carries no exit time, bash history survives history -c in a running shell, and sockstat names the session's source.
A rootkit that hooks getdents64 hides a process from ps and not from the task list, and its hook is found by a handler address outside the kernel's text and attributed, with stated care, to the module it points into.
The last two lessons read the evidence of the whole module across time and on an image you have not seen.