Linux Forensic Evidence Sources and Order of Volatility

What the evidence landscape is

Evidence on a live Linux host is not all equally durable. Some of it survives for months on disk; some vanishes the instant a process exits or the machine reboots. This sub maps every source of evidence on a Linux host and ranks them by how fast each decays, then derives the order you collect them in.

It teaches no single tool in depth. It teaches the priority order that governs live response, so that when you reach a compromised host you spend your first minutes on the evidence that will be gone soonest rather than the evidence that will keep.

Evidence has a shelf life

When you reach a compromised Linux host, you are not looking at a frozen crime scene. You are looking at a machine that is still running, still changing, and actively erasing some of the evidence you need with every second that passes. The contents of memory shift constantly. Network connections open and close. Temporary files are created and cleared. A reboot, whether triggered by you, by the attacker, or by a scheduled job, wipes everything that was not written to disk. Even leaving the system running costs you evidence, because normal operation overwrites the very disk sectors a deleted file once occupied.

This is the central tension of live response: the act of investigating changes the thing you are investigating, and waiting changes it too. There is no perfectly clean option. What you have instead is a principle for spending your limited time well, collecting the evidence that will vanish soonest before the evidence that will keep. That principle is the order of volatility, and it is the single most important idea in live Linux response. It is formalized in the forensic community's guidance on evidence collection (RFC 3227), and it applies to any operating system, but Linux makes it especially concrete because so much high-value evidence lives only in memory.

The landscape, ranked by volatility

Picture the evidence on a Linux host as layers, from the most fleeting at the top to the most durable at the bottom. The higher a layer sits, the sooner it disappears and the earlier you must collect it.

The vertical position is the whole lesson: the higher a layer sits, the sooner it is gone, so the earlier you must capture it. An attacker's fileless payload lives at the top and dies on reboot; the web log at the bottom may already be safe on a central server. You collect top to bottom.

Walk the layers from the top, because that is the order you collect them.

Memory is the most valuable and the most fragile evidence on a Linux host. A full memory capture contains every running process, the network connections each one holds, the files they have open, decrypted data, injected code that never touched the disk, and command lines that may have been cleared from shell history. Living-off-the-land and fileless intrusions, the kind Section 0.1 described, often exist only in memory. The moment the system reboots or the malicious process exits, this evidence is gone permanently. There is no recovering it from disk because it was never on disk. This is why memory capture, covered in Module 12, is the first collection step in a live response when the situation allows it.

Network state is nearly as fragile. The list of active connections, what each process is talking to, the routing and ARP tables, all of it changes second by second as connections open and close. A connection to an attacker's command-and-control server might be visible right now and gone in thirty seconds. Capturing the current network state, which Section 0.4 shows you how to read directly from the kernel, freezes a snapshot of who the host is talking to at the moment you arrive.

Temporary filesystems sit in the middle. Directories like /tmp, /run, and /dev/shm are common attacker staging grounds because they are world-writable, and /dev/shm in particular lives in memory rather than on disk. Their contents are stable while the system runs but vanish on reboot. They are more durable than live memory and network state but far more fragile than the main disk.

The disk is your durable evidence. Files, the logs from Section 0.2, filesystem timestamps, and even deleted data that has not yet been overwritten all survive a reboot. This is the layer most investigations spend the most time in, and it is the most forgiving, but it is not permanent: every minute the system runs, normal activity reuses the disk sectors that deleted files occupied, so even here, sooner is better.

Off-host evidence is the most durable of all. Backups, logs forwarded to a central server, and forensic images you have already captured exist independently of the compromised host and cannot be altered by an attacker still on the box. You collect these last precisely because they are not going anywhere, and a central log server is often where you find the evidence an attacker successfully deleted from the host itself.

A worked example makes the ranking concrete. Suppose the same five-stage intrusion from Section 0.2 is live in front of you. The attacker's reverse shell exists only as a running process holding an open network connection: that is top-layer evidence, lost the instant the process dies or the box reboots. The script they downloaded into /dev/shm is one layer down, stable while the machine runs but gone on reboot. The web shell they dropped into the uploads directory is on disk, durable across a reboot but degrading as the busy web server reuses storage. And the original malicious request sits in the web access log, the most durable on-host source, possibly already mirrored to a central log server beyond the attacker's reach. Four pieces of one intrusion, four different shelf lives. Collect them in the wrong order and you lose the reverse shell and the in-memory script while carefully preserving a log that was never going anywhere.

How you actually read each layer

Naming the layers is half the picture; the other half is knowing the command that freezes each one, because in a live response you are typing these under time pressure. The volatile layers, the top three, are captured with a small, memorable set of tools, all of which Section 0.4 and Module 12 develop in full:

The detail to internalize now is that all of these read the current state and write it somewhere safe. You run them and redirect the output to a file on external media or a network share, never to the local disk, because writing to the disk you are investigating overwrites the very unallocated space that may hold deleted evidence. The output is your frozen snapshot. Once it is captured, the live state can change all it wants; your evidence is already safe.

The collection order that follows

The order of volatility translates directly into a collection sequence. You move down the layers, fragile to durable, capturing each before it can change.

This sequence assumes you have decided to do live collection at all, which is itself a judgment. If the host holds memory-only evidence you cannot afford to lose, you collect live, in this order. If the priority is preserving the disk in a pristine state for legal proceedings and the volatile evidence is less critical, you might instead pull the power and image the disk cold, accepting the loss of memory to guarantee the disk is untouched. That decision, live triage versus cold imaging, is one of the first you make in any incident, and Module 16 develops the judgment behind it. The order of volatility tells you how to collect once you have decided to collect live.

In practice the decision turns on a few concrete questions. Is the suspected activity fileless or memory-resident, such as an injected process or a reverse shell with no payload on disk? If so, the evidence is in the top layers and a cold image captures none of it; you collect live. Is the host still actively connected to an attacker, where the live network state names the command-and-control address you need? Collect live. Conversely, is this likely to end in court, where the defense will challenge any change you made to the system? Then a pristine cold image may matter more than the volatile layers, and you weigh what you lose against what you protect. Is the system already so unstable that interacting with it risks crashing it before you capture anything? That argues for the fastest possible memory capture and nothing more. There is no universal right answer, only the right answer for the incident in front of you, and making that call quickly and defensibly is a skill the course builds deliberately rather than leaving to instinct.

Why this ordering changes how you think about an incident

The order of volatility reframes the first minutes of a response. The instinct under pressure is to start reading, to open a log and begin understanding what happened. But reading is analysis, and analysis can wait, while volatile evidence cannot. The disciplined move is to collect first and analyze second: freeze the fragile layers into files you can study at leisure, then investigate from those captures rather than from the live, decaying host.

There is a second reason this matters beyond preservation. An attacker who is still active on the host is also working against your evidence. They may be clearing logs, deleting their tools, and killing processes that would reveal them. Every layer you capture early is a layer they can no longer alter. Collecting in volatility order is partly a race against decay and partly a race against an adversary with their hand on the same evidence. The faster you freeze the volatile layers, the less of the story they can erase before you have it.

The next section turns the abstract layers into concrete commands. The reason memory and process state are collectable at all, the reason you can read exactly what every running process is doing, is a single design decision in how Linux works: it presents the live system as a set of files you can read. That is the subject of Section 0.4.