How a Linux Compromise Unfolds, and What Each Stage Leaves

Introduction

LX0.1 described the machine as a set of readable surfaces. This section follows one intrusion across them, because the surfaces only mean something once you have seen what actually writes to them.

The point is not the attack technique. It is that each stage of an intrusion leaves traces in different places with wildly different lifetimes, and an investigation arriving at any given moment can see some of them and not others.

An orientation built around a clever attack teaches a rare case and flatters the reader. The intrusion below is deliberately dull, because the ordinary one is what somebody will actually meet and it is quite hard enough.

You will finish able to describe an intrusion as a sequence of traces rather than a sequence of actions, say which of those traces survive the process ending, and recognize one place where the obvious reading is wrong.

The sequence is ordinary and that is the point, because most intrusions are.

Every stage in it is something the system does thousands of times a day for legitimate reasons. Files get written, processes start and exit, tasks get scheduled, and connections open, which is exactly why none of it triggered anything for eleven days.

$ cat timeline.txt
  02 Mar 09:14  a request reaches the web application and is answered abnormally
  02 Mar 09:14  a shell starts, owned by the web server account
  02 Mar 09:16  a file is written to a temporary directory and made executable
  02 Mar 09:16  that file runs, opens an outbound connection, and the shell exits
  02 Mar 09:31  a scheduled task is created to run the file at intervals
  13 Mar 16:02  the outbound traffic is noticed and an investigation starts

Six stages across eleven days. Only the fifth was intended to last, and it is the one an investigation finds first. The first four take two minutes, the fifth makes it survive a reboot, and the sixth is where somebody starts looking.

That distribution is typical rather than convenient. Intrusions are mostly short bursts of activity separated by long quiet periods, and the quiet is when the evidence decays.

The eleven day gap is the realistic part rather than a dramatic device. Detection by noticing unusual traffic rather than by catching the access is the common case, and it puts every investigation a long way downstream of the events it is trying to reconstruct.

Nothing in that sequence is sophisticated. Each stage is an ordinary operation the system performs constantly, and what makes it an intrusion is the account doing it and the sequence they appear in.

That is the harder problem rather than the easier one. Detecting something unusual is a matter of noticing; detecting ordinary operations performed by the wrong account in an unusual order needs every reading skill the later modules teach.

While the shell from stage two exists, the kernel answers a great deal about it.

Worth noting what is not required to get any of it. No agent, no prior configuration, no cooperation from the intruder and no tooling beyond reading files, which is the property LX0.1 established and this is the first place it pays.

$ ./stage-traces.py --pid 82
  /proc/PID/cmdline     present   what it was told to run
  /proc/PID/exe         present   the binary behind it
  /proc/PID/fd          present   open files and sockets
  /proc/PID/status      present   identity and capabilities
  /proc/PID/stat        present   start time, computable to the second

Five readings, all available, none needing any tool. A responder present at that moment would not have needed to be skilled to capture them. That is the machine at its most legible, and it is what an investigation arriving during stage two would have had.

The same five, seconds later.

$ ./stage-traces.py --pid 82
  /proc/PID/cmdline     GONE
  /proc/PID/exe         GONE
  /proc/PID/fd          GONE
  /proc/PID/status      GONE
  /proc/PID/stat        GONE

All five gone together. Nothing decays gradually here; the directory exists or it does not. There is no partial decay and no residue: the process directory exists while the process does, and the investigation in the scenario arrives eleven days after that.

Sorting the traces into categories before learning to read any of them is deliberate. The alternative is learning techniques in isolation and discovering their lifetimes one incident at a time, which is how most people acquire this and is slower than being told.

That note is the habit this whole course builds, and the two-minute intrusion above shows why.

There is a second property of the disappearance worth noticing, which is that it takes the account with it. While the shell runs, the kernel says which account owns it and what capabilities it holds; once it exits, establishing which account did anything depends on whatever a log happened to record.

That is why attribution gets harder rather than just sparser with time. The later investigation is not missing a detail of an otherwise complete picture; it is missing the thing that connected the activity to an actor.

It is worth flagging in an orientation because it is the kind of detail that produces a confident wrong answer rather than a visible failure. The reading succeeds, returns a real path, and answers a question nobody asked.

The binary path under a process directory is a reliable trace with one significant exception.

$ ./exe-vs-cmdline.py
  script, pid 92:
    exe      -> /usr/bin/dash
    cmdline  -> /bin/sh ./s.sh
  binary, pid 94:
    exe      -> /usr/bin/sleep
    cmdline  -> /bin/sleep 3

The second pair agree and the first pair do not. Nothing in either output flags which case you are looking at. For a compiled binary the path names the thing that ran; for a script it names the interpreter, and the script itself appears only in the command line.

That matters most in exactly the case an investigation cares about.

$ rm ./s.sh
$ ./exe-vs-cmdline.py --pid 92
  exe      -> /usr/bin/dash        the interpreter, NOT the script
  cmdline  -> /bin/sh ./s.sh       names the script, which no longer exists

A deleted script still running. The reading succeeded and answered a question about the interpreter. The path that would have recovered a deleted binary recovers the interpreter instead, and the only record of what actually ran is a command line naming a file that is gone.

So the two readings answer different questions. The command line says what was asked for and the binary path says what is executing, and for anything interpreted those are not the same thing.

Reading both is the practical answer and costs nothing. Two paths under the same process directory, and disagreement between them tells you immediately that something interpreted is running.

Drawing the bands against a timeline rather than listing them is worth the space, because the proportions carry the argument. The red band occupies two minutes of eleven days, and every investigation in this course arrives somewhere along the green ones.

The crossed arrow is exactly the scenario's position. Four stages of the intrusion are entirely in the red band, and everything an investigation can learn about them has to come from the green ones.

Seeing the surviving set written out is worth the space before the exercise, because it is less bleak than the timeline suggests. Three of six rows are present and a fourth is a request somebody else can answer, which is a workable position rather than a lost cause.

One framing before the exercise. Nothing about the intrusion above is clever, and that is deliberate: an orientation built around a sophisticated attack teaches a rare case. The difficulty in the exercise comes from when somebody looked rather than from what was done.

$ ./what-is-left.sh --host web01 --since '02 Mar 09:00'
  scheduled task        /etc/cron.d/sysupdate    created 02 Mar 09:31   PRESENT
  referenced file       /var/tmp/.cache/upd      mtime   02 Mar 09:16   PRESENT
  web server log        access log for 09:14     retained 30 days       PRESENT
  shell session         no record                command logging off    ABSENT
  process attribution   no process holds it      runs on a schedule     ABSENT
  outbound flow         10.20.4.61 -> external   from the network team  PENDING

Six rows: three present, two absent and one requested. The two absences are the shell and the attribution, which are exactly the rows the red band held.

Writing the absences as rows rather than omitting them is the habit worth taking from this. A reader given four present rows assumes the other two were never relevant, and a reader given six knows precisely what is missing and why.

The exercise above gives you the same intrusion read from eleven days later.

There is a further consequence of the script finding worth drawing out, because it changes what a durable trace is worth. A deleted binary is recoverable from a running process, which is one of the genuinely useful properties LX0.1 mentioned, and a deleted script is not recoverable the same way.

What you get instead is the command line naming a path that no longer exists. That is still evidence, and it is a name rather than contents, which means establishing what the script actually did depends on the filesystem or on a log rather than on the process.

Most of what runs on a compromised Linux host is interpreted. Shell scripts, and things invoked through an interpreter, are far more common than compiled binaries dropped on a server, which makes this the ordinary case rather than the exception.

Reading the six stages against what survives shows where the course spends its time.

The fourth row is the one an investigation usually finds first, and it is also the only stage the intruder deliberately made durable.

Starting there and working outwards is the normal shape. It names a file, the file has timestamps, and the timestamps bound when the rest of it happened, which is enough to begin.

The first stage is in a log or nowhere. A web request answered abnormally is recorded by the web server if it logs that request, and module three is about reading logs that were written by programs with their own ideas about what matters.

The middle stages are filesystem work. A file written and made executable leaves timestamps, and the fact that it was later deleted leaves its own traces, which is module two.

Those stages are also the ones an intruder most often tries to remove, which is its own kind of trace. A file that was deleted leaves evidence of the deletion, and module two is as much about that as about the file.

The scheduled task is the one that survives best. It exists to run repeatedly, so it is still there eleven days later, and module six is about the places persistence lives and why coverage matters more than a hit.

It is worth noticing that this is a choice the intruder made against themselves. Persistence and durability are the same property, so the thing built to keep running is the thing most likely to still be there when somebody looks.

And the outbound connection is off the host. Module nine deals with live connection tables and module six with what a flow record elsewhere can and cannot attribute.

That one is unusual among the six for being outside anybody local's control. Whether it is available depends on what the network team retains and how quickly somebody asks, which makes it the first request to send rather than the last.

One more thing the early view gives that has no later equivalent, and it is the reason module nine exists. A process holding both the file it wrote and the socket it opened joins those two facts in a single reading, from the kernel, with no inference at all.

Eleven days later the same join is an argument. A file whose timestamp matches a flow record whose timing matches a schedule is a good argument and it is still a construction, and somebody can disagree with it in a way they cannot disagree with a descriptor list.

That is worth understanding before the techniques start, because it explains why so much of this course cares about being present rather than about being clever.

The intrusion itself ran forwards and the investigation runs the other way, which is worth expecting from the start.

It starts at the last stage. Somebody noticed outbound traffic, so the first evidence is the most recent and the least informative about how anything began.

Each step backwards is harder than the last. The scheduled task is easy to find and says what runs; the file it runs is next; what wrote that file is a filesystem and log question; and the original request is either in a log or gone.

The difficulty is not intellectual. Each step reaches further back in time, so each one depends on a trace with a longer lifetime, and the traces run out before the questions do.

That is why the early modules are reading skills. You cannot work backwards through an intrusion without being able to read each surface it crossed, and the surfaces are the same regardless of which intrusion you are chasing.

It also explains why the course does not open with intrusion patterns. Knowing what an attack looks like is useless without being able to read the surfaces it crossed, and the surfaces are stable while the patterns change.

$ cat working-backwards.txt
  START    outbound traffic to an external address, reported 13 Mar
  BACK 1   which process holds it now                  -> nothing; it runs on a schedule
  BACK 2   what schedules it                           -> a task created 02 Mar 09:31
  BACK 3   what does the task run                       -> a file in a temporary directory
  BACK 4   when was that file written                   -> 02 Mar 09:16, and by which account
  BACK 5   what was that account doing at 09:14         -> the web server log, if it recorded it
  STOPS    the original request, if the web server logged that route

Six steps backwards and the last one is conditional. Each step is a different surface, which is why the reading skills come first. Everything before it is on the host and the final step depends on a decision somebody made about logging long before the intrusion.

One thing worth resisting when reading a timeline like this, because it is the natural response to seeing how much expires. The conclusion that everything must be collected constantly, on every host, in case somebody eventually looks, is not what this argues for and is not affordable anywhere.

What it argues for is knowing which half you have. An investigation that understands it arrived eleven days late asks different questions, reaches for different sources, and words its findings differently, and it does all of that without anybody having collected more in advance.

The estates that do keep more get an easier version of the same job rather than a different one. A host with command logging enabled turns one of the two absent rows above into a present one, which is worth having and does not change the method.

The intrusion above is the course's spine, and each module takes one part of it seriously.

Three things recur in every module. What category of trace is this, how long does it last, and what does its absence mean, which are the questions the whole course is organized around.

The timing question comes first. An investigation at 09:15 and one on 13 March are different jobs against the same events, and knowing which one you are doing decides what to reach for.

Asking it explicitly is worth the habit. A responder who has not established when the events happened relative to now is reaching for sources without knowing whether they can possibly still hold anything.

That pattern recurs enough to be worth naming now. An absent trace is a statement about the present rather than about the past, and several modules later turn on somebody having confused the two.

And the absence question is the subtle one. No process holding the connection is not evidence that nothing runs; it is evidence that nothing runs right now, and the scheduled task explains both.

Running the same six stages against an investigation starting at 09:15 shows what the eleven days actually cost.

Everything in the red band is available. The shell is running, its command line names what invoked it, its descriptors show the file it just wrote and the socket it just opened, and its start time is computable to the second.

None of that requires the responder to be quick or clever. It requires them to be there, which is a different property and one an estate can arrange for rather than hope for.

$ ./compare-timing.sh --intrusion nge-web01
  stage                        at 09:15        at 13 Mar
  the request                  web log         web log, if retained
  the shell                    RUNNING         gone
  what invoked it              cmdline         inferred at best
  the file it wrote            on disk + fd    on disk, if not removed
  the outbound socket          in its fd list  a flow record naming the host
  the scheduled task           not yet created present, and the strongest trace
  attribution to a process     direct          reconstructed

Seven rows and the difference is concentrated in three of them. Four rows are effectively unchanged by eleven days. The later investigation can still establish most of what happened; what it loses is the direct link between the activity and a specific running process.

That is the reassuring half of an otherwise uncomfortable comparison. A late investigation is not working from nothing; it is working from four solid rows and reconstructing the join between them.

That last row is the real cost of the delay. Eleven days later everything is an inference joining a file, a schedule and a flow record, where at 09:15 one descriptor list would have connected all three.

It is also the row that decides how a finding gets worded. Direct attribution supports a plain statement, and a reconstruction has to say what it is built from, which module nine works through properly.

Practice

There is an exercise worth running on your own estate that makes all of this concrete, and it needs no incident. Take a recent change somebody made deliberately, a deployment or a configuration edit, and try to reconstruct it from the host alone a week later.

Most estates discover the same thing. The durable half is there and the attributing half is not: you can see what changed and when, and establishing who did it depends entirely on whether something was logging at the time.

That is the same shape as the intrusion above with none of the stakes. It is also the cheapest way to find out what your hosts would actually give an investigation, which is a question worth answering before somebody needs the answer urgently.

Start a process on any Linux machine, note what its process directory answers, then stop it and look again. Everything disappears at once, and seeing that happen is the fastest way to understand why the rest of this course cares so much about timing.

One thing to carry forward. This section sorted the traces into four categories by hand and left the ranking rough. Putting every source on a Linux host in order of how fast it decays is a more careful job, and it is the one that decides what a first hour should actually do.

The shape of an intrusion is established. Every source of evidence on a Linux host, ranked by how fast each one decays, is LX0.3.