Building a Linux Investigation Lab

Introduction

LX0.9 established that the technique work happens in your own lab. This section is about building it, because an orientation that leaves a reader unable to run the first command has not finished its job.

The requirement is smaller than people expect. Every measurement in the previous nine sections was taken on an ordinary Linux machine with no special hardware, no forensic distribution and no supplied image, and reproducing all of them needs one virtual machine.

That is worth saying at the start because the lab is where most people stop. A course whose first practical step appears to need infrastructure gets read rather than done, and read is not how any of this becomes usable.

You will finish able to state what the lab has to provide, know which modules need a second host, and understand why the account you work as changes what you can see.

Published lab requirements for courses like this tend to describe an ideal rather than a minimum, which puts people off starting. The list below is what the techniques in these ten modules actually need, derived by going through them rather than by specifying something comfortable.

Working through the modules and asking what each requires produces a short answer.

$ ./lab-requirements.py
  module  needs                                        hosts   also
  M0      read /proc, time a collection, count configs  1      root
  M1      processes, descriptors, deleted-but-open      1      root
  M2      timestamps, deletion, recovery                1      root, a spare image file
  M3      logs in several formats, rotation             1      root, and time
  M4      initial access against a service              2      a reachable service
  M5      privilege escalation                          1      root, a weak config
  M6      persistence across 13 mechanisms              1      root
  M7      lateral movement, the trust graph             2      ssh between them
  M8      containers read from the node                 1      a container runtime
  M9      live response under load                      1      root, something busy

  1 VM:  8 modules      2 VMs: 2 modules      more than 2: none

Eight of ten on a single machine. Two need a second host and nothing in the course needs a third, which puts the whole lab inside what a laptop will run.

That is the number worth leading with, because the usual reason people never build a lab is an impression that it needs a rack of machines. Two thirds of the way through the course, nothing has needed a second host.

The two that do need a second are the two whose subject is the relationship rather than the host. That is not a scale requirement dressed up: initial access needs somewhere to arrive from and lateral movement needs somewhere to go, and neither exists on one machine by definition.

That figure is worth checking rather than trusting. A first pass at this count said seven of ten, which was a hand-written summary disagreeing with the tally beside it, and the tally was right.

The second machine only matters for two subjects. Initial access needs something to arrive at, and lateral movement needs somewhere to move to, and both are about relationships between hosts rather than about anything inside one.

Deferring it is the right call rather than a compromise. Building both at the start means maintaining two machines through eight modules that never use the second, and the second is more useful built when its module explains what it is for.

The scenario is not a misconfiguration. It is the ordinary permission model doing what it is supposed to.

$ ./privilege-check.py
  running as uid 0   root=True
    count /etc files              OK
    read own process dir          OK
    read connection table         OK
    list all pids                 OK
    other processes' descriptors  25 readable, 0 denied

$ su -s /bin/sh nobody -c ./privilege-check.py
  running as uid 65534   root=False
    count /etc files              OK
    read own process dir          OK
    read connection table         OK
    list all pids                 OK
    other processes' descriptors  0 readable, 30 DENIED

Four rows identical and one completely different. Counting configuration, reading your own process directory, the connection table and the list of process identifiers all work as any account, and every descriptor list of another process is denied.

The four that match are the reason the failure is confusing. A student checks that their commands work, finds the process table and the connection table reading perfectly, concludes the lab is fine, and then meets a technique that returns nothing with no explanation.

That last row is most of the course. LX0.4 built its whole argument on reading another process, LX0.2 recovered a deleted file through a descriptor, and LX9 joins a socket to a process the same way, and none of it works from an unprivileged shell.

Listing what is not needed matters as much as what is. Most people who never build a lab have been put off by a requirements list written for a different kind of course, and the honest version here is one machine and an account.

That note is the practical difference between a lab and a spare server.

There is a second thing the permission split explains, which matters well beyond the lab. The account an investigation runs as decides what it can establish, and that is a real constraint on production hosts rather than a lab detail.

A responder with an account that cannot read other processes can still count configuration, read the connection table and enumerate persistence. What they cannot do is attribute a socket to a process, recover a deleted file through a descriptor, or see another account's command line, which is most of what the middle of an investigation consists of.

So the access question belongs in the first ten minutes alongside everything else in LX0.7. Establishing what your account can actually read takes one command and decides which half of the course is available on that host.

The build is short and the first thing to do with it is not a technique.

Six steps, of which two are immediate and four are deferred. Nothing on the list has to be arranged, purchased or requested, which is the property that makes it something somebody actually does this evening.

$ cat lab-build.txt
  1  one VM, any mainstream distribution, 2 GB memory is plenty
  2  an account you can become root from
  3  snapshot it, before anything else
  4  a second VM, when you reach the modules that need one
  5  a container runtime, when you reach M8
  6  optional: a second distribution, to see where defaults diverge

Six steps and the third is the one that changes how the rest goes. The first two take minutes and the rest are deferred until a module asks. A snapshot taken before any work means every module can start from a known state, and a lab that cannot be reset gets treated carefully rather than used.

Deferring is the point rather than laziness. A lab built entirely up front is a project; one built a step at a time as the modules need it is something that never becomes a reason to postpone starting.

Take the baselines first. LX0.6 needs a clean-build persistence list, LX0.3 needs retention figures and LX0.7 needs a collection timing, and all three are readings of an untouched machine that cannot be taken once you have started breaking it.

This is the one instruction in the section that cannot be followed late. Every other step can be done whenever, and a clean-build baseline taken after three modules of deliberate damage is a baseline of a damaged machine.

$ ./baseline-lab.sh --snapshot clean
  persistence mechanisms present            13
  instances across them                     39
  collection cost                       13.64 ms
  log windows                    4 logs, 7 d to 12 mo
  acquisition interfaces        1 of 4 present
  recorded                      before any module was run

Six figures and the last line is why they are worth anything. None of the six can be recovered once a module has run. Every one of them is what this machine looks like untouched, which is the comparison every later module is made against.

The persistence baseline is the one that matters most and expires fastest. Module six plants things, module five changes configuration, and a clean count taken afterwards describes a machine that has already been worked on.

Drawing the whole lab as two boxes and a bar is deliberate, because the usual illustration of a security lab is a network diagram with a dozen components. The requirement here fits in one picture and most of it is about what is not needed.

The red bar runs under both machines for a reason. It is the one requirement that is not about scale, and it is the one the scenario failed.

It is also the only requirement in the picture that cannot be satisfied later. A second machine can be added when a module asks for it and a snapshot can be taken at any point before the work, and an account without the necessary access makes every module after the third return nothing.

Running a check against the lab before starting is worth seeing first, because every one of those three failures is invisible from inside a working shell. Nothing errors, nothing is misconfigured, and the machine is simply not set up for the work.

One framing before the exercise. Working as your ordinary account is the correct habit everywhere except here, which is what makes the failure hard to spot: the student is doing the safe thing, the commands are right, and the output is empty rather than refused.

$ ./lab-check.sh
  virtual machine        present
  distribution           supported
  memory                 4 GB, sufficient
  snapshot               NONE TAKEN
  running as             uid 1000, not root, no sudo
  descriptor lists       0 of 30 readable
  baselines              not recorded
  verdict                the commands will run; most will return nothing

Eight checks and three failures, none of which stops a command running. A shell on that machine feels entirely normal. The lab is real, the distribution is fine and the memory is generous, and the three rows that matter were never checked.

That is the general shape of lab problems. Almost nothing announces itself, the machine behaves, and the failures are all in what the work returns rather than in whether it runs.

The exercise above gives you a lab that looks correct and returns nothing.

An idle virtual machine produces figures that make several modules look trivial.

LX0.3 measured eight of nine sources not moving at all. That was a real reading and it was taken on a quiet host, and a student who only ever measures an idle lab will conclude that decay is a theoretical concern.

That reading is honest and it is also the least representative figure in the module. A production host is always doing something, a lab is almost never doing anything, and the gap between them is where a student concludes the course overstated its case.

The fix is to generate load deliberately. Processes starting and exiting, connections opening and closing, and something writing to disk are what make the decay measurements produce non-zero answers, and all three are a short script rather than a workload.

$ ./generate-load.sh & ./decay-census.py --interval 4
  source                  t=0   t=4s   change
  process table           129     52      -77
  connection table         66     61       -5
  files in /etc           773    773       +0
  quiet host, same census:  all sources, +0

The same census, twice, with one difference. The difference is the load, not the census. Without load every row reads zero and the exercise teaches nothing; with it the process table sheds sixty per cent and the lesson is visible.

Running both versions once is worth the two minutes. Seeing the same script return all zeros and then return minus seventy-seven is what makes every volatility argument in the course concrete rather than something taken on trust.

That is worth knowing before deciding the measurements in this course were exaggerated. They were taken on hosts doing something, which is the state a production machine is always in and a lab almost never is.

A two-machine lab covers the techniques and leaves real gaps.

Two rows the lab covers, two it does not, and one that nothing in the course reaches. Knowing which is which stops somebody trying to build the wrong thing. The last row is worth knowing about rather than worrying about, because it is what the first year of doing this for real supplies.

The common mistake is trying to build the third row. Standing up forty virtual machines to practice a class comparison is a great deal of work to approximate something the scenarios give for nothing.

It has no history. Every real estate has hosts built years apart, undocumented exceptions and machines nobody can account for, and a lab built this morning has none of that, which is the category LX0.9 said the fiction cannot teach either.

It has no scale. Checking whether a key appears on forty other hosts, or comparing a machine against its class, are real moves that two virtual machines cannot demonstrate, and those are where the scenarios in this course do the work.

That is the gap the scenarios in this course exist to fill. Every examiner exercise puts a reader in front of a decision that depends on breadth, which is exactly the thing two virtual machines cannot supply and a fictional estate can.

And it has no consequence. Nothing in a lab is serving customers, so the containment pressure LX0.7 described as the common case is the one thing that cannot be simulated, and it changes every decision it touches.

That gap cannot be closed by any lab and it is worth naming rather than pretending otherwise. Deciding whether eleven seconds of collection is worth delaying an isolation is a different act when a customer is affected, and no amount of practice produces the pressure.

One more reason to practice both sides that has nothing to do with attacking anything. Planting something and then finding it is the only way to know a search works, and that applies to every check in this course rather than only to persistence.

A decay census that returns zeros, an enumeration that finds nothing, a comparison with an empty residue: each of those is either a clean host or a broken check, and the outputs are identical. The only way to tell them apart is to have seen the same check return a known non-zero answer.

That is the same argument the platform applies to its own gates. A clause that judges no rows prints clean and looks like a pass, and a search that has never found anything is in exactly that position.

The whole of the preparation is short enough to do before the first module.

$ cat first-hour.txt
  00:00  build one VM, any mainstream distribution
  00:20  confirm you can become root
  00:21  snapshot: clean
  00:22  baseline: persistence mechanisms and instances
  00:24  baseline: log windows per source
  00:25  baseline: collection timing
  00:26  start load generator, re-run the decay census, confirm non-zero
  00:30  begin module one

Eight lines and half an hour. The snapshot at 00:21 is the entry every later module depends on, and the census at 00:26 is what proves the lab can show decay at all.

Build one machine and snapshot it. Any mainstream distribution with an account that can become root, and the snapshot is what lets every later module start from a known state.

Any mainstream distribution is genuinely fine. The differences between them matter for specific paths and defaults, and none of them changes whether the techniques work, so the right choice is whichever you can build fastest.

Take the three baselines. Persistence, retention and collection timing, all on the untouched machine, because none of them can be taken honestly afterwards.

Add load before measuring anything volatile. An idle host returns zeros for every decay figure in the course, which reads as the course being wrong rather than the lab being quiet.

It is the step most likely to be skipped and the one that decides whether several modules land. A student who measures an idle machine and sees every figure return zero has been shown a technique that appears to do nothing.

Reset between modules rather than only at the start. Each module leaves the machine changed, and a snapshot named after the module it precedes is what makes the exercises repeatable rather than single-attempt.

And build the second machine when a module asks for it. Two of the ten need it, both of them are about relationships between hosts, and nothing is gained by building it early.

The techniques are only half the practice. The other half is producing the thing they are meant to find.

Install your own persistence and then go looking for it. Writing a cron entry, adding a unit, appending a key and then running the enumeration from LX0.6 teaches both sides in one pass, and the count you get back is checkable against what you put there.

Doing it in that order matters. Planting first and searching second means the answer exists before the question, so the search can be graded rather than merely run, which is the difference between practicing a technique and confirming it works.

$ ./plant.sh --count 3 --quiet
  planted 3 mechanisms across 3 locations
$ ./enumerate.sh --against-baseline clean
  13 present, 13 searched, 3 not in the baseline
  /etc/cron.d/updater          added 14:02
  /etc/systemd/system/mon.service  added 14:02
  /root/.ssh/authorized_keys entry 4   added 14:03
  found 3 of 3 planted

Three planted and three found, which is the only way to know the enumeration works. The count on the left is what makes the count on the right mean something. A search that has never been tested against a known answer is a search nobody can calibrate.

Getting three of three on a first attempt is also worth being suspicious of. It means the plant and the search were designed by the same person against the same list, which is a closed loop rather than a test, and the next paragraph is how to open it.

Then plant four and find three. Deliberately putting something in a location your enumeration does not cover is how the coverage argument in LX0.6 stops being an assertion, and the miss is more instructive than the hits.

Designing that miss deliberately is the part worth effort. Choosing a location your enumeration does not cover, running the search, and watching it report a clean residue is the clearest demonstration available that coverage and correctness are different things.

Practice

There is one habit worth establishing on day one that costs nothing and pays across every module. Snapshot before each module rather than only at the start, and name the snapshot after the module.

Modules interfere with each other more than expected. The persistence you plant in one module is still there when you enumerate in another, the load generator from LX0.3 changes every timing in LX0.7, and a machine that has been through six modules bears no relation to the one the baselines describe.

Per-module snapshots also make the exercises repeatable. Getting a module wrong, resetting and doing it again is how most of this is actually learned, and a lab that has to be rebuilt to reset gets one attempt per technique.

Build the first machine and take the three baselines before starting module one. It is under an hour, every later module compares against those figures, and they are the one thing that genuinely cannot be produced later.

One thing to carry forward. Everything in this module has been about what a Linux host exposes and how to approach it, and none of it has actually read a process in anger.

The lab is ready. Module one starts with what a running process actually exposes, and every reading in it will work on the machine you have just built.