The Shape of a Linux Investigation

Introduction

Six sections so far have described what to read on a Linux host, and where to find it. None of them has said anything about the order the work happens in, who is waiting for it, or what the whole thing looks like from the outside.

The usual answer is a lifecycle diagram: identify, contain, eradicate, recover, arranged in a circle. That is a description of a policy rather than of an actual day's work, and it gives no help at all with the one decision that actually has consequences.

There is nothing wrong with it as a description of what has to happen. It exists to make sure nothing is forgotten and to give people outside the response a shared vocabulary, and both are genuinely useful.

You will finish able to say which phases of an investigation are irreversible, why the cheapest one goes first, and what an investigation owes the people waiting for it.

Most people have never measured it, which is why the estimate people carry is wrong by orders of magnitude. Collection feels like the heavy part of an investigation because it is the part with the most steps, and step count is not duration.

Timing the whole collection gives a figure worth knowing before planning anything around it.

$ ./time-collection.py
  process table           9.80 ms
  connection table        1.86 ms
  mounts                  0.10 ms
  modules                 0.01 ms
  persistence dirs        1.81 ms
  descriptor lists        0.06 ms
  TOTAL                  13.64 ms

Six sources and under fourteen milliseconds. That is the whole of what the first seven modules of this course read from a host. That is the process table, every connection, the mount and module tables, the persistence locations and a sample of descriptor lists, taken together.

The process table dominates, at about seventy per cent of the total, because it is the only step that walks a directory rather than reading a single file. Everything else is a handful of opens.

The figure is small enough to change how the first minutes are planned. Nothing in that list needs scheduling, approval or a decision about priority, because the entire thing costs less than the time between two keystrokes.

It also changes what is worth arguing about. Debating whether to collect the connection table before or after the process table is a discussion about two milliseconds, and the only ordering question with real stakes is whether any of it happens before the host changes.

Setting the collection against the rest of an investigation shows the shape.

$ cat where-the-time-goes.txt
  phase                          cost            repeatable later?
  collect everything cheap       14 ms           NO
  acquire memory, if justified   4 to 87 min     no, and only if the host allows it
  wait for off-host records      hours to days   yes, but the queue restarts
  assemble a timeline            hours           yes
  enumerate persistence          minutes         yes
  decide what it means           the rest        yes, and repeatedly

Six phases and one of them is milliseconds. The ratio between the first row and the last is not a rounding difference. The cheapest phase by four orders of magnitude is also the only one in the list that cannot be done again tomorrow.

The second column is the one that does the work, and it is worth reading on its own. Four of the six phases can be run again next week against the same evidence, which means nothing about them has to be decided under pressure.

That inversion is the whole shape. Effort does not decide the order, because the expensive phases are all repeatable and the free one is not.

Most professional instincts run the other way. Doing the cheap thing first feels like avoiding the hard work, and here the cheap thing is the only part that has a deadline.

Framing the split as reversibility rather than as urgency is what makes it decidable. Urgent is a judgment somebody can disagree with; whether a thing can be done again tomorrow is a question with an answer, and the four rows below sort themselves once it is asked.

That note is the correction to the diagram, and the figures behind it are all measured.

It is a correction to the order rather than to the content. Every phase the lifecycle names still happens, and one of them moves to the front because it is the only one with a deadline.

There is one row in that table worth more attention than the others, because it behaves unlike the rest. The off-host requests are slow and not decaying, which puts them in neither the collect-now group nor the whenever group.

Their position is decided by latency rather than by either axis. Nothing is lost by sending the request late except the days between when it was sent and when it could have been, and those days are real even though nothing expired.

That makes three ordering criteria in one sequence. Reversibility puts the readings first, latency puts the requests second, and everything else is genuinely free to happen in whatever order suits the people doing it.

It is also the row most often forgotten entirely. A responder deep in a host is not naturally thinking about a form to submit to another team, which is why it belongs in a written sequence rather than in somebody judgment on the day.

Ordering by reversibility produces a short opening that looks nothing like the lifecycle.

$ cat first-ten-minutes.txt
  00:00.00  read what backs the destination, mount options, fork counter
  00:00.03  collect everything cheap                            14 ms
  00:00.05  take the same readings again 3 s later              decay + comparison
  00:01     send the off-host requests                          latency, not decay
  00:02     check acquisition feasibility and justification
  00:04     enumerate persistence against this host
  00:10     everything remaining is repeatable and can wait

Seven entries and the host work is finished by minute four. Three of the seven are readings and the rest are requests or checks. Scoping, stakeholder updates and containment planning all happen after this, because none of them loses anything by waiting and all of the readings above do.

Nothing in those four minutes required knowing what the incident was. That is the property that makes the ordering possible: the readings are the same regardless of what turns out to have happened, so they do not have to wait for an understanding that takes an hour to form.

The second entry is the one people resist. Collecting before understanding the incident feels backwards, and the collection costs fourteen milliseconds and answers questions nobody has asked yet.

The objection is reasonable and the answer is arithmetic. Collecting without a hypothesis wastes fourteen milliseconds when it turns out to be unnecessary, and recovers an entire category of evidence when it does not.

Drawing both orders as bars is worth the space because the proportions are the argument. Every description of this trade-off in prose makes it sound like a judgment call, and at this scale it is not one.

The two bars contain the same work. No activity was removed, shortened or added. Only the first five millimeters moved, and that is the entire difference between a collection taken on arrival and one taken an hour into the incident.

That is worth dwelling on because it means the argument costs nothing to accept. There is no trade being proposed, no corner being cut, and no activity anybody has to give up in exchange.

Seeing what ten minutes can actually produce is worth the space beforehand, because the objection to collecting first is usually that there is nothing to say yet. There is, and it is enough for somebody to make a decision with.

One framing before the exercise. Spending the first hour on scoping is not negligence; it is following a published lifecycle in the order it is written, and every activity in that hour is work somebody has to do. What is wrong is only the position of one fourteen millisecond step.

$ cat assessment-10min.txt
  AT        16:04, ten minutes after the alert
  READING   web01 carries 2 persistence instances not in the class baseline
  READING   no process currently holds the reported outbound connection
  READING   acquisition not possible on this host: interfaces absent
  SENT      flow record and forwarded log requests, 16:01
  NOT YET   scope across the other 41 web hosts
  STATUS    current assessment, will be revised. Not a conclusion.

Six lines produced from fourteen milliseconds of reading and a few minutes of checking. Three are readings, one is a request, and two describe what is not yet known. The last line is what lets a host owner act on it without being misled about how settled it is.

The exercise above gives you the first hour of that response.

An investigation runs inside an organization, and three groups need different things from it at different times.

Whoever owns the host needs to know whether to act. They are deciding about containment, downtime and customers, and they need a current assessment rather than a finished one, which is usually available within the first ten minutes.

They are also the person most likely to be told nothing for an hour. An investigation with no assessment to give tends to give none at all, and the host owner makes their decision anyway, on less.

Four audiences and only the last one wants a finished account. The first three are making decisions while the investigation is still running. The other three are acting on incomplete information by design, which is why the assessments have to be dated and phrased as readings.

Nothing about that is a failure of the investigation. Decisions about downtime and resourcing cannot wait for certainty, so the useful skill is producing something honest and dated at ten minutes rather than producing nothing until it is safe.

Whoever owns the response needs scope. How many hosts, which accounts and whether it is still happening are questions about breadth rather than depth, and they change what resources arrive.

And whoever reads the eventual report needs provenance. That is the custody and handover work from LX9.8 and LX9.9, and it is the only audience whose needs are served by things written at collection time rather than by conclusions.

That audience is the one whose needs cannot be met retrospectively. The other three can be given a better answer tomorrow, and this one is reading a document whose quality was fixed at collection time, months earlier.

$ cat who-needs-what.txt
  00:10  host owner        "one host, active persistence, no evidence of spread yet"
  00:30  response owner    "scope check: same key on 0 of 41 other web hosts"
  day 2  report reader     coverage, custody, absences, and what each claim rests on
  standing                 every number above is a reading, and each is dated

Three audiences and three different deliverables. Giving any of them the wrong one wastes everybody time. The first two are current assessments that will be revised, and only the third is a finished account.

The common failure is giving the host owner the report reader deliverable. A careful, qualified, provenance-heavy answer is exactly right in two days and useless to somebody deciding in the next ten minutes whether to take a service down.

The hardest part of the shape is reporting honestly while the picture is incomplete.

$ cat wording.txt
  READING      "web01 carries 2 persistence instances not in the class baseline"
  READING      "no process currently holds the reported outbound connection"
  NOT          "the incident is contained"
  NOT          "web01 is clean"
  STATUS       current assessment at 16:04, will be revised
  WHY          the first two survive being quoted by somebody who was not here

Six lines, two of them things not to write. The last is the test: a reading that travels intact stays true, and a conclusion may not.

An assessment at ten minutes is not a conclusion. It is what the readings support so far, it will be revised, and saying so is what lets somebody act on it without being misled.

Saying that explicitly is what makes it usable. An assessment offered without qualification gets treated as settled, and one offered with a paragraph of caveats gets ignored, so a single status line does more work than either.

The phrasing does the work. One host shows persistence installed 02 March is a reading; the incident is contained is a conclusion, and the difference matters most to the person deciding whether to take a service offline.

And revisions are expected rather than embarrassing. An investigation that never updates its assessment has either finished in ten minutes or is not saying anything, and neither is likely.

Setting that expectation early is worth doing deliberately. A host owner told at ten minutes that the picture will change is prepared for the update; one told nothing until it is certain experiences the first revision as the investigation having been wrong.

That is the same discipline as every claim in this course. Claim the reading rather than the host, state what it rests on, and let the conclusion follow when the evidence supports one.

It is also why the wording rules keep reappearing. Nothing about a live reading, a log line, a persistence instance or a timeline changes the basic move, which is to say what was observed and let the conclusion be a separate sentence somebody can disagree with.

One more thing about assessments that only becomes obvious after a few incidents. The first one you give tends to be repeated back to you for the rest of the response, quoted by people who were not there and did not hear the qualifications.

That is an argument for wording rather than for silence. A reading that travels intact is still true when it arrives somewhere unexpected, and a conclusion that travels intact may not be, which is why the phrasing above is worth the effort at the moment of writing.

It also explains why dating every assessment matters. Two hours later the same sentence describes a picture that has moved, and a timestamp is what lets somebody notice they are quoting something superseded.

Nothing in the evidence announces that the work is complete.

$ cat closing.txt
  COVERED      every source on the landscape collected, or its absence recorded
  COVERED      13 of 13 persistence mechanisms present on this host
  OPEN         41 external destinations, none attributed
  OPEN         flow records requested 16:01, not yet returned
  ABSENT       command logging not enabled; application log window expired
  SCOPE        this host; three others in the segment not examined
  MEANS        every question the evidence can answer has been asked of it

Seven lines and three of them describe what was not reached. A report without those rows reads as though those questions were answered.

Coverage is the closest thing to an end. Every mechanism present was searched, every source on the landscape was collected or its absence recorded, and every question the evidence can answer has been asked of it.

It is a weaker ending than most people want and it is the honest one. An investigation cannot establish that nothing else happened; it can establish that every source capable of answering was asked, which is a checkable claim rather than a feeling.

The open items are the honest remainder. LX9.9 established that a handover names what was not reached, and a finished investigation does the same thing for itself rather than implying nothing was left.

They also protect whoever comes next. A named open item is somebody else work; an unnamed one is a question that never gets asked, and the difference is one line in a document.

Name what the investigation chose not to pursue. Scope decisions, deferred hosts and questions judged not worth the cost are all legitimate, and recording them separates a bounded investigation from an incomplete one.

And the absences belong in the conclusion. Command logging was not enabled and the application log had rotated past the window are both findings, and a report without them reads as though those questions were answered.

That is the same move LX0.6 made about persistence coverage and LX0.5 made about unrecorded categories. A report that names what could not be answered is bounded; one that is silent about them reads as complete and is not.

The ordering argument assumes somebody is letting you work, and sometimes nobody is.

That is the common case rather than the exception on anything customer-facing. Most responders meet containment pressure long before they meet an unhurried host, so an ordering rule that collapses under it is not much use.

A host actively causing harm gets contained. Data leaving now, an attacker moving between machines, or a service damaging customers are all reasons to pull the plug before any of the above, and that decision belongs to whoever owns the business risk rather than to the investigation.

That call is not the investigation to make, and pretending otherwise is how responders end up arguing for evidence against a customer impact they cannot see. The business owns the risk and the investigation owns what is possible within the time it is given.

What the fourteen millisecond figure changes is the conversation. Asking for fourteen milliseconds before containment is a different request from asking for an hour, and it is one almost anybody will grant.

Nobody refuses fifteen seconds. The request that gets refused is an open-ended one, and most responders have never measured their own collection well enough to make a bounded request instead.

$ cat containment-request.txt
  ASKED      "before you isolate it, 15 seconds on the host"
  GRANTED    yes
  TAKEN      cheap block, twice, 3 s apart              14 ms x2
             acquisition feasibility check               30 ms
             persistence enumeration                     1.8 s
  ELAPSED    11 s total
  RESULT     everything that would have died with containment, captured
  THEN       isolated at 16:04:11

Eleven seconds against an isolation that was going to happen anyway. The isolation time in the record is the real one. Nothing in the response was delayed meaningfully, and every reading that containment would have destroyed exists.

Recording both is what makes the account defensible later. Somebody reviewing the response can see that the delay was eleven seconds rather than inferring it, and the readings taken in that window are dated inside it.

That is the practical form of the whole section. The argument is not that investigation outranks containment; it is that the irreversible half of an investigation is short enough to fit inside the pause before one.

Reframing it that way also makes it winnable. An investigation arguing for time against a business risk loses and should; one asking for eleven seconds inside a decision already taken is not asking for a trade at all.

Practice

There is an exercise worth doing before any incident, and it is one command plus a stopwatch. Time your own collection script on a representative host from each class, and write the figures where whoever responds will see them.

The number is the argument. A responder asking for time on a host during an incident is negotiating against an unknown, and one who can say eleven seconds is negotiating against a figure somebody can accept immediately.

It is also worth knowing which host class is slowest. A machine with thousands of processes takes meaningfully longer to walk than the one measured here, and the difference is still seconds rather than minutes, which is the point.

Time your own collection on a representative host and keep the figure. It is almost certainly under a second, and knowing that number is what makes the argument for collecting first concrete rather than a matter of preference.

One thing to carry forward. This section ordered work by what expires, which is one answer among several. Each module in this course found a different property deciding its order, and the variation is the part worth understanding.

The shape is now established. What decides the order of work within each module of this course, and why the answer is different every time, is LX0.8.