The Evidence Landscape on a Linux Host

Introduction

LX0.2 sorted one intrusion's traces into four rough categories. This section does that job properly, because the rough version left the question of how fast anything actually moves entirely unanswered.

The received wisdom here is an order of volatility: a ranked list from memory down to archived media, learned once and applied everywhere. It is a perfectly reasonable ranking, and it is missing the one thing a decision actually needs, which is how much of each source is disappearing on the host in front of you.

You will finish able to measure decay for any source, separate that from the retention question it gets confused with, and place a source on both axes before deciding when to collect it.

The method is the same for every source and the results are not.

Counting entries is enough for this purpose and it is deliberately crude. A source whose count is stable while its contents change would be missed, and for deciding collection order the count answers the question at a fraction of the effort.

$ ./decay-census.py --interval 4
  source                  t=0   t=4s   change
  process table           129     52      -77
  connection table          2      2       +0
  socket inodes             2      2       +0
  neighbor table            1      1       +0
  mount table              27     27       +0
  loaded modules            0      0       +0
  logged-in users           0      0       +0
  files in /tmp            53     53       +0
  files in /etc           773    773       +0

One source moved and eight did not. Nothing about running the census suggests in advance which one it will be. The process table lost sixty per cent of its entries in four seconds, and everything else on that host was exactly as it had been.

That is not what the received ranking implies. It puts several of those sources near the top and would have a responder hurrying over things that had not changed in the interval, which is the scenario.

It also implies a spectrum where the measurement found a cliff. One source moving hard and eight not moving at all is not a ranked list with a top and a bottom; it is two groups with nothing in between.

The numbers are a property of the host rather than of Linux. A busy application server and a quiet file server give completely different answers to the same command, which is why this is a reading rather than a fact to memorize.

That is the part worth carrying rather than the figures. Somebody who memorizes minus seventy-seven has learned nothing transferable; somebody who runs two readings four seconds apart has a method that works on every host they meet.

The confusion is easy to make because the received ranking collapses them into one dimension. A single ordered list implies that being high on it means both changing quickly and being lost quickly, and those are separate properties with separate answers.

Decay is how fast a source changes. Retention is how long the old version is kept, and the two are independent.

$ ./retention-census.py
  logrotate configs present: 3
    dpkg           rotate=12   every=monthly
    apt            rotate=12   every=monthly
    alternatives   rotate=12   every=monthly

Twelve monthly rotations, which is a year. Nothing in the log names that figure; it lives in a separate configuration file. That figure is a decision somebody made in a configuration file, it differs per log, and nothing about the log itself tells you what it is.

The figure also differs per log on the same host rather than being a system-wide setting. Three configurations happened to agree here, and an estate with a dozen of them can easily have three different answers, none of which is written anywhere central.

Some sources have no retention policy at all.

$ ./retention-census.py --no-policy
  /proc                     synthesized per read; nothing is stored
  /etc                      kept until somebody edits it
  /var/lib/dpkg             kept until the package changes

Three sources and three different reasons. Grouping them as no retention would hide that the middle one keeps everything forever. The first keeps nothing because there is nothing to keep, the second keeps everything until somebody changes it, and the third is maintained by software with its own ideas.

Setting the sources out as four quadrants rather than a ranking is what makes the retention question visible at all. A list has one dimension and can only say earlier or later, where the thing a responder needs to know is sometimes neither.

That note is the correction the received order actually needs, and both readings take a few seconds.

There is a third question worth asking of every source, which the two axes do not cover and which decides more than either on some hosts. Does this source exist here at all, and was it enabled before the incident rather than after somebody noticed.

Command logging, process auditing and persistent journals are all optional, so a source can rank anywhere on both axes and simply not be present. That is not a gap in a collection; it is a fact about the estate, and it belongs in a record rather than in a silence.

The order of the three questions matters under pressure. Existence first, because a source that is not there needs no decision; then retention, because a source whose window has passed needs none either; and only then decay, which decides the order of what is left.

Putting all of the sources on both axes at once gives the map this section is building toward.

$ cat landscape.txt
  source                decay          retention        collect
  memory                continuous     none             now, if at all
  process table         -77 in 4 s     none             now
  connections           measured/host  none             now
  namespaces, cgroups   with processes none             now
  open descriptors      with processes none             now
  file timestamps       on write       until overwritten soon
  file contents         on write       until overwritten soon
  deleted file content  none           until reused      soon
  rotated logs          append-only    12 months here    days
  package database      on install     until changed     days
  configuration         on edit        until edited      days
  forwarded logs        none locally   somebody else's   request early
  flow records          none locally   somebody else's   request early

Thirteen sources across four collection windows. The windows matter more than the ranking within each of them. Five need collecting now, three soon, three within days, and two are requests to somebody else that should go out early because they have latency.

Within a window the order is free. Five sources that all retain nothing can be taken in any sequence, because they are all being read in the same few seconds and nothing is being traded against anything.

The last two rows are the ones people leave until last. They survive the host entirely, which makes them feel safe, and they belong to another team whose response time is the actual constraint.

The instinct is understandable and exactly backwards. Something outside the host cannot be destroyed by anything happening on it, which feels like safety, and the thing that actually limits it is how long somebody else takes to answer.

Plotting the two axes rather than listing a ranking is worth the space because the shape is the argument. A single ordered list has no way to show that a source sitting comfortably low on it can still be the one that expires first.

The amber point on the right is the one a single ranking hides. Rotated logs do not decay at all in the sense the process table does, and the window they cover slides forward until your period falls off the end.

It sits low on the vertical axis, which on a ranked list would put it near the bottom and imply it can wait. Its horizontal position is what matters, and no one-dimensional ordering has anywhere to put that.

Seeing the whole landscape as one output is worth the space before the exercise, because eleven seconds is the entire cost of knowing all of it. Nothing in that block required judgment; it required asking two questions of every source rather than one.

One framing before the exercise. Prioritizing by volatility is correct advice and the responder in it did what they were told. The ranking they applied is sound as a general statement about categories of evidence, and it cannot know which host they were standing on.

$ ./landscape.sh --host web01 --incident '02 Mar'
  measured decay      1 of 9 sources moving; process table -77 in 4 s
  no retention        5 sources: collect immediately
  until overwritten   3 sources: collect within the hour
  bounded retention   4 logs: 1 window already past the incident date
  external            2 requests: sent at 00:01
  absent              command logging not enabled on this host
  elapsed             11 s to produce all six lines

Six categories in eleven seconds, and the fourth row is a finding rather than a plan. The last row is the one that only exists because somebody looked for an absence. An expired window discovered at the start is a constraint to work around; discovered on day three it is a hole in a report.

The exercise above gives you a collection ordered by the received ranking.

Measuring is cheap, and it is worth being clear about what it actually decides.

It rarely reorders the top of the list. Kernel state goes first on any host because it keeps nothing, and the measurement confirms rather than revises that.

That is worth saying because it is the use most people would expect. Measuring in order to discover which volatile source is most volatile answers a question that does not arise, since they all go in the same pass.

Three of four change something, and the one that does not is the question the received ranking exists to answer.

It tells you how much hurry is warranted. A host losing sixty per cent of its process table every four seconds justifies a different pace from one where nothing moved, and the received ranking cannot distinguish them.

It quantifies what a delay cost. A responder who measured at the start can say how much of the process table turned over during a ten minute detour, which turns an unexplained gap into a figure.

That figure is only available if somebody took the first reading. A decay rate measured at the start converts a ten minute detour into a stated quantity, and the same detour with no measurement is an unexplained gap in a report.

And it finds the sources that were never going to move. Eight of nine on the measured host, which is eight things that could have been collected at any point in the next hour without loss.

Knowing that in advance removes most of the pressure from a collection. A responder who has established that eight of nine sources are stable can work carefully through them instead of hurrying through everything on the assumption that all of it is expiring.

Two rows of the landscape sit outside the host entirely, and they behave differently from everything above them.

They do not decay locally at all. A flow record exists on a collector somewhere and nothing you do on the host affects it, which makes it feel like the safest evidence in the set.

That property is genuinely valuable and it is also what makes them get postponed. Evidence that cannot be lost by waiting invites waiting, right up until somebody discovers the request takes four working days.

Their constraint is a queue rather than a clock. Requesting them takes a form, an approval or another team's attention, and the delay is measured in days rather than seconds.

That difference is worth naming because it needs a different response. Nothing you do faster on the host affects it, and the only thing that helps is asking sooner, which costs a minute and is easy to forget while dealing with a machine.

That is why they go out first despite ranking last. The ordering rule for everything else is decay; for these two it is latency, and sending the request in the first ten minutes costs nothing and saves days.

It is also the one instruction in this section that survives having no measurements at all. A responder who does nothing else from this module and sends the off-host requests in the first ten minutes has recovered most of the value in it.

$ cat first-ten-minutes.txt
  00:00  measure decay: two readings, 4 s apart, 9 sources
  00:01  send the flow record and forwarded log requests    <- latency, not decay
  00:02  collect everything with no retention: kernel state
  00:06  collect the filesystem timestamps
  00:10  everything remaining is stable and can wait

Five entries and the second is the one that looks out of place. It is also the only entry that depends on somebody outside the response. It is the least urgent source by decay and the most urgent by latency, and putting it second costs a minute.

One more thing the two axes explain that a ranking cannot, which is why the earlier modules of this course are ordered as they are. Modules one through three teach the sources with no retention and the ones that survive until overwritten, because those are the categories where a reader has to act rather than request.

The later modules mostly work with sources that are stable. Persistence locations, configuration, package state and the trust graph are all things that sit still while somebody thinks about them, which is why those modules can afford to be about judgment rather than speed.

That split is worth carrying as an expectation. Where a module feels urgent it is usually because its sources retain nothing, and where it feels reflective the sources are waiting for you.

The map is worth having in a form you can work through rather than recall.

$ cat landscape-check.txt
  MEASURED     decay on this host: 1 of 9 sources moving, process table -77 in 4 s
  NO RETENTION memory, processes, connections, descriptors, namespaces: collected
  UNTIL REUSED file timestamps, contents, deleted content: collected
  BOUNDED      rotated logs: 12 months here, incident is 11 days ago, safe
  STABLE       configuration, package database: collected, could have waited
  REQUESTED    flow records and forwarded logs, sent 00:01
  NOT PRESENT  command logging was not enabled on this host

Seven lines covering every source on the machine. Two of the seven record things that were checked rather than collected. The fourth is a check rather than a collection, and the last is the row that only appears if somebody looks for it.

Those two rows are what makes the record a claim about the host rather than a list of files. Checking a window and finding it expired, and looking for a source and finding it absent, are both results that only exist because somebody wrote them down.

Say which sources were absent rather than omitting them. A source that does not exist on a host is a fact about the estate, and leaving it out of the record makes an absence look like an oversight.

A reader given six collected sources assumes those were the six that existed. The same reader given six collected and one absent knows the estate made a choice, and that choice is often more useful to whoever owns the hosts than anything in the collection.

Give the rotation windows too. They are the half of the record that explains an absence, and an investigation that checked and found a window expired has established something rather than failed to find anything.

And record the decay figures alongside. They justify the order, they cost four seconds, and they are the difference between a sequence somebody can check and one they have to take on trust.

One row of the landscape fails in a way that nothing else on it does, and it is worth a section of its own.

A rotated log does not decay. Lines are appended and never modified, the file is stable between rotations, and every reading of it returns the same thing it returned yesterday for the period it covers.

That is worth stating precisely because the word volatile gets applied to logs loosely. Nothing in a rotated log is unstable; every line that is there will still be there tomorrow, and the file simply stops covering the period you care about.

What moves is the boundary. Each rotation discards the oldest kept file, so the window the log covers slides forward, and an incident date that was comfortably inside it last month can be outside it this month.

The rate it moves at is a configuration decision rather than a property of the data. A log rotating daily and keeping seven files covers a week regardless of how busy the host is, and one keeping twelve monthly files covers a year on the same machine.

$ ./log-window.sh --incident '02 Mar'
  log            rotate  every    oldest retained   incident inside window?
  auth           4       weekly   06 Feb            yes, by 24 days
  web access     12      monthly  01 Apr last year  yes, comfortably
  application    7       daily    06 Mar            NO, expired 4 days ago
  package        12      monthly  01 Apr last year  yes

Four logs and one of them is already gone. The expired one is the application log, which is usually the most specific. Nothing decayed and nobody deleted anything: a daily rotation keeping seven files covers a week, and the incident was eleven days ago.

The pattern is common enough to expect. Logs that produce the most volume rotate fastest, and the ones producing the most volume are usually the ones closest to whatever an investigation is chasing.

That check takes one command and is the cheapest thing in this section. It is also the only way to find out, because a log whose window has moved past your incident looks exactly like a log with nothing in it.

It is also worth doing before reading the log rather than after. An investigator who greps a log for an incident date and finds nothing has learned something ambiguous; one who checked the window first knows whether they have found an absence or an expiry.

Practice

There is a version of both censuses worth running on an estate rather than a host, and together they produce something most organizations do not have. A decay figure per host class and a retention figure per log answers, in advance, what any investigation on those machines will be able to reach.

The retention half is the one that produces findings immediately. Estates routinely discover that their busiest application logs keep a week, that two host classes built at different times disagree, and that the log somebody would most want during an incident is the one with the shortest window.

Neither census needs an incident and neither takes long. The first is two readings four seconds apart and the second is a directory of configuration files, and having both written down turns two live decisions into lookups.

Run the decay census on a machine you are responsible for and count how many sources actually moved. On most hosts the answer is one or two, and knowing which ones they are is what turns the received ranking from something memorized into something you have checked.

One thing to carry forward. Nearly every source in that landscape was read the same way, by opening a path and parsing what came back, which is a property of the platform rather than a coincidence of how the census was written.

The landscape is now mapped. Why so much of it is readable as ordinary files, and what that design decision gives an investigator, is LX0.4.