Why Investigating a Linux Host Is Different

Introduction

This module orients you before any of the techniques start. It is the very first sub of a course about investigating Linux hosts, and the place to establish what makes that a different job rather than the same job with different commands.

The usual framing is that Linux investigation is simply harder, because there is no central place to look. That is half right and it is the less useful half, because the same design that scatters the configuration exposes the running system in a way nothing else does.

You will finish able to say what Linux does not give an investigator, what it gives instead, and why the second thing shapes almost every technique in the rest of this course.

The claim gets made constantly and is worth measuring on an actual host.

$ find /etc -type f | wc -l
773
$ find /etc -type d | wc -l
151

Seven hundred and seventy-three files across a hundred and fifty-one directories. Nothing on the host knows that number; it took a walk of the tree to produce. That is one host's configuration, and none of it is indexed, queryable or held anywhere central.

That is one ordinary host rather than a heavily customized one. The figure varies with what is installed and the shape does not: configuration on Linux is a directory tree that has grown by accretion, with each program putting its own file wherever its authors chose.

The formats differ too.

$ ./format-census.py /etc
  freeform     38
  colon         19
  key=value     10
  ini-like       1
  json           1

Five distinct shapes in one sample. The first category is the largest, and freeform means a format specific to whichever program reads it. A parser that reads one of them reads none of the others, which is why so much Linux investigation is reading rather than querying.

That has a direct consequence for method. A question like what is configured to start at boot has no single answer to look up, and the course answers it by knowing where to look rather than by knowing what to ask.

It also explains a pattern students notice early and find frustrating. Many answers in this course begin with a path rather than with a concept, and that is not the author preferring detail; it is that on this platform the location genuinely is the knowledge.

That pairing is not a coincidence and it is the useful half of the observation. A system built on the idea that everything should be a file applies it to configuration, where it scatters things, and to kernel state, where it makes the machine unusually legible.

The same design that refuses to centralize configuration exposes the running kernel as ordinary files.

$ ls /proc | wc -l
101
$ ls /proc/self | wc -l
53

Fifty-three entries describing one process. Every running process on the machine has the same set. Not a subset chosen by a tool vendor and not an interface that might be unavailable, but the kernel's own record laid out as files anybody can read.

$ ./what-each-answers.py /proc/self
  status    identity, capabilities, both process numbers
  maps      what is mapped into it
  fd        every file and socket it holds open
  cmdline   what it was told to run
  environ   what it was started with
  cgroup    which container, if any
  ns        which namespaces it is in
  net       its network namespace view

Eight of the fifty-three, and each is a question an investigation asks. Reading them needs no tooling, no agent and no cooperation from anything except the kernel.

The other forty-five are not filler. Memory maps, scheduling statistics, limits, mount tables and namespace links are all there, and later modules reach for most of them, but these eight are the ones an investigation opens first on almost every host.

Setting the trade out as four rows rather than as a complaint is deliberate. The registry point is usually delivered as a difficulty, and it is half of a design decision whose other half is the reason this course can teach readings rather than products.

That note is the habit the whole course builds, and it is available to you from the first day.

There is a second thing the process directory gives that has no equivalent elsewhere, which is worth naming now because it recurs throughout. Every one of those entries is a current reading rather than a stored record, so nothing about it can be stale, edited by a previous administrator, or left behind by software that has since been removed.

That distinction between what a system currently is and what somebody once wrote down about it runs through the whole course. A package database records an intention from installation time; a process table records what the kernel is doing at the moment you read it, and only one of those can be wrong about the present.

It also sets an ordering habit worth having from the start. Where two sources answer the same question, prefer the one the kernel maintains for its own purposes over the one maintained by software for somebody else's, because the first has no reason to be inaccurate and no mechanism for being edited.

Where is the list of installed software has no answer, and three narrower questions do.

Each of the three is narrower than the original and each is worth something on its own. Together they cover most of what somebody actually wanted when they asked the unanswerable version.

What did a package manager install is answerable from its database. That covers most of a normal system and it is a claim about what was managed rather than about what is present, which are different sets.

What is present and unmanaged is answerable by comparison. Walking the binary directories and subtracting what the package database claims gives the remainder, and that remainder is where an investigation usually looks.

The subtraction is the whole technique and it is worth noticing how unglamorous it is. Two lists and a difference, using nothing but a package query and a directory walk, producing the set almost every investigation on this platform eventually cares about.

What is running right now is answerable from the process table. It is the smallest of the three sets and the most immediately useful, and it comes from the kernel rather than from any database that could be stale or edited.

It is also the only one of the three that cannot be out of date. A package database describes what was installed and a directory listing describes what is on disk, and both can disagree with what the machine is actually doing at this moment.

$ ./three-questions.sh --host web01
  managed by the package database        1,847 binaries
  present on disk in binary directories  1,863
  present and unmanaged                     16   <- the interesting set
  running at this moment                    41
  of the 41 running, unmanaged               2

Five figures and the last two are where an investigation starts. The first three exist to produce the fourth and fifth. Sixteen unmanaged binaries is a morning's work, and two of them running is a conversation.

Drawing the two panels side by side is worth the space because the point is a trade rather than a deficiency. Most descriptions of Linux investigation stop at the left panel, and the right one is where almost every technique in this course actually comes from.

The two panels are the same design choice, drawn twice. A system that puts configuration in ordinary files also puts kernel state in ordinary files, and an investigator loses one thing and gains the other.

Seeing what the rephrased question returns is worth the space before the exercise, because the useful output is unglamorous. The interesting set on a healthy host is small, dated, and mostly explicable, which is exactly what makes an inexplicable entry stand out later.

One framing before the exercise. Asking where the list of installed software is is not a naive question; it is the right question on several other platforms and it transfers badly rather than being wrong. What makes it fail here is an architectural fact rather than a gap in anybody's knowledge.

$ ./unmanaged.sh --host web01 --list
  /usr/local/bin/nge-deploy        not in any package database   mtime 2025-11-04
  /usr/local/bin/backup-rotate     not in any package database   mtime 2025-11-04
  /opt/vendor/agent/bin/collector  not in any package database   mtime 2026-01-19
  ... 13 more, all under /usr/local or /opt ...
  running now   collector (pid 2841), nge-deploy (pid 9914)

Sixteen entries and two of them running. The dates cluster, which is usually a deployment rather than a person. Nothing there is a finding yet, and a list of sixteen with dates and locations is a morning of ordinary checking rather than a mystery.

That is the tone worth setting for the whole course. Most of what an investigation turns up is explicable, the skill is in producing the short list quickly and honestly, and the genuinely alarming entry is rare enough that it stands out against a background somebody bothered to establish.

The exercise above gives you a question imported from another platform and what it should become.

The registry point is the famous one and three others do more work in practice.

Every row is a question that produces a confident wrong answer rather than an obvious failure, which is what makes them worth naming in an orientation.

Confident wrong answers are the expensive kind. A command that fails tells you to look elsewhere; a command that returns a plausible list tells you nothing is wrong, and the four rows above all fail in the second way.

Logging is per service and per format. There is no single event stream with defined identifiers, so a Linux investigation reads several logs written by several programs in several conventions, which module three is about.

The consequence is that there is no single question you can ask about what happened. Four sources may each hold a fragment, in four conventions, with four ideas of what a timestamp looks like, and assembling them is module three rather than a preliminary to it.

A deleted file can still be open. Removing a file removes its name, and a process holding it keeps the content reachable until it exits, which is a recovery route that does not exist on systems where deletion is final.

That one surprises people from every other background, and it is the single most useful thing in this section for somebody starting tomorrow. An intruder who removes their tool has removed its name, and module one recovers the content from the process still holding it.

Almost everything is optional. Auditing, process accounting, command logging and persistent journals are all things an estate may or may not have enabled, so the first question about any source is whether this host has it at all.

Those three differences appear repeatedly. Each one is a place where an assumption imported from another platform produces a confident wrong answer, and each has a module that deals with it properly.

One more thing that follows from all four differences together. A Linux investigation is unusually dependent on knowing what a particular estate chose to enable, because so much of the useful evidence is optional and so little of it is centralized.

That makes the first question on any host a survey rather than a search. Which logs exist, which are forwarded, whether auditing is on, and what the package database covers, all of which is cheap to establish and decides what the rest of the investigation can hope to answer.

The shape of the knowledge is unusual and worth knowing before you start acquiring it.

Most of what you learn is where things live. Which file holds which fact, what its format is, and what it does not record, which is why so much of this course is specific paths rather than general principles.

The tooling matters less than on other platforms. A path and a parser answers most questions, so the course teaches readings rather than products, and the readings keep working when the products are unavailable.

And the knowledge is durable. Kernel interfaces change slowly, the files this course reads have been in the same places for a long time, and what you learn here does not expire with a vendor's release cycle.

That is unusual enough to be worth weighing when deciding what to learn. A skill built on a particular product expires when the product does, and the files this course reads were in the same places a decade ago and will be in a decade more.

One practical consequence of learning it as paths worth flagging early. The knowledge feels like trivia while you are acquiring it, because a list of file locations is not obviously a skill, and it becomes a method once there is enough of it.

That is the awkward middle of learning this subject. Somebody three modules in knows a lot of paths and does not yet feel like an investigator, and the change comes from the paths adding up rather than from a further technique arriving.

The scenario's analyst was not wrong to ask, and the question needed rephrasing rather than answering.

$ cat rephrasing.txt
  ASKED       "where is the list of installed software"
  WHY IT FAILS no central registry exists; 773 config files, 4 package databases
  ASK INSTEAD  what did the package manager install       -> its database
               what is present and unmanaged              -> the difference
               what is running now                        -> the process table
  ON THIS HOST 1,847 managed, 16 unmanaged, 41 running, 2 both running and unmanaged

Five lines turning an unanswerable question into three answerable ones. Each of the three uses a different source, and one of them is the kernel rather than a database. The last line is where the investigation actually begins.

Notice which of the three came from the kernel. The package database is a file somebody maintains and the process table is the kernel's own account, which is the distinction the rest of this course keeps returning to.

The same applies to almost every figure in this module. Where a number appears in the rest of the course it came from reading a file the kernel or a package manager already maintains, which is why the techniques survive changes of tooling.

And notice that none of the three needed a product. Every figure in that block came from reading files and comparing lists, which is what most of this course turns out to be.

One honest thing to say about the scattering before moving on, since this is an orientation and not a sales pitch. The absence of a registry is a genuine cost and it does not get better with experience: a practitioner who has done this for a decade still has to know that a particular setting lives in a particular file, and there is no shortcut that makes that knowledge unnecessary.

What experience changes is the size of the search rather than its nature. Somebody who knows the estate knows which three files matter for a given question, where somebody new checks twenty, and both are doing the same thing.

That is worth knowing at the start because it sets a realistic expectation of the learning curve. The course front-loads locations deliberately, and the early modules will feel like memorization because a meaningful part of the skill is exactly that.

The architecture above decides the shape of everything that follows, and it is worth seeing the map once.

Four modules are about reading the machine. The filesystem, the logs, what ran and who was there: all of them paths and parses against sources that may or may not exist on a given host.

The split is worth seeing before starting rather than discovering three modules in. A student who expects each module to introduce a new technique will be surprised that module six is the same readings asked a different question, and one who expects that is reading it correctly.

$ cat course-map.txt
  M1  foundations        processes, descriptors, deleted-but-open files
  M2  filesystem         timestamps, what a deletion leaves, recovery
  M3  logs               several sources, several formats, the gaps
  M4  initial access     how somebody arrives on a Linux server
  M5  privilege          how they stop being an ordinary account
  M6  persistence        where it lives, and coverage over hits
  M7  lateral movement   the trust graph, read rather than inferred
  M8  containers         a container read entirely from the node
  M9  live response      a host that cannot be taken down

Nine modules, and the first three are the reading skills the other six use. Nothing after module three introduces a new place to read from; they introduce new questions. Nothing in the later modules is a new kind of operation; they are the same readings pointed at different questions.

The last two are the ones that could not have been written first. Containers and live response both rest on the earlier reading techniques, and both turned out to need their own ordering principles, which is what LX0.8 is about.

That is also why this orientation is the last thing written rather than the first. An introduction composed before the modules existed would have described what the course intended to teach, and this one describes what it turned out to establish.

Practice

There is a version of the three-question comparison worth running across an estate rather than one host, and it answers something most organizations do not know. How many binaries on a production machine are outside package management is a number nobody tracks, and it is the denominator for a whole class of investigation.

The answer is rarely zero and rarely alarming. Locally compiled tools, vendor agents installed by script, and things somebody copied in during a migration all show up, and knowing the normal figure for a host class is what makes an abnormal one visible.

It is also cheap enough to be worth doing before any incident. One comparison per host class, written down, and the next investigation starts by asking whether sixteen unmanaged binaries is the usual number here rather than by treating all sixteen as suspicious.

Run the three-question comparison on any Linux machine you have access to: what the package manager claims, what is actually present in the binary directories, and what is running. The gap between the first two is usually small and always interesting, and finding it once makes the rest of this course concrete.

One thing to carry forward. This section described the machine as a set of readable surfaces, which is true and static. What an investigation actually chases is a sequence of events across those surfaces, and seeing one intrusion end to end is what makes the surfaces mean anything.

The difference is now established. What an intrusion against one of these hosts actually looks like, start to finish, is LX0.2.