Everything Is a File, and What That Means for an Investigation

Introduction

LX0.3 mapped thirteen separate evidence sources, and nearly every single one of them was read the same way: open a path, parse whatever comes back. That was not a coincidence of how the census happened to be written.

It follows instead from a design decision the system applies almost everywhere, and that decision is usually quoted as a slogan. The slogan is roughly true, the places where it stops being true matter a great deal, and one of them will silently destroy an entire collection.

You will finish able to say what the principle actually gives an investigator, name where it breaks, and recognize the reading that returns nothing while appearing to succeed.

The slogan itself is worth stating carefully, because it is usually repeated as though it were a complete description of the system. What is true is narrower and more useful: a great many things that are not files are reachable through a filesystem interface.

The practical form of the idea is that questions with nothing to do with files are answered by opening one.

$ ./readable-as-files.py
  /proc/self/status             readable   process identity and capabilities
  /proc/net/tcp                 readable   the connection table
  /proc/self/mounts             readable   every mounted filesystem
  /proc/sys/kernel/hostname     readable   a kernel tunable, readable and writable
  /proc/self/cgroup             readable   container membership
  /proc/1/environ               readable   another process environment
  /proc/self/maps               readable   memory regions of a process

Seven different subjects and one operation. No two of them would share an interface on a system that had not made this decision. Network state, process identity, container membership and kernel configuration are not related to each other, and reading all of them uses the same verb.

That is what LX0.1 meant by paths rather than products. There is no interface to learn per subject, no agent that has to be present, and no vendor deciding which fields to expose.

It also explains why the techniques survive changes of tooling. A utility that reports connections is reading the same path you can read, so learning the path rather than the utility means the knowledge outlasts whichever program is installed.

The resemblance is close enough to be useful and not close enough to be trusted.

$ ./size-vs-content.py
  /proc/self/status         st_size=0        bytes actually read=1403
  /proc/uptime              st_size=0        bytes actually read=10
  /etc/hostname             st_size=3        bytes actually read=3

Two of three report a size they do not have. The one that is honest is the one stored on disk. The kernel files are synthesized when something reads them, so there is nothing to measure in advance and the size field is filled with zero.

Nothing about the path distinguishes them. Both are absolute paths, both open, both read, and the only way to know which kind you have is to know the directory conventions or to read and see.

The ordinary file behaves as expected, which is what makes this dangerous. A path under the configuration directory and a path under the process directory look identical, support the same operations, and disagree about something basic.

If every file misreported its size the problem would be obvious within an hour of writing any script. Half of them behaving correctly is what lets a collection work in testing and fail on the paths that matter.

Setting out the four differences rather than warning about one is what makes this transferable. Somebody told only about the size field will meet the timestamp problem or the repeat-read problem later and treat each as a fresh surprise, when all four follow from the same fact about synthesis.

That note is the scenario in one line, and following it costs nothing.

It is also the only rule in this section somebody needs to remember. The four differences explain why it is right, and a collection that simply never asks the metadata first is correct whether or not anybody understood the explanation.

There is a second consequence of synthesis that catches people later rather than immediately. A kernel file has no meaningful modification time, so anything that decides what to collect by looking at what changed recently will either take everything or take nothing, depending on which way the timestamps happen to fall.

That matters for incremental collection specifically. A script that collects files modified since the last run works correctly across the configuration directory and has no basis at all for deciding about the process directory, because those timestamps describe the reading rather than the data.

The same applies to comparison. Two collections taken a day apart can be diffed usefully for stored files, and diffing the synthesized ones compares two moments rather than detecting a change, which is a different question wearing the same clothes.

Running that script's logic against a list of ordinary kernel paths gives the figure exactly.

$ ./skip-empty.py --paths kernel-sources.txt
  files it would collect:        0
  files it would SKIP as empty:  5

  /proc/self/status       reported 0 bytes, actually 1403
  /proc/net/tcp           reported 0 bytes, actually 1050
  /proc/self/mounts       reported 0 bytes, actually 2159
  /proc/uptime            reported 0 bytes, actually 12
  /proc/self/cgroup       reported 0 bytes, actually 151
  total silently lost: 4775 bytes

Five of five skipped and nothing reported. A hundred per cent loss rate on exactly the half the kernel answers. The script did what it was written to do, the optimization is sensible on ordinary files, and the archive it produced looks like a successful collection.

The rate is what makes this worth a section rather than a footnote. A bug losing some evidence is a quality problem; one losing every synthesized reading while preserving every stored file produces a collection that is confidently, uniformly wrong about the machine.

The fix is not a special case for kernel paths.

Special-casing by path prefix would work and it is the wrong instinct, because it requires a list that has to stay correct. Reading the content needs no list and no knowledge of which kind of file arrived.

$ ./copy-by-reading.py /proc/self/status copied.txt
  copied.txt   1403 bytes

A copy that reads rather than sizes works everywhere. The same call handles both kinds without being told which it has. Nothing about it needs to know which kind of file it has, which is the point: the principle holds for reading and only for reading.

Most standard copy routines already behave this way. The failure in the scenario came from a check added in front of the copy rather than from the copy itself, which is the usual shape: the correct operation wrapped in an incorrect decision about whether to perform it.

Drawing both kinds of path into the same operation is the shape worth keeping. They converge at the read, which is the part that works identically, and everything that diverges is downstream of it.

The crossed arrow is the scenario. Nothing failed, nothing was logged, and an archive was produced that a reader would reasonably assume was complete.

Both branches leave the same box, which is the part worth internalizing. The read that works and the check that destroys the collection are one line apart in any script, and only one of them announces itself.

Seeing the archive it produced is worth the space before the exercise, because the failure is entirely invisible from the outside. Nothing is malformed, nothing errored, and the only signal is a column of zeroes that a size-based collection would also have produced from genuinely empty files.

One framing before the exercise. Skipping empty files is a reasonable optimization written by somebody being careful about archive size, and on every ordinary file it is correct. The script is not badly written; it is written for one of the two kinds of file the platform has.

$ tar -tvf collection-web01.tar
  -rw-r--r--  1403  etc/hostname
  -rw-r--r--   773  etc/passwd
  -rw-r--r--  2891  etc/crontab
  -rw-r--r--     0  proc/self/status
  -rw-r--r--     0  proc/net/tcp
  -rw-r--r--     0  proc/self/mounts
  -rw-r--r--     0  proc/uptime
  7 files, 5067 bytes

Seven entries and four of them empty. The three that worked are the three stored on disk. The archive is well formed, the file count looks right, and a reader who does not check byte counts per entry will take it as a complete collection.

Counting the entries is what most people check and it passes. Seven paths were requested, seven appear, the archive is valid, and every property except the one that matters agrees with a successful collection.

The exercise above gives you the archive that collection produced.

Repeating the principle as though it were a universal makes three significant gaps invisible.

Memory contents are not reachable this way on a modern host. LX9.6 found three of four acquisition interfaces absent on a test machine, so the address space of a running process is exactly the thing the principle does not hand you.

That gap is worth knowing early because the slogan implies otherwise. Somebody who believes everything is a file will go looking for a path to a process address space, and on most current hosts there is not one.

One yes and three noes, and the yes covers most of what this course does. The proportion is worth knowing before going looking for a path that cannot exist. The three noes are where an investigation has to acquire, reconstruct or ask rather than open.

Past state is not a file. The process directory answers what is true now and holds nothing about what was true an hour ago, which is why LX0.3 separated decay from retention.

That absence is the reason LX0.3 spent a section on retention. The kernel keeps nothing, so anything about the past has to have been written down by a program at the time, which is a different category of source with different failure modes.

And what is off the host is not a file at all. Flow records and forwarded logs live somewhere else, and no path on this machine reaches them regardless of how file-like everything else is.

That one is obvious stated plainly and routinely forgotten in practice. A responder deep in a host, reading paths for an hour, does not naturally switch modes to send a request, which is why LX0.3 put it in the first ten minutes.

Those three are where the investigation has to do something other than read. Requesting, acquiring and reconstructing are different operations, and knowing the principle has edges is what stops somebody looking for a path that does not exist.

Recognizing which of the four you are facing is most of the value. Reading is cheap and immediate, acquiring is expensive and conditional, reconstructing is an argument, and asking is a queue, and treating any of them as though it were reading wastes the time the others need.

The most investigative consequence of the principle is that one process can read another process's state directly.

An environment, a command line and a descriptor list all belong to some other process. Reading them needs the right permissions and nothing else, which is how almost every technique in this course reaches the thing it is examining.

The permission model is doing the work rather than any special capability. An account able to read those paths can read them, and an account that cannot gets nothing, which is why so much of this course assumes elevated access and says so.

$ ./read-another.sh --pid 9914
  cmdline   /bin/sh /var/tmp/.cache/upd
  cwd       -> /var/www
  exe       -> /usr/bin/dash
  fd/3      -> socket:[418822]
  fd/4      -> /var/tmp/.cache/upd (deleted)
  environ   PATH=..., HOME=/var/www, 14 variables
  status    Uid: 33 33 33 33, CapEff: 0000000000000000

Seven readings about a process nobody owns. None of them required the process to cooperate or even to be aware of it. The fourth row is a file that no longer has a name and is still reachable, which is the recovery route LX0.1 mentioned and LX0.2 qualified for scripts.

That property is what makes live investigation possible at all. Anything requiring cooperation from the thing under examination is worthless on a compromised host, and none of these readings ask for any.

That is the shape of a Linux investigation. Almost everything is a path under a process directory, read with ordinary permissions, with no cooperation from the thing being examined.

It is also why the techniques transfer between hosts so readily. The paths are the same on a web server and a database server, the permissions question is the same, and only the contents differ.

One more thing the descriptor directory gives that is worth knowing before the later modules, because it changes what a deleted file means. A link marked as deleted is not a record that something was removed; it is a live handle to content that still exists and can still be read.

That is the difference between a trace and the thing itself. Most of what an investigation recovers is evidence about an object, and this is the object, reachable through a path, for exactly as long as some process keeps it open.

It also explains why the process directory is worth reading before anything else on a live host. Every one of those handles disappears when its process exits, and with it the last reachable copy of anything that has already been unlinked from the filesystem.

Anything that collects from a Linux host will meet the zero-size problem, so it is worth building for deliberately.

Read to find out what is there. Never size, never test for emptiness, and never optimize a read away on the basis of metadata that was never populated.

That single rule covers the whole problem and needs no understanding of why. A collection that never consults metadata before reading is correct on both kinds of file without anybody having to know which kind arrived.

Record the byte count you actually read. It is the only figure that means anything for a synthesized file, and it lets a reader confirm the collection got something.

It also gives a reader something to check. A manifest listing a path and a byte count can be verified against the archive, and one listing only paths cannot distinguish a successful collection from the scenario.

Take repeat readings deliberately rather than by accident. Two reads of the same kernel path return different content legitimately, which LX9.8 turns into a custody argument rather than a defect.

The distinction matters for what gets claimed. Two differing reads used as corroboration is sound, and the same two presented as a verification of each other is the mistake LX9.8 takes apart at length.

Keep the two file kinds distinguishable in the output. An archive that mixes stored files and synthesized readings without marking which is which leaves a reader unable to tell an empty file from a failed read.

And test the collection against a kernel path before trusting it. Any script that handles the configuration directory correctly may still lose everything under the process directory, and the failure is silent in both directions.

One directory under each process is worth singling out on its own, because it turns the abstraction from a convenience into a join.

Every open file and socket appears there as a link. A process holding a file, a socket, a pipe or a device shows all of them the same way, and following each link says what it is.

The uniformity is the useful part rather than the novelty. An investigation asking what this process has open gets one answer covering files, sockets, pipes and devices together, where a system modeling each separately would need four questions and four answers to reconcile.

$ ./descriptors.sh --pid 9914
  fd/0  -> /dev/null                    a device
  fd/1  -> pipe:[418790]                a pipe to another process
  fd/2  -> /var/log/app/err.log         an ordinary file
  fd/3  -> socket:[418822]              a network socket
  fd/4  -> /var/tmp/.cache/upd (deleted) a file with no name left
  fd/5  -> anon_inode:[eventpoll]       a kernel object with no path at all

Six descriptors and six different kinds of thing. A tool would have to model each kind; a link does not. Each one is reachable by the same operation, and the last two are objects that have no home in the filesystem at all yet still appear as links.

The last two are the interesting ones for an orientation. A file with no name and a kernel object with no path are both things that have no filesystem existence, and both are reachable here because the abstraction covers them anyway.

The socket number is what makes it a join. That figure appears in the connection table as well, so a socket in a descriptor list and a row in the network table can be matched exactly rather than inferred by timing, which is the technique LX0.2 said the eleven day delay destroyed.

Joins like that are rare and worth recognizing. Most of what an investigation does is argue that two facts belong together on the basis of timing, and this is two records sharing an identifier, which is a different quality of claim entirely.

Practice

There is a test worth running once against whatever your team uses to collect, and it takes a minute. Point it at five kernel paths, run it, and count the bytes in the output rather than checking whether it reported success.

Scripts fail this more often than people expect. Anything that was written and tested against ordinary files has never met a path that misreports its size, and the optimizations that are correct everywhere else are exactly the ones that break here.

The same test is worth repeating whenever the collection changes. A refactor that introduces a size check, a skip for empty files or a copy that preallocates by size reintroduces the failure, and nothing in the output will say so.

Take any collection script your team relies on and run it against a handful of kernel paths, then count the bytes in what it produced. The failure mode here is silence rather than an error, so the only way to find it is to check the output itself rather than the exit code.

One thing to carry forward. Everything in this section was the kernel answering a question about the present. The other large category of evidence on a Linux host is written by programs about the past, and it behaves nothing like this.

The abstraction is now understood. What the logs on a Linux host do and do not record, and why the gaps are structural rather than accidental, is LX0.5.