Where Persistence Lives on a Linux Host

Introduction

LX0.5 established that most of what an intrusion does is never recorded anywhere. What survives instead is whatever was built to keep running, which makes persistence the most reliable thing an investigation finds.

That reliability is the intruder working against themselves, as LX0.2 noted. Anything designed to survive a reboot is by definition designed to still be there when somebody eventually looks.

It is also the thing most often searched badly. The usual approach is a published list of locations worked through until something turns up, and both halves of that are wrong in practice: the list does not match the host in front of you, and turning something up is not by itself a reason to stop.

Both halves of the usual approach come from published references, which is why the mistake is so widespread. A long list read as a procedure produces a search that is simultaneously too broad for the host and too shallow to finish.

You will finish able to enumerate persistence against the host rather than a list, explain why finding one instance settles nothing at all, and recognize what a clean result actually means.

Published lists are long because they cover every distribution and every configuration anybody might meet. That is the right way to write a reference and the wrong way to search a specific machine.

Checking a standard list of mechanisms against an actual machine produces a split.

$ ./persistence-census.py
  PRESENT: 13
    /etc/cron.d                  drop-in cron jobs
    /etc/cron.daily              daily scripts
    /etc/systemd/system          system units
    /usr/lib/systemd/system      vendor units
    /etc/systemd/user            user units
    /etc/init.d                  sysv init scripts
    /etc/profile                 login shell profile
    /etc/profile.d               login shell drop-ins
    /etc/bash.bashrc             interactive shell rc
    /etc/ld.so.conf.d            library search path
    /etc/modules-load.d          kernel modules at boot
    /root/.ssh/authorized_keys   ssh keys
    /etc/passwd                  accounts and shells
  ABSENT: 10

Thirteen present and ten absent. Nothing about the host is unusual; that split is what a normal machine looks like. A checklist written for a different host searches all twenty-three, finds nothing in ten places that do not exist, and reports those ten as clean.

The ten absent ones are not exotic either. Several are the standard cron directories, missing because this machine schedules through a different mechanism, which is an ordinary build decision rather than hardening.

The absences are not gaps in the host. A machine without a system crontab or per-user cron directories simply does not use those mechanisms, and reporting them as checked is reporting a search that could not have found anything.

They also vary for uninteresting reasons. Distribution, version, whether a desktop environment is installed and how the image was built all change which mechanisms exist, and none of that variation has anything to do with security.

That is worth distinguishing in a record. Not present and searched and found nothing are different results, and a list that conflates them produces false confidence proportional to how many locations it names.

It also compounds with the length of the list. A checklist naming forty locations on a host that has fifteen of them looks like far more thorough work than one naming fifteen, and it is the same search with twenty-five extra rows that could not have found anything.

The mechanisms are not layers of a single scheme. Each one independently restarts something.

$ ./independence.py
  mechanism                 alone, enough?  what triggers it
  a systemd unit            YES             boot, and restart on failure
  a cron drop-in            YES             a schedule, regardless of boot
  a shell profile entry     YES             anybody logging in
  an authorized key         YES             nothing; it grants access directly
  a library preload         YES             every process that starts
  an account shell          YES             that account being used

Six mechanisms and six different triggers. No two of them fail together, because no two share anything. Removing any one of them leaves the other five working, because none depends on any other.

Independence is what makes the count meaningful rather than alarming. Six routes is not a sign of a sophisticated intruder; it is a property of the platform, available to anybody who reads a list of places to write a file.

That is the shape of the scenario exactly. A cron drop-in was found and removed, the removal was real, and five independent routes were never examined.

It is worth noticing that the responder was not careless. They searched, they found something real, they removed it, and the only step missing was continuing through a list that was neither long nor difficult.

Framing it as coverage rather than as a warning about thoroughness is what makes it actionable. Be thorough is advice nobody can act on; thirteen of thirteen searched is a number somebody can produce, check and disagree with.

That note is the whole difference between the scenario and a complete search.

It is also the difference between two reports that read almost identically. Persistence removed and every mechanism present was searched are one sentence apart, and only one of them describes what was actually established.

There is a further reason the independence matters that goes beyond completeness. An intruder installing more than one mechanism is not being thorough for its own sake; they are expecting somebody to find one, and the second exists precisely to survive that.

That changes what a removal means. Taking out the mechanism you found is exactly what the arrangement anticipated, and reporting it as resolved is the outcome it was designed to produce.

It also suggests where to look next. Two mechanisms installed minutes apart, as in the census above, is one event rather than two, and the timestamps that connect them are the same filesystem evidence LX0.2 relied on.

The fix for both of those problems is a single pass that records three separate things per location.

$ ./enumerate.sh --host web01
  location                     present  searched  found
  /etc/cron.d                  yes      yes       1  sysupdate
  /etc/cron.daily              yes      yes       0
  /etc/systemd/system          yes      yes       0
  /usr/lib/systemd/system      yes      yes       0
  /etc/systemd/user            yes      yes       0
  /etc/init.d                  yes      yes       0
  /etc/profile.d               yes      yes       0
  /etc/bash.bashrc             yes      yes       0
  /etc/ld.so.conf.d            yes      yes       0
  /etc/ld.so.preload           NO       n/a       n/a
  /etc/modules-load.d          yes      yes       0
  /root/.ssh/authorized_keys   yes      yes       1  key added 02 Mar
  /etc/passwd                  yes      yes       0
  13 present, 13 searched, 2 found

Thirteen rows and two findings rather than one. The count of rows is the claim; the count of findings is not. The second is the authorized key, which no amount of looking at running processes would have revealed and which the scenario's search never reached.

It is also the more serious of the two. A scheduled task runs something on a machine an investigation is already examining; a key grants somebody the ability to come back, from outside, whenever they choose.

The extra finding came from nothing clever. It came from continuing down the list past the point where something had already been found, which is the only difference between this search and the one in the scenario.

The preload row is the honest kind of absence. It is marked as not present rather than as clean, so a reader can tell the difference between a mechanism that was ruled out and one that does not exist here.

Three columns rather than two is the whole mechanism for that. A single found column forces every location into hit or clean, and the third state is where most of the misleading confidence comes from.

Drawing six arrows into one box is the shape worth keeping. The removal crosses out a box at the top and changes nothing about the five arrows still arriving, which is the entire argument against stopping on a hit.

The dashed arrow is the one the scenario removed. Five solid arrows still reach the box, and nothing about removing the first affected any of them.

Drawing the removed box crossed out rather than deleting it is deliberate. The work was done and the arrow is gone, and the diagram still shows five routes arriving, which is the state the report described as resolved.

Seeing the finished version is worth the space before the exercise, because the difference from the scenario is one line of ordering. The same two findings, reported after a coverage claim rather than instead of one, support something entirely different.

One framing before the exercise. Removing the cron drop-in was correct work, done properly, and the removal was real. What went wrong is the sentence written afterwards, which claimed something the search had not established.

$ cat finding-persistence.txt
  HOST       web01
  MECHANISMS 23 checked against this host; 13 present, 10 not present
  SEARCHED   13 of 13 present mechanisms
  BASELINE   web class, recorded 2026-01-14, 39 instances
  FOUND      2 not in the baseline
             /etc/cron.d/sysupdate                added 02 Mar 09:31
             /root/.ssh/authorized_keys entry 3   added 02 Mar 09:18
  MEANS      every mechanism present was searched. Not: the host has no persistence.

Seven lines and the coverage claim comes before the findings. Reversing those two lines produces the scenario exactly. The last line is the one that distinguishes a finished search from a lucky one.

The exercise above gives you that report and the host it was written against.

One row of the census behaves unlike all the others and deserves naming early.

Every other mechanism eventually causes something to run, which means every other mechanism eventually becomes visible to somebody watching the machine. This one never does, and an investigation that has only ever found persistence by noticing something running has no habit that covers it.

One reliable yes and three conditional or absent. The reliable one is also the mechanism least often used. That is the argument for reading the filesystem rather than relying on what the live host happens to be doing when you arrive.

The conditional rows are the deceptive ones rather than the absent row. A shell profile entry that happens to be visible because somebody is logged in teaches a responder that live reading finds this class of thing, right up until the day nobody is logged in.

An authorized key runs nothing. It sits in a file, it starts no process, it opens no socket, and it appears in no process table on any host at any time.

That makes it invisible to everything in LX0.4. Reading the process directory, the connection table and the descriptor lists is the most powerful technique on a live Linux host, and none of it touches a key waiting in a file for somebody to use.

That is worth sitting with, because the process directory has been the answer to almost everything so far. Here it is the wrong instrument entirely, and no amount of skill with it helps.

So persistence needs a filesystem search rather than a live one. That is a different operation from everything the earlier sections taught, and it is the reason this course treats persistence as its own module rather than a consequence of process analysis.

That also means persistence survives things the rest of the evidence does not. A key in a file is still there after a reboot, after the process table has turned over a thousand times, and after every log covering the period has rotated away.

Enumerating every mechanism on a working host returns a great deal that belongs there.

That is the part most descriptions of persistence hunting skip, and it is the part that decides whether the technique is usable. A method that produces forty results on every healthy machine is not a method somebody will run twice.

Most entries are the system doing its job. Log rotation, package maintenance, certificate renewal and monitoring agents all install exactly the kind of scheduled task an investigation is looking for, and they outnumber anything else by a wide margin.

The ratio is worth internalizing before the first real search. Something like forty legitimate instances to zero or one malicious is the normal case, so an approach that treats every instance as suspicious generates a day of work and no findings.

That is why a baseline is worth more than a signature. Knowing what this class of host normally carries turns a list of forty scheduled tasks into a list of one that was not there last month, which is a question anybody can answer.

$ ./compare-baseline.sh --host web01 --class web
  mechanism instances on this host        41
  matching the baseline for this class    39
  not in the baseline                      2
    /etc/cron.d/sysupdate                  added 02 Mar 09:31
    /root/.ssh/authorized_keys entry 3     added 02 Mar 09:18

Forty-one down to two by subtraction. The subtraction took seconds and replaced a day of judging each instance on its merits. Neither of the two is proven malicious by being unusual, and both are now a short question rather than a long search.

Judging on merits is what a search without a baseline forces. Every one of the thirty-nine legitimate instances has to be recognized as legitimate by somebody, and the two that matter are indistinguishable from them without the comparison.

And the dates cluster. Two additions thirteen minutes apart on the same day is the kind of pattern that turns two separate oddities into one event, which is what the timeline work in LX0.5 exists to support.

Clustering is doing real work there and it is not proof. Two things installed thirteen minutes apart could equally be one deployment, which is why the next move is a question to whoever owns the host rather than a conclusion.

One thing worth resisting when the baseline produces a short list, because the temptation is strong after a long search. Two entries that differ from the baseline are not two findings; they are two questions, and most of what a baseline surfaces turns out to be a legitimate change nobody recorded.

A deployment, a new monitoring agent or an engineer solving a problem at three in the morning all produce exactly this signature. Unusual is a starting point rather than a verdict, and the next step is asking whoever owns the host rather than writing a conclusion.

That is also the cheapest step available. A question to the platform team resolves most differences in minutes, and the ones that survive it are worth the full weight of everything else in this course.

The record is what turns a search into something somebody can rely on.

$ cat persistence-report.txt
  CHECKED    23 standard mechanisms against this host
  PRESENT    13
  NOT PRESENT 10, which is a fact about the host rather than a clean result
  SEARCHED   13 of 13
  BASELINE   web class, 39 instances, recorded 2026-01-14
  FOUND      2 not in the baseline, 13 minutes apart on 02 Mar
  MEANS      every mechanism present was searched. Not: no persistence exists.

Seven lines with coverage above findings. The third row is the one that stops ten unsearchable locations inflating the claim.

State how many mechanisms exist on this host. Thirteen of twenty-three here, and the figure differs per machine, which is what makes a generic checklist misleading rather than merely incomplete.

Mark absent separately from clean. A location that does not exist was not ruled out, and conflating the two inflates apparent coverage with searches that could not have found anything.

The wording matters as much as the column. Not present is a statement about the host; searched and clean is a statement about what was in it, and a report that says checked for both leaves a reader unable to tell which they have.

Report the coverage before the findings. Thirteen of thirteen searched is the claim that matters, and two found is a detail inside it rather than the headline.

Ordering it that way also protects the findings. Two entries presented after a coverage statement are two entries in a completed search; the same two presented alone invite a reader to assume they are the whole story.

Give the baseline date alongside. A comparison against a list recorded a year ago carries different weight from one recorded last month, and a reader has no way to judge the difference unless the date travels with the finding.

And say what a clean result means. Every mechanism present was searched and nothing unexplained was in any of them, which is a real statement and is narrower than the host has no persistence.

The subtraction in the previous section only works if somebody recorded what normal looks like first.

It is a single enumeration per host class. Running the census on a freshly built machine, before anything has touched it, produces the list every later comparison is made against.

The cost is genuinely small and the reason it rarely exists is ownership rather than effort. Nobody is responsible for knowing what a clean machine carries, so the question only gets asked during an incident when the answer would have been most useful beforehand.

$ ./baseline.sh --class web --from-build-image
  mechanisms present on a clean build     13
  instances across all of them            39
    /etc/cron.d                            6   log rotation, package cache, certs
    /etc/cron.daily                        9   distribution maintenance
    /etc/systemd/system                   11   service units and their timers
    /usr/lib/systemd/system                8   vendor units
    /etc/profile.d                         3   environment setup
    /root/.ssh/authorized_keys             2   two deployment keys
  recorded  2026-01-14, rebuild when the image changes

Thirty-nine instances on a machine nobody has attacked. Every one of them is legitimate and every one would appear in a naive search. Every one of them is the kind of artifact a persistence search looks for, which is why a search without a baseline returns thirty-nine things to investigate.

That number is the honest reason persistence hunting has a reputation for being tedious. The tedium comes entirely from the absence of a baseline, and it disappears the moment one exists.

The date at the bottom is the maintenance cost. A baseline drifts as the build image changes, so it is worth regenerating with the image rather than treating it as written once.

A stale baseline is worse than none, which is worth stating plainly. Differences against a year-old list are mostly ordinary drift, and an investigation that treats them as findings spends its time on deployments and patches.

Practice

There is a readiness exercise here worth an afternoon, and unlike most of them it produces something reusable. Run the census on one clean machine per host class and store the output, because that single artifact turns every future persistence search from an open-ended hunt into a subtraction.

The census itself is the easy half. Deciding what counts as a host class, and keeping the baselines current as build images change, is the part that needs somebody to own it, and a stale baseline is worse than none because it produces confident differences that are just drift.

It also answers a question nobody usually asks. Most teams have no idea how many scheduled tasks and units a normal machine of theirs carries, and the figure is routinely high enough to explain why ad hoc persistence searches feel hopeless.

Run the census on a host you are responsible for and count how many of the standard mechanisms actually exist. The number is usually well under the length of any published checklist, and knowing your own figure is what makes a completed search a claim rather than an aspiration.

One thing to carry forward. Five sections have now described what to read and where, and none of them has said anything about the order the work happens in or who is waiting for the answer.

The locations are now established. How an investigation actually proceeds from detection to a finished account, and who does what along the way, is LX0.7.