In this section

Linux Persistence Investigation: systemd Units, Cron, Timers, SSH Keys and UID 0 Accounts

Module 0

Introduction

Persistence is whatever an intruder leaves so that losing their current access does not end the intrusion. A reboot kills their processes, a password change locks their account, a closed session ends their shell; persistence is the thing that brings them back anyway.

On Linux it is almost always a file that something else reads: a unit file that systemd reads at boot, a crontab that cron reads every minute, an authorized_keys file that sshd reads at every login.

That is the idea Module 6 is built on, and this lesson applies it to four of the course's cases at once. Each host kept something; most kept more than one thing; and every planted entry sat beside legitimate entries of the same kind.

1. Bootsystemd units, init scripts: run when the host starts2. Schedulecron and systemd timers: run at a time or an interval3. Loginkeys, accounts, shell start-up files: run or admit when someone signs in4. Every executionthe dynamic linker, PAM: run inside other programs

Persistence classified by what triggers it, the way Module 6 enumerates it.

The figure classifies persistence by what triggers it rather than by where it lives. That is the order Module 6 enumerates in, because a list organized by trigger is a list of readers, and every reader is a place to look.

01

Services That Start at Boot

A unit file among the package units

On SRV-NGE-MCR-WEB01, the collection recorded every systemd unit that was running:

nginx.service        vendor  /lib/systemd/system/nginx.service        2025-08-04
mysql.service        vendor  /lib/systemd/system/mysql.service        2025-08-03
php8.1-fpm.service   vendor  /lib/systemd/system/php8.1-fpm.service   2025-07-30
ssh.service          vendor  /lib/systemd/system/ssh.service          2025-08-09
rsyslog.service      vendor  /lib/systemd/system/rsyslog.service      2025-06-09
cron.service         vendor  /lib/systemd/system/cron.service         2025-11-02
ufw.service          vendor  /lib/systemd/system/ufw.service          2025-11-14
syshealth.service    local   /etc/systemd/system/syshealth.service    2026-03-10

Seven of the eight are vendor units, installed by packages into /lib/systemd/system: the web server, the database, PHP, SSH, logging, cron and the firewall. One is local, in /etc/systemd/system, written on 10 March: syshealth.service, which runs /usr/local/bin/.syscheck with Restart=always, as lesson 0.1 found.

Where a unit file lives is a quick first filter. Packages install units under /lib/systemd/system or /usr/lib/systemd/system; administrators and intruders put them in /etc/systemd/system. Reading the local units first gives a short list on almost any server, and on this one the list has one entry. Module 6's boot lesson goes further, into the overrides and drop-in directories where a unit can be changed without its file moving at all.

The same shape appears on SRV-NGE-BRS-DB01 from lesson 0.5. A unit file, /etc/systemd/system/system-health.service, written at 02:30, runs the hidden SUID shell /usr/local/sbin/.health-monitor. The process list at collection had it running as PID 4521, started at 02:31, connected to 45.33.32.156. Two hosts, two intruders, the same choice: a local unit with a name that sounds like monitoring.

That naming is deliberate. A unit called syshealth or system-health is exactly what a busy administrator expects a monitoring agent to be called, and nobody stops a service that sounds like it keeps the server healthy. Module 6 treats the name as noise and reads what the unit runs instead.

The unit's settings say as much as its name. Restart=always on WEB01 means systemd starts the program again whenever it exits, so killing the process achieves nothing while the unit is enabled.

The unit on BRS-DB01 ran a copy of bash with the SUID bit set, so the service gave its starter nothing systemd had not already given it as root, but it kept a root process alive with an outbound connection. Reading a unit means reading what it runs, as whom, and what happens when it stops.

There is also a question every local unit raises: who wrote it? The unit file's owner is root on both hosts, which only says it was written by root or by something running as root. The time says more.

On WEB01 the unit file was written at 02:42, less than three minutes after the program it runs and eleven minutes after the intruder became root; it is part of the same session, and Module 6's dating lesson works through exactly that reasoning.

02

Schedules, in Two Systems

Cron entries and a systemd timer

Linux has two scheduling systems, and an investigation has to read both. Every cron entry the collections recorded on three servers:

WEB01    root       0 2 * * *     /usr/local/bin/backup.sh >/dev/null 2>&1
WEB01    root       */5 * * * *   /usr/local/bin/certbot-renew.sh
WEB01    root       30 3 * * 0    /usr/local/bin/log-rotate.sh
WEB01    root       */15 * * * *  /usr/local/bin/.syscheck >/dev/null 2>&1
APP01    root       0 2 * * 0     /usr/local/bin/backup.sh
APP01    root       */5 * * * *   /usr/local/bin/health-check.sh
APP01    appuser    0 3 * * *     /opt/app/scripts/log-rotate.sh
APP01    monitoring */1 * * * *   /usr/local/bin/node_exporter --web.listen-addres
APP01    deploy     @reboot       /tmp/.cache_helper -o stratum+tcp://pool.minexmr
BRS-DB01 root       0 2 * * *     /usr/local/bin/backup-tool --run
BRS-DB01 postgres   30 1 * * *    pg_dump production > /backups/production_daily.s
BRS-DB01 root       */5 * * * *   /usr/local/bin/health-check.sh

12 entries; 2 were planted. On WEB01, root's crontab gained a line at 02:40 on 10 March running .syscheck every fifteen minutes, beside a backup script, a certificate renewal and a weekly log rotation that had been there for months.

On APP01, deploy's crontab gained an @reboot line running the miner from /tmp with its pool address and wallet in the arguments, beside a backup, a health check, a log rotation and a metrics exporter.

The planted entries stand out on three counts, none of which needs to know what the program does. Each runs a file with a leading dot. Each runs it from a location packages do not use: /tmp, and /usr/local/bin under a hidden name. And the APP01 entry is @reboot, a schedule that means "every time the host starts", which turns a cron table into a boot mechanism.

The third server's three entries were all legitimate. That does not make its schedule clean: root's nightly entry runs backup-tool, which lesson 0.5 showed reads a configuration file every account can write. A legitimate schedule that trusts something writable is persistence waiting for someone to use it, and Module 6 reads targets as carefully as entries.

The second scheduling system is systemd's own timers. On SRV-NGE-DEV03 from lesson 0.3, the install of a malicious npm package created one:

[Unit]
Description=Node.js Telemetry Collection
[Timer]
OnBootSec=1min
OnUnitActiveSec=15min
[Install]
WantedBy=timers.target

[Unit]
Description=Node.js Telemetry Collection
[Service]
Type=oneshot
ExecStart=/usr/bin/node /usr/local/lib/node-telemetry.js

The timer fires one minute after boot and every fifteen minutes after the service last ran. The service it starts is a oneshot unit that runs a Node.js script from /usr/local/lib, and the journal recorded each start, as lesson 0.3 counted.

Nothing about this appears in any crontab, so an investigation that reads only cron would miss it entirely. Module 6's lesson on schedules is titled two systems, one estate for exactly that reason.

The timer and the cron line on WEB01 run at the same interval, fifteen minutes, which is a common choice for anything meant to call home: frequent enough to recover quickly when a connection drops, infrequent enough not to stand out. Reading a schedule's interval alongside its target is a quick way to tell a maintenance job from a beacon.

03

Keys and Accounts

Persistence that runs nothing at all

The quietest persistence on Linux needs no program. It is a way back in that the host's own authentication accepts. Every authorized key the collections recorded on two servers:

WEB01    deploy    /home/deploy/.ssh/authorized_keys      deploy@buildserver
WEB01    deploy    /home/deploy/.ssh/authorized_keys      backdoor@workstation
WEB01    sysadmin  /home/sysadmin/.ssh/authorized_keys    sysadmin@nge-jump01
WEB01    root      /root/.ssh/authorized_keys             automation@buildserver
BRS-DB01 dbadmin   /home/dbadmin/.ssh/authorized_keys     dbadmin@jumpbox
BRS-DB01 backup    /home/backup/.ssh/authorized_keys      backup@nge-backup
BRS-DB01 root      /root/.ssh/authorized_keys             backdoor@workstation

On WEB01, deploy's authorized_keys holds two keys: the expected one from the build server, and one with the comment backdoor@workstation, written at 02:28 on 10 March by the web server account. On BRS-DB01, root's own authorized_keys holds a key with the same comment, added at 02:27 on 16 March.

Different hosts, different intruders, and the same comment string on the key each planted. Both were added to accounts with the power to do damage: one to an account with sudo, one to root itself.

A key comment is whatever the person who made the key typed, so the shared text is not proof of a shared intruder. It is a lead worth noting, and a reminder that comments are often left at their defaults or set carelessly.

What the key line proves is simpler: whoever holds the matching private key can log in as that account, from anywhere sshd accepts connections, for as long as the line stays.

The second server also has an account that should not exist:

useradd  03-16 02:26:10  sysadm uid=0 gid=0 home=/root
login    03-16 02:28  sysadm pts/2 from 45.33.32.156  2026-03-16T03:00:00Z
login    03-17 03:15  sysadm pts/1 from 45.33.32.156  2026-03-17T03:30:00Z
login    03-18 02:45  sysadm pts/1 from 45.33.32.156  still logged in

sysadm was created with UID 0, so it is root under another name, and it logged in by key from the intruder's address on the night it was made and on the two nights after, the last session still open when the collection ran.

An account is the most durable persistence of all: it survives reboots, it survives the removal of every planted file, and it looks like an administrator's login in every log. Module 6's login lesson starts by listing every account with UID 0 and every key in every authorized_keys file, because those two lists are short and nearly everything on them should be recognizable.

The key on WEB01 also shows why the file's permissions matter. deploy's .ssh directory was recorded with mode drwxrwxrwx, writable by every account on the server. sshd's StrictModes setting, on by default, normally makes it refuse keys from a directory with permissions that loose, yet the authentication log records this key being accepted at 02:31.

Either StrictModes was off on this server or the recorded mode does not describe the directory as it was at 02:31; the records cannot say which, and the disagreement belongs in the finding. Module 6's login lesson covers how to check which keys sshd would actually accept.

04

Redundancy, and What It Means for Cleanup

Every host kept more than one way back

Put the four hosts side by side and a pattern appears:

What the four hosts kept

More than one way back

WEB01

a root cron line every 15 minutes, a service with Restart=always, a key for deploy

APP01

an @reboot line in deploy's crontab

BRS-DB01

a service running a SUID shell, a root key, a UID 0 account

DEV03

a timer every 15 minutes starting a oneshot service

Two of the four hosts have three mechanisms each, and on both the mechanisms cover different triggers: boot, schedule and login. Removing the service would leave the cron line; removing both would leave the key; and on BRS-DB01, removing the service and the root key would still leave sysadm, with a password the intruder set themselves.

Intruders plant redundant persistence precisely because they expect some of it to be found, and a running process is usually what a responder finds first, so the quiet key behind it is the mechanism most likely to survive the first cleanup.

That has a direct consequence for cleanup, which is where Module 6 ends. A host is not clean when one mechanism has been removed, or two. It is clean when every reader has been checked and every trigger accounted for, and the list in this lesson is a minimum, not a complete inventory.

Module 6's later lessons cover the readers these cases did not use: shell start-up files that run at every login, the dynamic linker that can load a library into every program, PAM modules that run inside authentication itself, and triggers that respond to events rather than paths.

It also changes how persistence is dated. Each mechanism has its own time: the unit file's, the crontab file's, the key file's, the account's creation in the log.

On WEB01 they fall within fourteen minutes of each other, which says one person in one session. A mechanism with a very different time is either from a different visit or from a different intruder, and Module 6's dating lesson is about telling which.

Redundancy also explains a common failure in response. An administrator who finds and kills a strange process, sees it come back, finds the unit and disables it, and then declares the host clean has removed one of three mechanisms.

The intruder logs in with the key the next night, and the incident starts again with less evidence than before. Module 6's workflow is designed so that cleanup starts only after the enumeration has finished.

The redundancy is also a gift to the investigator. Every extra mechanism is another record, written at another moment, and the times from all of them together date the intruder's session better than any one alone.

05

Finding Them, in Order

The enumeration Module 6 builds

The cases here were found because each collection recorded the right things. On a host nobody has looked at, persistence has to be enumerated, and the order matters because the list of places is long.

The order this module uses, in four steps:

An enumeration order for a host nobody has read

Readers, then time

1. Privileged readers

local units and timers, root and system crontabs, UID 0 accounts

2. Login readers

every authorized_keys, every account with a shell, shell start-up files

3. Deep readers

linker preload and configuration, PAM, kernel modules

4. Close the loop

sort every candidate by time and read the minutes around each find

Start with the readers that run with the most privilege and the least scrutiny: local systemd units and timers, root's crontab and the system cron directories, and every account with UID 0. Then the login readers: every authorized_keys file, every account with a shell, and the shell start-up files of each. Then the deeper readers: the dynamic linker's preload file and configuration, PAM's configuration, kernel modules.

At each step, compare against a baseline, because a list of everything is useless without a list of what should be there. A baseline can be another server built from the same image, the configuration management system's record of what it deployed, or the package manager's list of the files it installed; any of the three turns an inventory into a set of differences.

The cases also show the most efficient filter, which is to read where things live before reading what they are. Units in /etc/systemd/system, files with leading dots, programs in /tmp, /dev/shm or under hidden names in /usr/local: each filter cuts a long list to a short one, and the planted entries here all fell on the short side.

On WEB01 the local-unit filter alone cut eight units to one, and across three servers the hidden-file filter cut twelve cron entries to two. That is not because intruders are careless. It is because persistence has to be written somewhere an account can write, and those places are few.

The filters have limits worth stating. An intruder who writes a unit into /lib/systemd/system, names it after a real package and gives the file an old timestamp passes every filter here. Those are rarer, because they need more care, but they exist, and Module 6's later lessons cover the checks that catch them: comparing installed files against the package manager's own records of what it installed.

The times close the loop. Once one planted entry is found, sorting every candidate by modification time and reading the minutes around it usually finds the others. On WEB01 that sort, done in lesson 0.2, found the unit and the program together; the key and the cron line were within fourteen minutes.

06

What Module 6 Builds On It

Persistence

Module 6 finds every foothold on a host across accounts, keys, timers, services, shell initialization, library preloading, PAM and the kernel. It starts with what persistence has to do and how to enumerate the readers, then takes boot, login and the two scheduling systems, moves into the linker and on-demand triggers, and ends with dating an installation and a workflow.

What Module 6 builds

Module 6

6.1 and 6.2

what persistence has to do, and enumerating the readers

6.3 to 6.5

boot, login, and schedules in two systems

6.6 to 6.8

the linker, on-demand triggers, and triggers that are not a path

6.9 and 6.10

dating an installation, and the persistence workflow

The four hosts here are the first half of that in short, and between them they used every trigger in the figure except the last. Services are the boot lesson, cron and the timer are the schedules lesson, keys and the UID 0 account are the login lesson, and the redundancy is the reason for the workflow at the end.

The module's material is mostly lab hosts, because persistence is best learned by planting it: an investigator who has written a unit, a timer, a key and a preload entry by hand knows what each leaves, and where. The habits to carry into it are the ones this lesson used. Classify persistence by its trigger, and enumerate the readers.

Read where an entry lives before what it is called. Read both scheduling systems, cron and systemd timers, on every host. List every key and every UID 0 account. Expect more than one mechanism, and treat a host as clean only when every reader has been checked against a baseline and every find has been dated.

It also reaches beyond the mechanisms these cases used. None of the four hosts here touched the dynamic linker, PAM or a kernel module, and those are the mechanisms most likely to survive an ordinary cleanup because few people know to look. Module 6's later lessons cover each one, with the records it leaves and the check that finds it.

Module 6's lab hosts are built with every mechanism planted at once, so that the enumeration can be practiced against a known list and its gaps found before a real host finds them.

Do

Practice

Decide which entries were planted
prepared evidence

The prepared evidence below is every scheduled entry and unit from four servers, legitimate and planted together. Decide which an intruder added, which unit on WEB01 is not from a package, and whether the third server's schedule is clean. Each answer comes with its reasoning, so check it against the rows rather than only the choice.

Read the Runs column for paths and names before anything else. A named script in a standard location is probably what it says; a hidden file in /tmp or a local directory is the place to start.

Then read the When column. Fixed times at night are typical of backups and rotation; short intervals are typical of health checks and of beacons; @reboot is unusual for anything a package installs.

The column alone does not decide anything, but it ranks the list. Finally, read the User column against what each account does: a backup account running backups is expected, a login account like deploy running anything at boot is not.

Then, on a lab host you can rebuild, add one cron line, one local unit, one timer and one authorized key, and run the commands that list each: crontab -l for each user, systemctl list-unit-files, and a find for authorized_keys files. See which of your plants each command shows, and which it does not, and note that systemctl list-timers is the only one that shows the timer's next run.

Four hosts kept a way back, and two kept three each: services with names that sound like monitoring, cron lines and a timer running hidden files, keys with the same comment on two servers, and a second root account that logged in three nights running.

Every one sat beside legitimate entries of the same kind, and every one was found by reading where it lived, then confirmed by its time. A host is clean only when every reader has been checked.

The next lesson follows intruders off the host they entered, across the trust between servers that keys and credentials create.