Reading width
Wide uses the full column for everything, text, diagrams, code, and exercises. Narrow keeps the standard reading width.
Text size
Scales the body text. Headings and code blocks keep their size.
In this section
Linux Privilege Escalation Investigation: Sudo Rules, SUID Binaries and a World-Writable Configuration
Introduction
An intruder rarely arrives as root. They arrive as whatever account the door gave them: the web server's account, a user whose password was guessed, a service account whose key was stolen. Everything an investigation finds after that point depends on whether and how that account became root, because root can read every file, change every log and install anything.
The routes are few, and each leaves a different kind of record. This lesson compares the first privileged command on three servers, then follows one escalation in detail on a server where the easy route was closed.
The routes from an ordinary account to root on Linux, and what each one leaves.
The figure lists the routes in roughly the order an intruder tries them, from the one that needs nothing clever to the one that needs a vulnerability. The first two leave the clearest records; the last can leave almost none.
Three First Privileged Commands
Two allowed by sudo, one refusedThe first command run as root, or attempted, on each of three servers:
WEB01 02:31:11 Accepted publickey for deploy from 203.0.113.99 port 45221 ssh2: RSA S
02:31:22 deploy : TTY=pts/1 ; PWD=/home/deploy ; USER=root ; COMMAND=/bin/bash
APP01 03:47:45 sudo_success deploy as root: /bin/bash
DB01 02:19:30 sudo_failure build-agent: /bin/bash (user NOT in sudoers)
On the web server from lessons 0.1 to 0.4, the intruder logged in as deploy with the key they had planted and ran sudo to a root shell eleven seconds later.
On the application server from lesson 0.4, deploy's guessed password was followed by sudo to /bin/bash. In both cases the route to root was simply a sudo rule: the account was allowed to become root, and the intruder used the permission the account already had.
That is the most common escalation on Linux, and it barely deserves the name. Nothing was exploited. It also leaves the clearest record of any route, as the end of this section shows.
The question it raises is about the sudo rule itself: should a deploy account be able to run an unrestricted root shell? Module 5's lesson on reading a sudo rule for what it grants is about exactly that, because a rule written for convenience is an escalation route for anyone who gets the account.
The third server is different. On SRV-NGE-BRS-DB01 in INC-NE-2026-0612, a RHEL database server, the intruder guessed the password of a contractor account, build-agent, and tried sudo about a minute later. sudo refused: the account was not in the sudoers file. The rest of this lesson is about what happened next, because an intruder refused by sudo does not usually give up.
Look at the form of each record, too. On the web server the sudo record is the classic text line: who ran it, from which terminal and directory, as which user, and the full command.
On the application server the parsed record carries the same facts in fields. Both name the command, which is what makes sudo the best-recorded route to root: a rule-based escalation writes its own evidence every time it is used. The routes later in this lesson do not.
Comparing the three also shows why the first privileged command is worth finding in every case. It marks the moment the intruder's reach changed from one account's files to the whole host, and every finding after it has to be read with that in mind.
Looking for Another Way
The enumeration after sudo said nobuild-agent's shell history from the moment of the refused sudo:
02:19:25 sudo -l
02:19:35 id
02:20:00 find / -perm -4000 -type f 2>/dev/null
02:21:00 ls -la /usr/local/bin/backup-tool
02:21:15 strings /usr/local/bin/backup-tool | grep -i config
02:22:00 cat /etc/backup-tool.conf
02:22:30 ls -la /etc/backup-tool.conf
02:23:00 echo 'PRE_BACKUP_CMD="/bin/bash -c cp /bin/bash /tmp/rootbash && chmod u
02:23:30 /usr/local/bin/backup-tool --run
02:24:00 ls -la /tmp/rootbash
02:24:30 /tmp/rootbash -p
02:24:45 whoami
sudo -l to see what, if anything, the account could run. id to confirm who it was. Then the search that defines this kind of escalation: find every file on the system with the SUID bit set. A SUID program runs with the privileges of its owner rather than of the person who starts it, so a SUID program owned by root runs as root for everyone.
Most are standard system tools that are written to be safe; the intruder is looking for one that is not. Locally installed software, in /usr/local or /opt, is where unsafe SUID programs are most often found, because it has not been through a distribution's review.
The search found /usr/local/bin/backup-tool. The intruder listed it, ran strings over it looking for the word config, read /etc/backup-tool.conf and listed its permissions. Four commands in ninety seconds, each one a reasonable next step from the last, and each one aimed at a single question: does this root program trust anything I can change?
Note the order of the first two commands as well. sudo -l came before the refused sudo in the history, at 02:19:25, and the authentication log recorded the refusal at 02:19:30. sudo -l lists what an account may run and is itself a sudo invocation, so on a host that logs it, a sudo -l from an account that has never used sudo before is an early sign that someone is exploring.
That pattern, enumerate, inspect, find the trust, is what Module 5 calls reading the escalation from what it left. The exploit itself is usually one line; the enumeration before it is longer and says more about how the intruder thought. Here it says they knew what to look for and found it in about three minutes.
It also says something about where the evidence came from. All of this is in a shell history, which the intruder cleared at the end of the session, as lesson 0.3 showed. The lines survived in this collection.
In a case where they did not, the enumeration would be visible only in an audit log, if one was kept and not emptied, and Module 5 is careful about what can be claimed when it is missing.
A SUID Program and a File Anyone Could Write
The baseline and the findingThe two files the escalation turned on, with the file it produced:
-rwsr-xr-x root 45056 2025-11-15 10:00 /usr/local/bin/backup-tool
-rw-rw-rw- root 892 2026-03-16 02:23 /etc/backup-tool.conf
-rwsr-xr-x root 1396736 2026-03-16 02:31 /usr/local/sbin/.health-monitor
backup-tool is owned by root, has the SUID bit set, and had not changed since 2025-11-15. Root's crontab runs it every night at 02:00 with --run. It is part of the server's baseline: installed deliberately, months before the intrusion.
On its own it is a risk, not a finding, and an investigator who reported every SUID program on a server as a finding would bury the one that mattered under dozens that did not.
Its configuration file is the finding. /etc/backup-tool.conf is owned by root but has mode -rw-rw-rw-, which means every account on the server can write to it. A root program that reads a configuration file every account can change will do whatever that file says. The intruder appended a line at 02:23:00, and the file's modification time, 02:23:00, records it.
The line set a command to run before each backup: copy /bin/bash to /tmp/rootbash and set the SUID bit on the copy. The intruder ran backup-tool by hand thirty seconds later, the program ran as root, the command ran as root, and the copy of bash became a SUID root shell that any account could start.
The escalation on SRV-NGE-BRS-DB01
Baseline plus one writeThe card separates the two halves the way Module 5 does. The baseline, a SUID root program that trusts a configuration file, had been on the server for months. The finding is the one write that used it. A report that calls backup-tool the vulnerability will lead to the wrong fix; the vulnerability is the permission on the file it reads.
The size of the configuration file is small, 892 bytes, and the change was one appended line. Nothing in the file's metadata says what the line was; only the history does.
On a host where the history was gone, the investigator would know the file changed at 02:23 and would have to read its current contents to see what was added, which is why a collection should keep the files an escalation turned on and not only their metadata.
The schedule makes it worse. Even if the intruder had not run backup-tool by hand, root's crontab would have run it at 02:00 the next night, with the injected command in its configuration. A world-writable file read by a root cron job is a delayed escalation that anyone on the server could have triggered, and probably the first thing to fix.
Two Records of One Minute
When the history and the authentication log disagreeThe escalation's last step appears in two records, and they do not tell the same story:
history 02:24:30 /tmp/rootbash -p
history 02:24:45 whoami
auth 02:24:45 auth_success service=su user=root from local
The history shows /tmp/rootbash -p at 02:24:30, the SUID shell started with the option that keeps its root privileges, and whoami fifteen seconds later. The authentication record shows a successful su to root at 02:24:45, from local. But there is no su in the history, and a SUID shell started this way does not go through su or PAM at all.
Both cannot be complete descriptions of how root was reached. Either the history is missing a su command, or the authentication record describes something the history does not, or one of them is wrong. The records here cannot settle which. What they do settle is that the account had root by 02:24:45, because the next command created an account with UID 0, which only root can do.
This is the situation Module 5's lesson on ordering by lineage rather than by clock exists for. Two clocks in two records give two moments, and neither clock is more trustworthy just because its record is more formal; what decides the order is which action made the next one possible.
The SUID shell was made at 02:23:30 and started at 02:24:30, and everything root did followed from it, whatever the authentication record says happened in between.
The general habit is to write the disagreement down rather than smooth it over. A finding that says root was obtained through a SUID shell made from an injected backup configuration, with an unexplained su success recorded at the same minute, is more useful and more honest than one that picks a story and drops the record that does not fit.
Module 11's lesson on the finding and what it does not support takes this further.
There are ordinary explanations worth testing before deciding either record is wrong. A collection tool that maps raw log lines into event types can misclassify a line; a clock difference between the shell and the logger can shift a record by seconds; and a PAM module can log a session change that is not a su at all.
Module 3's lesson on what a timestamp actually means, and Module 5's on ordering, are where those tests are made. Here the fixed point is the useradd: whatever the su record means, root existed by then.
What Root Did With It
Making root permanentFrom 02:26, the session's commands were all about keeping root:
02:26:00 useradd -o -u 0 -g 0 -M -d /root -s /bin/bash sysadm
02:26:30 echo "sysadm:Str0ngP@ss2026!" | chpasswd
02:27:00 mkdir -p /root/.ssh
02:27:30 echo "ssh-rsa AAAAB3NzaC1yc2E...attacker-key...== backdoor@workstation"
02:28:00 chmod 600 /root/.ssh/authorized_keys
02:30:00 cat > /etc/systemd/system/system-health.service << EOF
02:31:00 cp /tmp/rootbash /usr/local/sbin/.health-monitor
02:31:30 systemctl daemon-reload && systemctl enable system-health.service && sys
02:35:00 rm /tmp/rootbash
02:35:15 > /var/log/audit/audit.log
02:35:30 history -c
A new account, sysadm, created with UID 0 and GID 0, which makes it root under another name; the authentication log recorded the useradd at 02:26:10. A password set for it.
An SSH key added to root's own authorized_keys, with the comment backdoor@workstation. A systemd service, and the SUID shell copied to /usr/local/sbin/.health-monitor for the service to run. Then the original shell deleted, the audit log emptied and the history cleared.
Every one of those is a way back in that does not need the escalation again. The sysadm account logged in by key from the same outside address on the next two nights and was logged in when the collection ran.
The hidden SUID copy was running as PID 4521 with a connection to 45.33.32.156:4443. Lesson 0.6 reads persistence in detail; what matters here is that it all came after, and because of, the escalation.
The UID 0 account deserves a note of its own. useradd normally refuses to give a new account a UID that is already taken, and the -o option in the history overrides that check.
An account other than root with UID 0 is root in every respect the kernel cares about, under a name that looks like an administrator's. Listing every account with UID 0 is one of the first checks Module 6 makes on any host.
That is also Module 5's lesson on what root does to your evidence. Once an intruder has root, every record on the host is theirs to change, and this one used root to empty the audit log within minutes.
The records that survived did so because the intruder did not think of them, not because they could not reach them. A finding about anything that happened after 02:24 rests on those survivors.
It is also why an escalation finding is worth precision. The time root was obtained divides the host's records in two: before it, the intruder could change only what build-agent could; after it, anything. Every later record has to be weighed with that line in mind.
Notice how quickly it all happened. From the guessed password at 02:18:22 to the cleared history at 02:35:30 is about seventeen minutes, and the persistence was in place well before the end. That speed is typical of someone following a practiced routine, and it means an alert that takes an hour to reach a person arrives after the intruder has already made their return permanent.
What Module 5 Builds On It
Privilege escalationModule 5 reconstructs escalation chains from what they leave rather than from what an exploit claims.
It starts with what an escalation leaves and how to separate the baseline from the finding, orders events by lineage, and then reads each route: sudo rules, capabilities, the service manager, the kernel route that often cannot be confirmed, and credentials a foothold could read. It ends with what root does to the evidence and a workflow.
What Module 5 builds
Module 5The cases here touch most of those. The two sudo escalations are the sudo rule lesson; the SUID program is the baseline lesson and the file trust is the service manager lesson's close relative; the disagreement about su is the lineage lesson; and the emptied audit log is the lesson on what root does to the evidence.
The habits to carry into it are the ones this lesson used. Find the first privileged command. When sudo refuses, read the enumeration that follows. Separate what was on the host before from what the intruder changed.
Name the weakness precisely, a permission or a rule, not the program that suffered from it, so the fix lands in the right place. And when two records disagree about how root was reached, write both down and say what each supports.
The module's order follows an investigator's rather than an attacker's. It begins with what an escalation leaves and how to tell it from the baseline, because that is the first question on a real host, and only then reads the routes one by one. The last lesson, the workflow, puts them in the order that finds the most for the least effort on a host nobody has looked at yet.
Module 5's lab hosts are built with deliberately weak configurations of each kind, so that each route can be walked and its records read on a host where the ground truth is known.
The module is also careful about the routes that leave little. A kernel exploit can turn any account into root while writing nothing to any log; the evidence is usually a crash, a module, or a process running as root whose parent should not have had root.
Module 5's lesson on the kernel route is about what can and cannot be claimed in that case, which is often less than an investigator would like.
Practice
Decide which file made root possible prepared evidenceThe prepared evidence below is every file record collected from SRV-NGE-BRS-DB01. Decide which file was the weakness, which SUID file belongs to the baseline, and which file is the escalation's product kept for later. Each answer comes with its reasoning, so check it against the rows rather than only the choice.
Read the mode column for the letter s in the owner's execute position, which marks a SUID file, and for w in the last three positions, which marks a file every account can write. Then read the dates: anything unchanged for months is baseline until something shows otherwise.
Finally, read the owners and the paths together. A root-owned file in /etc that every account can write is a configuration mistake waiting to be used. A root-owned SUID file with a hidden name in a local directory, written in the middle of the night, is not a mistake at all.
Then, on a lab host, run the same find command the intruder ran and list every SUID file on a fresh install.
Keep that list; it is your baseline, and on a real host the files that are not on it are where an escalation investigation starts. Then find every file under /etc that every account can write; on a fresh install that list should be empty, and any entry on a real server deserves a question.
Three servers, three first privileged commands. Two intruders became root through sudo rules their accounts already had; the third, refused by sudo, found a root program that trusted a file anyone could write, and turned it into a SUID shell in under six minutes.
The finding is the permission on that file, and the time root was obtained divides every later record into what the intruder could have changed and what they could not.
The next lesson reads what all of these intruders left behind to come back, and the order in which to look for it.