Reading width
Wide uses the full column for everything, text, diagrams, code, and exercises. Narrow keeps the standard reading width.
Text size
Scales the body text. Headings and code blocks keep their size.
In this section
0.7 Endpoint Foundations: Windows and Linux
Introduction
Most intrusions that matter end on an endpoint: a laptop that ran an attachment, a server that accepted a password, a file server where shadow copies were deleted.
Endpoint detection is about what a machine records when a process starts, a file is written, a credential store is opened or a scheduled job is created, and about the small number of fields in those records that separate an attack from the same tool used for an ordinary reason.
This lesson reads both platforms on the course month. On Windows the records come from Defender for Endpoint, Sysmon and the Security log, as lesson 0.3 showed. On Linux they come from syslog: sshd, sudo and cron writing their own messages, and the kernel's audit subsystem recording program executions through auditd. The questions are the same on both, and the habits in each section carry across.
The Process Is the Unit
parents, children and the tree they makeOn 3 March a user on one laptop opened a file from their Downloads folder called Remittance_Advice_0303.hta. It was an HTML application, and what it did is recorded as three process starts in five seconds:
DeviceProcessEvents
| where DeviceName startswith "ne-taylor-lt"
| where Timestamp between (datetime(2026-03-03 10:20)
.. datetime(2026-03-03 10:25))
| project Timestamp, InitiatingProcessFileName, FileName,
ProcessCommandLine
The Splunk view reads the same Defender records, forwarded to the defender index, and returns the same three rows:
index=defender category="AdvancedHunting-DeviceProcessEvents"
DeviceName="ne-taylor-lt*"
earliest="03/03/2026:10:20:00" latest="03/03/2026:10:25:00"
| table _time InitiatingProcessFileName FileName ProcessCommandLine
Each row has the program that started (FileName) and the program that started it (InitiatingProcessFileName). Read together, the rows form a tree:
No single row in that tree is unusual on its own. mshta.exe is a signed Windows binary that runs HTML applications; certutil.exe is a signed certificate tool; rundll32.exe loads DLLs all day on every Windows machine.
What is unusual is the shape: an HTML application starting a certificate tool to download a file into a temp folder, then starting a DLL loader on that same file. Windows detection is mostly about shapes like this one, and a rule that reads one field at a time misses them.
The tree also says where to stand. A rule on the parent and child together, mshta starting certutil, is narrow and precise. A rule on the child alone has to decide from its own command line, which is where the next section goes. Module 10 builds both, and tests which one survives an attacker who changes the HTA for a script or a shortcut file.
The tree depends on a field every Windows source records but names differently: InitiatingProcessFileName in Defender, ParentImage in Sysmon, ParentProcessName in the Security log, as lesson 0.3 showed. It also depends on time.
Defender records the parent's own start time and process ID, so the tree can be rebuilt across hours, while a rule that joins parent and child by name alone can be fooled by two unrelated processes with the same name.
Module 10 uses the process identifiers whenever a rule has to walk more than one level of a tree, and Module 8 extends the walk across sources, from an endpoint event to the sign-in that came after it.
The Command Line Decides
one binary, three reasons to run itcertutil ran three times in the month, on three different laptops:
DeviceProcessEvents
| where FileName =~ "certutil.exe"
| project Timestamp, DeviceName, InitiatingProcessFileName,
ProcessCommandLine
The Splunk search over the forwarded Defender records returns the same three runs, on the same three laptops:
index=defender category="AdvancedHunting-DeviceProcessEvents" FileName="certutil.exe"
| table _time DeviceName InitiatingProcessFileName ProcessCommandLine
A rule on the binary's name alone alerts three times and is right once. The arguments say what each run was for:
Three certutil runs in the month, read by their arguments
one binary, three purposesThe download uses -urlcache, a cache-management option that also fetches any URL it is given. The analyst's revocation check uses -urlfetch, which also contains the letters "url" and also makes a network request, for a certificate's revocation list.
A rule that searches the command line for the substring "url" catches both and is wrong half the time; a rule that matches the whole argument -urlcache is right every time in this month. The practice at the end of this lesson is exactly that change.
Signed system binaries that can download, run or proxy code are called living-off-the-land binaries, and the pattern repeats across all of them: rundll32, regsvr32, mshta, msiexec and bitsadmin each have ordinary uses and abusive ones, separated by arguments and parents rather than by name.
The attacker's choice of a signed binary is deliberate: it is already trusted, already allowed by application control in most estates, and already present on every machine. That is why the rule has to be about what the binary was asked to do.
The parent adds a second line of defense. The download's certutil was started by mshta; both look-alikes were started by PowerShell in an interactive session. A rule matching -urlcache alone catches this attack and any attacker who uses the same option from a different launcher.
A rule that also requires an unusual parent is quieter and narrower, and an attacker who launches certutil from PowerShell walks past it. Choosing between them is a trade between missed attacks and noise, and Module 10 makes it on the evidence of the whole month rather than one case.
Persistence on Windows
a task that runs at every logonAn attacker who gets code running once wants it to run again after a reboot. On Windows the commonest way is a scheduled task. On 26 February one was created on a laptop:
DeviceProcessEvents
| where FileName =~ "schtasks.exe" and ProcessCommandLine has "/create"
| project Timestamp, DeviceName, InitiatingProcessFileName,
ProcessCommandLine
The Splunk version reads the same command-line field from the forwarded records and finds the same single task:
index=defender category="AdvancedHunting-DeviceProcessEvents"
FileName="schtasks.exe" ProcessCommandLine="*/create*"
| table _time DeviceName InitiatingProcessFileName ProcessCommandLine
It is the only task created with schtasks in the month. Split into its arguments, it says a great deal about intent:
Two parts of the command are camouflage. The task's name sits in the folder where Windows keeps its own tasks and borrows a product's name, and both can be changed at no cost. Two parts are hard to fake. The program runs from the user's Temp folder, a place ordinary software does not install to, and it runs with the highest privileges at every logon.
The look-alike in the month is Chrome's updater, which an installer registered on another laptop on 5 March: a task with a name built from a product and a GUID, running a program installed under Program Files. Defender for Endpoint recorded that one as a ScheduledTaskCreated event rather than a schtasks command, because installers register tasks through the Task Scheduler interface and leave no command line.
Module 10 builds the rule on the program's folder rather than the task's name, reads the registration event as well as the process, and tests it against the updater so that the first legitimate browser update does not raise an alert.
The two records together also give the rule a second chance. An attacker who creates a task through PowerShell's scheduled-task cmdlets leaves no schtasks command line, but the registration event still names the task and carries its full definition, including the program it runs. A rule that reads only the process misses that attacker; a rule that reads the registration catches both, and the program's folder is in both.
Access to the Credential Store
the same access, two very different reasonsLSASS is the Windows process that holds credentials in memory, and reading its memory is one of the most common steps in an intrusion. Sysmon records every process that opens LSASS, with the access rights it asked for:
Event
| where Source == "Microsoft-Windows-Sysmon" and EventID == 10
| where extract(@'Name="TargetImage">([^<]+)', 1, EventData) endswith "lsass.exe"
| extend SourcePath = extract(@'Name="SourceImage">([^<]+)', 1, EventData),
GrantedAccess = extract(@'Name="GrantedAccess">([^<]+)', 1, EventData)
| extend SourceImage = tostring(split(SourcePath, "\\")[-1])
| summarize Records = count(), Hosts = dcount(Computer)
by SourceImage, GrantedAccess
The antivirus engine opens LSASS hundreds of times across the estate with a narrow set of rights, enough to query it. Two other processes opened it once each with every right there is, enough to read its whole memory. Access rights separate routine checks from memory reads, but they do not separate those two:
DeviceProcessEvents
| where ProcessCommandLine has_any ("comsvcs", "WerFault")
and DeviceName has_any ("ne-sowerby-lt", "ne-bevan2-lt")
| project Timestamp, DeviceName, InitiatingProcessFileName,
ProcessCommandLine
On one laptop rundll32 called a function in comsvcs.dll named MiniDump, from a command prompt, writing LSASS's memory to a file in the Windows temp folder: credential theft. On another, Windows Error Reporting started from LSASS itself, because LSASS had crashed, and wrote a crash dump: the system doing its job. Both requested full access and both produced a dump file.
What separates them is, again, the parent and the arguments. A crash dump is started by the crashing process; a credential dump is started by a shell.
This is the pattern Module 10 tests most carefully, because a rule that alerts on every full-access read of LSASS fires on every crash, and a rule tuned to ignore WerFault can be bypassed by an attacker who names their tool WerFault.exe. A rule has to key on more than one field at once, and lesson 0.9 shows how the look-alike becomes a test.
The antivirus engine's hundreds of reads are worth noticing too. They are the reason a rule on any access to LSASS is unworkable, and the reason the access mask matters: a rule on full access drops them at once.
Every endpoint detection has a population like this one, ordinary software doing something that looks sensitive many times a day, and finding it before the rule ships is most of the work of making the rule usable.
What Linux Records About a Login
guessing, success and rootLinux records logins differently. There is no single process table; sshd, sudo and cron each write their own messages to syslog, in their own text formats. The public web server's night of 7 March reads like this:
Syslog
| where Computer == "WEB-NGE-MCR-02" and ProcessName in ("sshd", "sudo")
| where TimeGenerated between (datetime(2026-03-07 01:00)
.. datetime(2026-03-07 01:30))
| extend Kind = case(SyslogMessage startswith "Failed password", "failed",
SyslogMessage startswith "Accepted", "accepted",
SyslogMessage has "COMMAND=", "sudo", "other")
| where Kind != "other"
| summarize Records = count(), FirstSeen = min(TimeGenerated),
LastSeen = max(TimeGenerated) by Kind
The Splunk search parses the same sshd and sudo messages with the same three patterns, and counts the same way:
index=linux sourcetype=syslog host="WEB-NGE-MCR-02" (process=sshd OR process=sudo)
earliest="03/07/2026:01:00:00" latest="03/07/2026:01:30:00"
| eval kind=case(like(message, "Failed password%"), "failed",
like(message, "Accepted%"), "accepted",
like(message, "%COMMAND=%"), "sudo", true(), "other")
| where kind!="other"
| stats count as records min(_time) as first_seen max(_time) as last_seen
by kind
212 failed passwords from one address over twenty-one minutes, then one accepted password from the same address, then a root shell through sudo two minutes later:
Each of those on its own is weak evidence. Internet-facing SSH servers see failed passwords constantly. A successful login is what SSH is for. An administrator running sudo is routine. The look-alike in the month makes the point: an engineer on a database server failed four times from an internal address and then got in, a mistyped password with exactly the same shape on a much smaller scale.
What makes the web server's night an intrusion is the sequence and the scale together: a public address, hundreds of failures, a success at the end of them, and privilege straight afterwards.
Module 11 builds the rule as a sequence, with the failure threshold set from the estate's own history, and Module 7 writes the same correlation as a portable Sigma rule. Both depend on parsing text fields out of a message, the shape problem from lesson 0.3.
The text format has a cost the Windows records do not. The account, the address and the method are words inside a sentence that sshd writes, and a different SSH server or a different version can write the sentence differently.
The parsing that turns those words into fields is part of the rule, and it has to be tested on the estate's own messages. Module 11 pulls the fields out once, in a shared parser, so that every Linux rule reads the same names.
Persistence on Linux
a crontab for an account that never had oneTwo days after the SSH intrusion, someone on the same server installed a crontab for the web server's account. The audit subsystem recorded the command that did it, and cron recorded every run afterwards:
Syslog
| where Computer == "WEB-NGE-MCR-02"
| where TimeGenerated between (datetime(2026-03-09 03:10)
.. datetime(2026-03-09 03:21))
| where ProcessName in ("audisp-syslog", "crontab", "CRON")
| project TimeGenerated, ProcessName, SyslogMessage
The Splunk search reads the same three programs' records from the forwarded syslog for the same eleven minutes:
index=linux sourcetype=syslog host="WEB-NGE-MCR-02"
(process=audisp-syslog OR process=crontab OR process=CRON)
earliest="03/09/2026:03:10:00" latest="03/09/2026:03:21:00"
| table _time process message
The audit records show crontab executed as root from a logged-in user's session (auid 1001), installing a file from /tmp as the web account's schedule; cron then confirms the account's crontab was replaced.
Eight minutes later the job ran for the first time: download a script from 198.51.100.61 and pipe it into a shell. That is the address the macro cradle on a Windows laptop fetched from in lesson 0.2, so the same infrastructure served both platforms.
The job ran for the rest of the month:
Syslog
| where ProcessName == "CRON" and SyslogMessage startswith "(www-data) CMD"
| summarize Runs = count(), FirstSeen = min(TimeGenerated),
LastSeen = max(TimeGenerated) by Computer
Two servers, and that is the point. On the second web server the web account's job appeared on 9 March and ran 916 times before the month ended.
On the first, the same account has run a job since the month began: a curl request to the server's own address that warms its page cache. Both are the web account running curl from cron. One has a history; the other appeared overnight on a server that had been broken into two days earlier.
That is the condition Module 11 detects: a schedule appearing for an account on a host where that account never had one, built from each host's own history rather than a list of forbidden commands. A rule that alerted on the web account running curl from cron would alert every hour on the cache job, and a rule that ignored it would ignore the attack too.
The auditd records in this section are worth noticing for what they need. Linux records a program execution only if an audit rule asks it to; the web servers have a rule keyed exec that records every execution, and without it the crontab command would have left only cron's one-line message.
Module 11 ends by writing those audit rules, because on Linux the detection depends on configuration someone has to deploy first.
Turning the Lights Off
what an attacker removes, and what is leftAttackers who expect to be watched remove the watching. On the file server on 8 March an account added a Defender exclusion for a folder, downloaded a program into it, and cleared the Security log:
union
(DeviceProcessEvents
| where DeviceName startswith "srv-nge-mcr-fs01"
and Timestamp between (datetime(2026-03-08 02:10)
.. datetime(2026-03-08 02:25))
| project TimeGenerated = Timestamp, Record = "process",
Detail = ProcessCommandLine),
(SecurityEvent
| where EventID == 1102
| project TimeGenerated, Record = "Security 1102",
Detail = strcat("audit log cleared by ", Account))
| order by TimeGenerated asc
The order is the tell. The exclusion came first, so that the program written into the excluded folder would not be scanned. The log was cleared last, to remove the logon and process records that would show how the account got there.
Clearing the Security log is itself recorded: event 1102 is written into the freshly emptied log, naming the account that cleared it, and Defender for Endpoint recorded the wevtutil command independently, in its own process records, which the cleared log could not touch.
Two days later on the Linux web server, the same idea: auditctl -D deleted every audit rule. What the server sent afterwards shows exactly what was lost:
Syslog
| where Computer == "WEB-NGE-MCR-02"
and TimeGenerated > datetime(2026-03-10 02:39)
| summarize Records = count(), LastSeen = max(TimeGenerated)
by ProcessName
| order by LastSeen asc
The Splunk search over the forwarded syslog shows the same thing, one program at a time:
index=linux sourcetype=syslog host="WEB-NGE-MCR-02" earliest="03/10/2026:02:39:00"
| stats count as records max(_time) as last_seen by process
| sort last_seen
The audit stream's last record is the configuration change that deleted its own rules. sshd, sudo and cron kept writing, because they log through syslog and not through audit; the program-execution records stopped completely. A rule watching executions on that server went blind at 02:40, and its health records, as lesson 0.6 showed, would have called every run after that a success.
Both platforms have look-alikes. Intune pushes Defender exclusions for an accounting application through policy, and the backup server runs wevtutil every week to export its Security log to an archive.
Module 10 and Module 11 each detect the tampering itself, separate it from the managed change by who made it and how, and pair it with a check that the source is still delivering. An attacker who turns the lights off has made the one move that is guaranteed to be noticed, if anyone is looking for the dark.
What Modules 10 and 11 Build On It
two endpoint detection packsModule 10 builds a Windows pack: Office starting hidden encoded PowerShell; living-off-the-land binaries in rundll32, mshta and certutil; a logon task running from a temp folder; LSASS memory access and the crash dump that looks like it; Defender exclusions and cleared logs; and a final lesson that tests the whole pack against the month's attacks and look-alikes together.
Module 11 builds a Linux pack: a web server account downloading and running a script; SSH guessing that succeeds and goes to root; a crontab for an account that never had one; log tampering and an unusual outbound connection; and the audit rules every Linux detection depends on.
Each rule in both packs comes back to the same few fields this lesson read: the parent, the arguments, the account, the folder, and the sequence. A rule that reads the binary's name alone is the commonest weak endpoint detection, and every look-alike in this lesson is there to make that weakness visible before the rule reaches production.
Both packs are tested the same way, against the month's labeled attacks and look-alikes together, on the schedule the rules will run on. A Windows rule that catches its attack and fires on the crash dump fails its test, and so does a Linux rule that catches the crontab and fires on the cache job.
The test, rather than a reviewer's reading of the query, is what decides that a rule is ready, and the test set grows every time a new look-alike turns up in production.
Practice
The exercise below is section two's certutil rule. Make it catch the download and leave both people doing their jobs alone.
- Run the starting query. Note which certutil runs it returns and what each was for.
- Read the arguments. Find the one argument that only the download uses.
- Match it exactly. Use a whole-term match rather than a substring.
- Check the score. The attack caught, both look-alikes left alone.
- Note what is next. Write down the parent you would add to make the rule harder to evade.
Endpoint records are rich, and most of their fields describe ordinary work. A detection earns its place by reading the few that separate an attack from that work: the process that started this one, the arguments it was given, the folder it ran from, the account behind it, and what happened next.
On Linux the same questions are asked of text messages, and the answers depend on audit rules someone deployed first.
The next lesson leaves the endpoint for identity, cloud control planes and the network, where the records are about accounts and requests rather than processes.