0.3 Your First Assessment

Introduction

A detection program is usually inherited before it is designed.

The rules were written by people who have moved on, for threats somebody once cared about, against data that has changed since. Each rule looks reasonable when you read it, and reading it can't tell you the three things that matter: whether its alerts were attacks, whether the attacks it was written for set it off, and what it costs the analysts who triage it.

The way to find out is to run the rules the way production runs them and judge the results against cases whose answer is known. This lesson does that for five rules Northgate already runs, over the whole month, with every alert compared against the label key from lesson 0.2.

The chart is the queue as an analyst would see it, rows per rule. The sections judge each bar.

Rachel Okafor's request is short, and it is the right first question for any inherited rule set.

Before the team changes any inherited rule, she wants a page that says, for each one, what it alerted on in the last month, whether those alerts were worth an analyst's time and how much of it they took, and which attacks of the kind it claims to detect happened without it firing. The five rules in the queue:

Password spray from one address            SigninLogs
Defender exclusion written                 DeviceRegistryEvents
AdministratorAccess attached               AWSCloudTrail
Signed binaries that fetch and run code    DeviceProcessEvents
Role assigned in the subscription          AzureActivity

They cover sign-ins, Windows endpoints, AWS and Azure, which is a fair sample of an estate's rules, and each is the first version of a rule a later lesson rebuilds.

That makes them realistic rather than careless: each is the rule a competent engineer writes first, from the behavior's name and one table, before anyone has run it against the estate's own administration. The first, password spraying from one address, counts the accounts that fail from one source in an hour:

SigninLogs
| where ResultType != "0"
| summarize FailedAccounts = dcount(UserPrincipalName)
    by IPAddress, bin(TimeGenerated, 1h)
| where FailedAccounts >= 10

The same rule in Splunk counts failing users per source address in hourly bins, on the sign-in category:

index=azuread sourcetype="azure:monitor:aad" category=SignInLogs signature_id!=0
| bin _time span=1h
| stats dc(user) as failed_accounts by src_ip _time
| where failed_accounts >= 10

Read on its own, the rule says something precise: ten accounts failing from one address in an hour is a spray. Whether that's true at Northgate, with its eighty-five accounts and one office address, is a question about Northgate's data, and the label key can answer it, but only if the rule is run as it actually runs.

That distinction is the one this lesson rests on. A rule pasted into a console and run over the month returns every match at once, with hindsight.

The same rule deployed returns what each run could see in its own window, with records that had arrived by then, and an analyst sees each run's rows as an alert. The two can differ, and the second is the one the queue is made of.

All five run hourly over the previous hour, and Sentinel executes a scheduled rule about five minutes after its scheduled time.

The course's scheduler reproduces that: 707 runs per rule across the month, each reading the hour before it and only the records that had reached the workspace by the time it ran. A run whose query returns at least one row raises an alert carrying those rows. Run all five and count:

Password spray from one address             3 alerting runs,  3 rows
Defender exclusion written                  3 alerting runs, 14 rows
AdministratorAccess attached                1 alerting run ,  1 row
Signed binaries that fetch and run code     5 alerting runs,  7 rows
Role assigned in the subscription           2 alerting runs,  2 rows

14 alerts in a month, carrying 27 rows between them. On the face of it that's a quiet, well-behaved rule set, a few alerts a week, nothing an analyst would call noise.

The counts can't say more than that. An alert's row count measures how much a rule found in one run, not whether any of it mattered.

The exclusion rule's 14 rows came in three alerts and the spray rule's 3 in three, and nothing in either number says which was the attack. Every assessment that stops at volume ends the same way: the busiest rule gets tuned, and nobody finds out whether the quiet ones work.

How rows become alerts is a setting, too. Sentinel can raise one alert per run carrying all its rows, or one alert per row, and Module 9 sets it per rule. Here every alerting run counts once, which is how an analyst would meet the exclusion rule's thirteen policy writes: as two alerts with several rows each, which look like one problem twice.

Two details in how the runs were made matter later. The lookback equals the frequency, so every hour is read exactly once and nothing is read twice. And each run only sees records that had arrived, which is invisible here, because none of these five sources was late in a way that changed a result, and decisive in Module 6, where it isn't.

Each row in each alert can be matched against the label key: does it name an instance's entities within the instance's window? A row can belong to any case in the key, including cases its rule was never written for, so every row is checked against all 47 test sets together rather than against its rule's own set. The month's queue, judged:

rule                                      alerts  rows  attack  look-alike  other
Password spray from one address                3     3       2           1      0
Defender exclusion written                     3    14       1          13      0
AdministratorAccess attached                   1     1       1           0      0
Signed binaries that fetch and run code        5     7       5           2      0
Role assigned in the subscription              2     2       1           1      0
all five                                      14    27      10          17      0

Of 27 rows, 10 were attacks and 17 were look-alikes, and none matched nothing. The rule set's precision, counted by rows, is 37 percent, which means more than half of what an analyst opened was ordinary work. By rule it ranges from 100 percent for the AWS rule to 7 percent for the exclusion rule, and the average hides that spread entirely.

Counted by case instead of by row, the queue caught eight attacks and flagged five look-alikes, which is a fairer picture of what the rules can tell apart, because one look-alike repeated thirteen times is still one mistake. The cases behind the rows, attack and look-alike, rule by rule:

Password spray from one address
  attack      Password spray from inside the office, through its NAT address
  attack      Password spray from one external address
  look-alike  Old passwords after a forced change, from the office NAT address
Defender exclusion written
  attack      A Defender exclusion added by hand on a file server, then a binary ...
  look-alike  Intune pushes a Defender exclusion for the finance ERP client
AdministratorAccess attached
  attack      A leaked key creates an administrator user and stops CloudTrail
Signed binaries that fetch and run code
  attack      Word starts hidden encoded PowerShell that downloads a second stage
  attack      rundll32 and comsvcs.dll MiniDump against LSASS
  attack      An HTA runs certutil to fetch a DLL, and rundll32 runs it
  look-alike  An administrator hashes a downloaded installer with certutil
  look-alike  An analyst checks a certificate's revocation with certutil -urlfetch
Role assigned in the subscription
  attack      Owner on the subscription granted to a newly invited guest
  look-alike  An eligible Owner assignment activated through PIM for a change window

The picture by rule is very different from the picture by volume. The spray rule caught both sprays it saw and flagged one look-alike, the office's own address on the day everyone's old password stopped working. The signed-binary rule caught three attacks, two of them ones it wasn't written for. The exclusion rule caught its attack once and flagged the same administrative change thirteen times.

No row matched nothing, and that is worth pausing on, because in production most unmatched rows would be ordinary activity nobody labeled. This corpus labels its look-alikes on purpose, so here an unlabeled row would be a surprise worth investigating.

A real assessment gets the same separation from triage notes instead of a key: an alert an analyst closed as expected administration is a look-alike with a name, and counting those per rule is the same measurement.

The exclusion rule matches any write under the Defender exclusions key. Its 14 rows came in three runs:

run at 4 March 10:05    5 rows
run at 4 March 11:05    8 rows
run at 8 March 03:05    1 row

Group what it matched by the process that wrote the key, and whether the key is the policy branch an administrator's tooling writes:

DeviceRegistryEvents
| where RegistryKey has @"\Windows Defender\Exclusions\"
| summarize Writes = count(), Devices = dcount(DeviceName)
    by Writer = InitiatingProcessFileName, Policy = RegistryKey has @"\Policies\"

The Splunk form computes the same grouping from the registry events' fields:

index=defender category="AdvancedHunting-DeviceRegistryEvents"
    RegistryKey="*\\Windows Defender\\Exclusions\\*"
| eval policy=if(like(RegistryKey, "%\\Policies\\%"), "true", "false")
| stats count as writes dc(DeviceName) as devices by InitiatingProcessFileName policy

Two writers, one of them responsible for nearly everything:

writer                                                 writes  devices  policy key
Microsoft.Management.Services.IntuneWindowsAgent.exe       13       13  yes
MsMpEng.exe                                                 1        1  no

The Intune agent wrote the same exclusion to 13 laptops in the policy branch, the finance ERP client's folder pushed to the finance team, which is the look-alike the key names. One write came from MsMpEng.exe outside the policy branch on a file server, at 02:14 on a Sunday morning: a person adding an exclusion by hand, which is the attack.

Thirteen alerts were one administrative decision, delivered to thirteen machines. An analyst triaging them would have found the same answer each time, in the Intune policy and its change record, and closed each one as expected.

This is the commonest shape of noise in an inherited rule. The rule's logic is right about the behavior, an exclusion is worth knowing about, and wrong about the population, because it can't tell the management tool doing its job from a person doing something unusual.

Raising a threshold would hide the attack along with the noise, since the attack is a single write. The fix lesson 10.5 builds keys on who wrote the exclusion and where, which removes all thirteen and keeps the one.

The cost of a rule like this is easy to underrate, because it is paid in attention rather than time. Thirteen alerts in a month is not a burden, but each one an analyst closes as expected trains them to close the next, and the next one in this queue was the attack. A noisy rule's real cost is the attention it teaches the queue to withhold.

The signed-binary rule alerts on mshta, certutil or rundll32 starting, unless rundll32 is opening a Control Panel item. It returned 7 rows. Count them by device and binary:

DeviceProcessEvents
| where FileName in~ ("mshta.exe", "certutil.exe", "rundll32.exe")
    and not(ProcessCommandLine has "Control_RunDLL")
| summarize Starts = count() by DeviceName, FileName

The same count in Splunk, on Defender's process events:

index=defender category="AdvancedHunting-DeviceProcessEvents"
    (FileName="mshta.exe" OR FileName="certutil.exe" OR FileName="rundll32.exe")
    NOT ProcessCommandLine="*Control_RunDLL*"
| stats count as starts by DeviceName FileName

The starts, machine by machine, with the binary that each one ran:

device                              binary         starts
ne-collins-lt                       rundll32.exe        1
ne-greaves-lt                       certutil.exe        1
ne-sowerby-lt                       rundll32.exe        1
ne-taylor-lt                        certutil.exe        1
ne-taylor-lt                        mshta.exe           1
ne-taylor-lt                        rundll32.exe        1
ne-webb-lt                          certutil.exe        1

Its own test set is the HTA chain on ne-taylor-lt, mshta fetching a DLL with certutil and running it with rundll32, and it caught it.

Judged against every set, it also caught the second stage of the Word macro on ne-collins-lt and the LSASS dump on ne-sowerby-lt, because both passed through rundll32. And it flagged the two certutil look-alikes, an administrator hashing an installer and an analyst checking a certificate's revocation.

A rule that catches attacks it wasn't written for sounds like good news, and in part it is. It also means the rule's alerts can't be triaged by their title: an analyst opening "signed binaries that fetch and run code" might be looking at a credential dump, and the investigation guide written for the HTA chain won't help.

Breadth that catches by accident is breadth nobody tested, and an attacker who changes the binary leaves this rule with nothing. Catching the credential dump was luck about which binary the attacker chose, and the assessment should record it as luck rather than as coverage of credential access.

The other side of the ledger is the look-alikes. Both are certutil run by people whose job includes running it, on their own laptops, with arguments that fetch nothing and run nothing. The rule reads no argument except Control_RunDLL, so it can't tell them from an attack.

Lesson 10.2 keeps the breadth and requires the clause each binary's abuse needs, a URL for certutil, a user folder for rundll32, which removes the administrators and keeps the HTA chain, and leaves the credential dump to the rule lesson 10.4 builds for it.

The queue only shows what fired. The key also shows what didn't. Take the techniques the five rules caught at least once, and list every attack of those techniques in the month. Two produced no alert at all:

s01-distributed    T1110.003
  Password spray rotated through residential proxy addresses
s15-vpn-spray      T1110.003
  Password spray against the SSL VPN from six rotating addresses

The first is a password spray, the technique the spray rule claims. Count it the way the rule counts, failed accounts per address per hour, then summarize each hour:

SigninLogs
| where ResultType != "0"
    and TimeGenerated between (datetime(2026-03-07T01:00:00Z)
        .. datetime(2026-03-07T08:00:00Z))
| summarize FailedAccounts = dcount(UserPrincipalName)
    by IPAddress, bin(TimeGenerated, 1h)
| summarize Addresses = dcount(IPAddress), MostPerAddress = max(FailedAccounts),
    Accounts = sum(FailedAccounts) by Hour = TimeGenerated
| order by Hour asc

The Splunk form bins the failures by address and hour, then summarizes each hour:

index=azuread sourcetype="azure:monitor:aad" category=SignInLogs signature_id!=0
    earliest="03/07/2026:01:00:00" latest="03/07/2026:08:00:00"
| bin _time span=1h
| stats dc(user) as failed_accounts by src_ip _time
| stats dc(src_ip) as addresses max(failed_accounts) as most_per_address
    sum(failed_accounts) as accounts by _time
| sort _time

The spray, hour by hour, as the rule's own grouping would count it:

hour     addresses  most accounts from one  accounts failing
01:00             5                       2                 8
02:00             7                       2                12
03:00             4                       2                 5
04:00             7                       2                 9
05:00             3                       2                 5
06:00             4                       2                 5
07:00             2                       1                 2

In its busiest hour, 12 accounts failed, more than the rule's threshold, from 7 addresses, and no address failed more than two. The rule counts per address, so it never saw more than two at once. Nothing was wrong with any single run; the attacker chose a shape the rule's grouping can't see, and the queue gave no sign that anything had happened. Lesson 12.1 counts the tenant instead.

The second is the VPN spray of 10 March, 86 rows of failed VPN logons from six rotating addresses. The spray rule reads SigninLogs, and the VPN's authentication never reaches Entra ID: it is in Syslog, from the gateway.

The rule didn't miss it in its logic; it was never shown it. That is a source miss, and the fix isn't a better rule but a second one reading the gateway, which Module 4 parses into a shape a rule can count.

Across the whole month, the five rules caught eight of the 44 attacks that left rows in production tables. That isn't a verdict on them: most of the 44 use techniques none of the five claims, from beaconing to inbox rules, and whether anything at Northgate covers those is the coverage question Module 18 measures.

An assessment judges a rule against what it says it does, and the rule set against what the estate needs, and keeps the two apart.

The two misses are the two kinds every assessment finds. A logic miss is the right data read the wrong way, fixed by changing the rule. A source miss is the wrong data, fixed by changing what the rule reads, and a rule set's coverage claim is only honest when it says which sources each rule depends on.

Two rules came out well. The AdministratorAccess rule raised one alert, on the attack, and nothing else. The role-assignment rule raised two, the guest given Owner and a planned PIM activation.

A queue made only of rules like these would be easy to work, and that is exactly why they need the most careful reading: a rule that never alerts falsely might also be a rule that is easy to step around.

The AWS rule catches AdministratorAccess being attached to a user. The key holds a second attack with the same goal, s12-existing-user: "A leaked key adds a key and an allow-all inline policy to an existing user, and drops KMS from the trail", on 8 March. Everything that key did:

AWSCloudTrail
| where UserIdentityAccessKeyId == "AKIAV4R7NB2KJ3XW5ELQ"
| summarize Calls = count() by EventName
| order by Calls desc

The same calls in Splunk, selected by the access key that made them:

index=aws sourcetype="aws:cloudtrail" userIdentity_accessKeyId="AKIAV4R7NB2KJ3XW5ELQ"
| stats count as calls by eventName
| sort - calls

The attacker's calls with that key, the most frequent first:

GetCallerIdentity          1
ListUsers                  1
ListAccessKeys             1
CreateAccessKey            1
PutUserPolicy              1
PutEventSelectors          1

No AttachUserPolicy, so the rule never had anything to match. The attacker wrote an inline policy allowing everything with PutUserPolicy, which grants the same power without the managed policy's name, onto a user that already existed, so there was no CreateUser either.

The rule is precise about one procedure and blind to the goal, and lesson 12.6 rebuilds it to judge what a policy grants rather than what it is called. This course's Splunk view of CloudTrail carries the call and the caller but not the policy document, which is why the AWS rule has no Splunk form here and why 12.6 judges the policy in KQL.

The role-assignment rule's look-alike, s13-pim-activation, is the same operation done properly: an eligible assignment activated through Privileged Identity Management for a change window. The two writes look identical in AzureActivity; what differs is the path, an approved activation against a direct assignment to a new guest. Lesson 12.5 checks the path rather than the caller.

One more measure belongs in the assessment: how long each attack ran before its rule said anything. The scheduler records the run that first alerted on each case, and the key records when the case began:

rule    attack                      started        first alert    minutes
spray   s01-spray                   23 Feb 02:10   23 Feb 03:05        55
spray   s01-inside-spray            10 Mar 13:05   10 Mar 14:05        60
excl    s18-exclusion-path          8 Mar  02:14   8 Mar  03:05        51
aws     s12-iam-backdoor            13 Mar 04:51   13 Mar 05:05        13
lol     s17-hta-certutil-rundll32   3 Mar  10:12   3 Mar  11:05        52
owner   s13-owner-to-guest          12 Mar 21:02   12 Mar 22:05        63

Every case was alerted on within about an hour, and the reason is the schedule rather than the rules. An hourly rule over the previous hour sees an event at the first run after that hour closes, five minutes past the next hour, so the wait is between five and sixty-five minutes after the matching event, plus however long after the case began that event came.

The slowest, s13-owner-to-guest, took 63 minutes from the invitation that began it to the alert on the role assignment that followed. Time to detect is part of a rule's quality, and lesson 18.4 makes it a program measure.

An hour is acceptable for a spray and too slow for an exclusion added before a binary is dropped, so a rule's schedule belongs in its specification, decided from what the attacker does next.

Everything above fits on one page, which is what Rachel Okafor asked for. One verdict per rule, with the reason and the lesson that addresses it:

spray   keep, tune: 2 sprays caught, the office NAT look-alike flagged; blind to a
        spray spread across addresses (12.1) and to the VPN (4.5)
excl    rebuild: 13 of 14 rows are Intune's policy writes; key on the
        writer and the branch (10.5)
aws     keep, widen: quiet and correct, and blind to the same goal by an inline
        policy on an existing user (12.6)
lol     keep, narrow: 5 rows on three attacks, 2 on administrators' certutil (10.2)
owner   keep, check the path: the guest's Owner caught, the PIM activation
        flagged (12.5)

The page is short because every line rests on a measurement rather than a reading. It says which rules produce work and which produce detections, names the two attacks that went unseen and why, and puts the rebuild before the tuning, because the noisiest rule is the one whose noise is hiding its attack.

That ordering is the assessment's real recommendation, and it would have been the opposite if the queue had been judged by volume.

Two things are deliberately absent from the page. It doesn't rank the rules by alert count, because count measured the wrong thing in section 2, and it doesn't propose new rules for the techniques none of the five claims, because that is a coverage decision with its own evidence, not an assessment finding.

This is the method the course uses on every rule it builds, scaled down to one page. Modules 1 to 9 build the pieces: reading a rule, the threat behind it, the telemetry it needs, the data engineering that makes it readable, the specification and tests, the query, its portable form, correlation and deployment.

Modules 10 to 14 apply them to each attack surface, and Modules 15 to 18 run the result as a program.

Practice

The assessment in section 8 is this lesson's task. Then run the same method on a rule set of your own.

Next, lesson 0.4 takes one detection and writes it in every form the course uses: a specification, KQL, SPL, a portable Sigma rule, and the tests that prove it.