Reading width
Wide uses the full column for everything, text, diagrams, code, and exercises. Narrow keeps the standard reading width.
Text size
Scales the body text. Headings and code blocks keep their size.
In this section
0.9 Testing and Tuning
Introduction
A detection is only as good as the evidence that it works, and that evidence comes from running it against cases where the answer is known.
Every rule in this course has been judged that way, on the labeled month: attacks it should catch, look-alikes it should leave alone, and everything else it should stay quiet on. This lesson is about that method itself, and about what happens when a rule fails it and has to be tuned.
The figure is the whole argument in miniature. A rule for one of the most common ransomware preparation steps, written the way most people would write it first, catches two of six attacks in its test set and flags a scheduled backup job. Nothing about reading the query reveals either problem. Running it against labeled cases reveals both in seconds.
What a Test Set Holds
attacks, look-alikes and where each came fromA test set is a named list of labeled cases for one behavior. Each case is a set of records in the corpus marked as an attack or as a look-alike, with when it happened and what it was. The set for removing a machine's ability to recover from ransomware holds eight:
Test set recovery-all: Recovery removed: production and emulation
label keyTwo of the attacks come from production: the file server on 15 March, the vssadmin command lesson 0.3 followed through three sources, and the backup server the same night, where recovery was removed by four other means.
Four come from the detection lab: Atomic Red Team tests run on purpose on 9 March to produce the same technique through different routes. The two look-alikes are real administration on the same servers: the backup job removing its oldest shadow copy each week, and an administrator growing shadow storage.
Each kind of case answers a different question. Production attacks prove the rule catches what actually happened. Emulated attacks prove it catches what an attacker could have done instead. Look-alikes prove it does not catch the work the estate does every day. A set missing any of the three can be passed by a rule that will fail in production, and Module 15 builds sets with all three on purpose.
The cases are labeled once, by the people who know what they are, and stored with the corpus. That is what lets every rule in the course be scored the same way, by the platform rather than by the author's reading of their own query, and it is what the course's own exercises have been doing since lesson 0.1.
A label is a claim too, and it can be wrong. A case marked benign that was really an attack teaches every rule tested on it to stay quiet.
The course's labels carry a title for every case and, where a case could be read either way, a note on why it is labeled as it is, so that anyone who disagrees can check the reasoning against the records rather than take the label on trust.
Positive and Negative Cases on One Rule
three versions, scored case by caseThe first version of the rule looks for vssadmin deleting shadows, the command every write-up of ransomware mentions. The second adds every other route in the set. The third excludes the weekly prune. Each version scored on every case:
The first version catches the file server and the matching Atomic test, and misses every other route: WMI from the command line, WMI from PowerShell, boot recovery turned off with bcdedit, and the backup server's four methods. It also flags the weekly prune, because the prune uses the same command with different arguments. Its recall is a third, which is what a rule written from the obvious example usually scores.
The second version catches everything, and still flags the prune. The third adds one condition, that a vssadmin deletion is not of the oldest copy only, and passes: every attack, no look-alikes, nothing else in the month. That condition is visible in the look-alike's own command line, and only because the look-alike was in the set did anyone have to look for it.
Without the prune in the set, the second version would have shipped and alerted every week on the backup server, and the likeliest response to that noise is an exclusion for the whole server, which would have hidden the attack on it on 15 March.
Testing case by case also shows what a fix costs. The exclusion in the third version is narrow: it leaves alone only a deletion that names the oldest copy.
An attacker who deletes copies one at a time with that same option would pass it, and the right response is not to drop the exclusion but to add that edge case to the set, so the next version of the rule is tested against it. A test set grows every time someone thinks of a new way past the rule.
Emulation
what an Atomic Red Team run leaves behindReal attacks in a month of production data are few. Emulation adds more, by running published test procedures for a technique in a controlled environment and keeping the records they produce. The lab ran the Atomic Red Team tests for inhibiting system recovery on the morning of 9 March:
LabDeviceProcessEvents
| where Timestamp between (datetime(2026-03-09 10:40)
.. datetime(2026-03-09 10:50))
| project Timestamp, InitiatingProcessFileName, FileName,
ProcessCommandLine
13 process records from four tests in eight minutes. Each test ran from PowerShell, as the Atomic framework runs them, usually through a command shell; one test also ran its own cleanup, turning recovery back on. Those details matter when the records are used as test cases.
A rule that required cmd.exe as a parent would pass the Atomic runs and miss the backup server, where the attacker used PowerShell directly. A rule that alerted on any bcdedit recovery change would alert on the cleanup too.
Emulation broadens a test set; it does not replace production cases.
Atomic tests use the commands their authors chose, from a framework with its own parents and arguments, on a lab machine with nothing else happening. They prove a rule reaches routes the month's attackers did not take, and they are cheap to rerun after every change. They do not prove a rule is quiet in production, which is what the look-alikes are for.
Module 15 runs a test end to end and captures what it produced as a labeled case.
The lab records also show why emulated cases are kept apart from production ones. They live in their own tables, from a machine named as a lab host, so that a rule scored against the month is never credited with catching an attack that only happened in the lab, and so that a production rule never alerts on a test someone ran on purpose.
The test set joins the two only when a rule is being judged, and the board in section two shows each case's origin in its name.
Hits, Misses and Time to Detect
a rule measured on its own scheduleScoring a query against a test set says whether it can find a case. Running it on its schedule says whether it does, and how quickly. Lesson 0.6 showed a rule whose query was right and whose schedule missed the attack; every measure in the course is therefore taken on the rule's real schedule. R6, the session-replay rule, measured that way:
R6, session replay, measured on its own schedule
every hour, one hour backThe replay was caught, nothing else alerted, and the alert came 34 minutes after the stolen session was first used. That number is a property of the schedule as much as the query.
An hourly rule with a one-hour lookback waits for its window to close before it can see an event at the start of it, and then adds the platform's delay. Running it every fifteen minutes with an hour's lookback would cut the wait and, as lesson 0.6 showed, raise the same alert several times unless it is suppressed.
Time to detect matters because some attacks finish quickly. A replayed session reading mail does its damage in minutes; on the file server in this month, encryption began fourteen minutes after the shadow copies were deleted.
Module 15 measures the delay for every rule in the library, and Module 18 reports it across the program alongside recall and false positives, because a rule that catches everything an hour late can still be too slow.
The measure has a blind spot, named in the last line above. A rule is scored on the test sets someone wrote for it, and R6 passes the one it was written for. Section six runs it on the second.
The five lines in that record are the measures every rule in the library carries: what it caught, what it flagged that it should not have, how long it took, why, and what it was not tested against.
Together they are a rule's report card, and the last line is the one most often left out. A rule's measures describe its test sets as much as the rule, and a reader needs to know which sets they came from before trusting them.
Tracing a False Positive to Its Cause
a rule that was noisy because of its windowR4 alerts when someone makes an Azure change that no one else with the same job title has made. Lesson 0.6 showed it raising far more alerts than one attack could explain. Replayed on its schedule over the month, on its deployed lookback and on a longer one:
R4, Azure change no peer has made, on two lookbacks
scheduler replayThe query defines "what peers have done" as the changes in the same window it is checking:
let Titles = IdentityInfo
| summarize arg_max(TimeGenerated, JobTitle) by AccountUPN
| project Caller = AccountUPN, JobTitle;
let PeerDone = AzureActivity
| where TimeGenerated < ago(1h)
| join kind=inner Titles on Caller
| distinct JobTitle, OperationNameValue;
With a one-hour window, almost nothing has happened in the previous hour, so almost every change looks like one no peer has made. The rule is not wrong about the guest's Owner role, which it catches either way. It is wrong about everything else, and the cause is a single setting rather than the logic.
That is the usual shape of a false positive. The tempting fix is a higher threshold, or an exclusion for the accounts that alert most, and either would have quietened R4 while leaving the cause in place.
Tracing each false positive to the reason it matched is slower and finds fixes that make a rule better rather than quieter: here a lookback long enough to hold two weeks of peer history. Module 16 opens with this kind of trace.
The same trace also produces a test. The 10 changes the one-hour version flagged are now known to be ordinary peer activity, and keeping a few of them as labeled look-alikes means a future change that shortens the lookback again fails its test instead of filling a queue. A false positive traced and fixed is worth more as a test case than as a closed ticket.
The one other alert left on the longer lookback deserves the same trace rather than a shrug. It might be a genuinely new change by someone in that role, which is what the rule is for, or it might be a job title recorded differently in the directory, which is the identity problem from lesson 0.3 in a new place.
Either answer improves the rule or the data, and neither is found by raising a threshold.
A Fix That Trades One Failure for Another
the condition the attacker copiesR6 alerts when one session is used from a second address with a second client. On the test set it was written against, it is perfect. The month also has a replay where the attacker presented the victim's own user agent, and two look-alikes where a person's session moved between networks:
As deployed, R6 misses the copied-agent replay: the client is the same, so its condition fails. Drop the client condition and it catches the replay, and flags the traveller and the personal laptop, whose sessions also moved between addresses with the same browser. The condition that keeps ordinary travel quiet is exactly what the attacker copied.
This is the commonest result of testing a rule honestly: no version of it passes every set, and each fix moves the failure somewhere else. The way out is a different signal rather than a different threshold.
Module 12 looks for signals a copied user agent cannot carry across, and Module 16 tests the fix against both sets together. A rule that only ever met the first set would have shipped with the gap.
The practice at the end of this lesson starts from the address-only version and asks for the client condition, which passes the first set. Its last step runs the result on the second set as well, because seeing a fix fail elsewhere is the point.
It is tempting to read the board as a verdict on R6, and it is really a verdict on the test sets. Before the copied-agent case was labeled, R6 had a perfect record that described only the attack someone had thought of.
The second set did not make the rule worse; it made the rule's weakness visible. That is why the course keeps adding cases to sets long after a rule ships, and treats a new case that a rule fails as a success of the process rather than a failure of the rule.
An Exclusion That Opens a Bypass
a tuning change nobody testedOn 6 March an analyst tuned R6 in the portal to stop it alerting on people moving to their phones. The audit record holds the rule's query before and after the edit; the line that was added:
SentinelAudit
| where SentinelResourceName == "Session replayed from a new client"
and tostring(parse_json(ExtendedProperties).CallerName) == "p.sharma@ne.com"
| extend P = parse_json(ExtendedProperties)
| extend After = tostring(parse_json(tostring(P.UpdatedResourceState))
.properties.query)
| project TimeGenerated, Caller = tostring(P.CallerName),
Added = extract(@"\n(\s+and LaterAgent !has [^\n]+)", 1, After)
The change looks reasonable and passes R6's test sets, which contain no mobile clients at all. Read as a test case rather than a filter, it is a bypass:
Why this exclusion is a bypass
reviewA user agent is text the client sends. An attacker replaying a stolen cookie can send any user agent, and an attacker who knows or guesses that mobile clients are excluded will send one.
The exclusion keys the rule's silence on a field the adversary controls, which is the defining mistake of tuning and the most common one. The same mistake turns up as exclusions on process names, file names, task names and window titles, all of which the attacker writes.
Exclusions should key on facts the attacker cannot choose: a managed device's identity, an address range the organization owns, an application ID issued by the tenant. Every exclusion should also arrive with the edge case that tests it, here a replay presenting a mobile agent, so that the change and its weakness are reviewed together.
Module 16 builds that review, and Module 17's pipeline tests every rule change against labeled data before it deploys. An exclusion added in the portal, like this one, skips both.
The analyst was not careless. The alerts were real noise, the change stopped them, and nothing in the portal asked what the change would let through. That is the case for putting tuning through the same process as writing a rule: the person tuning sees the noise, and only a test shows the cost.
Lesson 0.6 showed the same change from the audit side, as a write the pipeline never reconciled; here it is the same event seen as a test the rule was never given.
What Modules 15 and 16 Build On It
testing, then tuningModule 15 tests: positive, negative and edge-case tests for one rule; building a labeled replay dataset; running an Atomic Red Team test and capturing what it produced; and measuring a rule's hits, false positives, misses and latency.
Module 16 tunes: tracing a false positive to its cause; thresholds and baselines that move with the estate; an exclusion that opens a bypass; and query cost, scan volume, joins and the scheduling budget every workspace has.
Cost is the one subject this lesson has not shown. Every rule reads data on every run, and a workspace has a finite budget for scheduled queries.
A rule that scans a month of sign-ins every fifteen minutes to compute a baseline can be correct and still be the reason other rules are throttled. Module 16 measures what each rule reads and moves expensive work, such as baselines and lookups, into summary tables computed once.
Both modules rest on one habit this lesson has shown five times: change a rule only with a test that shows what the change does. Recall, false positives, time to detect and cost are all measured, never estimated, and every change is scored against every set the rule has, not only the one it was written for.
The result is a library whose claims can be checked. Each rule carries the sets it was tested on, its scores on each, its measured delay and its cost, and each of those numbers comes from a run that anyone can repeat.
When a rule is tuned, the numbers change in a commit that shows why. That is what lets a detection team say what it can and cannot see, and be believed.
Practice
The exercise below is section six in reverse. Start from the address-only rule and make it pass the session-replay set.
- Run the starting query. Note which case the address-only rule flags and why it is not an attack.
- Find what separates them. Compare the user agents of the first and later requests in each session.
- Add the condition. Require a different client as well as a different address.
- Check the score. The replay caught, the traveller left alone.
- Run it on the second set. Score the same rule on session-replay-agent and write down what it now misses.
A rule is judged by the cases it is run against. Production attacks show it catches what happened, emulation shows it catches what could have, and look-alikes show it stays quiet on ordinary work. Measured on its real schedule it also has a time to detect.
Tuning is the same method in reverse: trace each failure to its cause, fix the cause, and test the fix against every set, because the fix for one case is often the failure in another.
The last lesson keeps rules in code, deploys them through a pipeline, and reviews the rules an AI assistant writes.