In this section

0.9 Testing and Tuning

Module 0

Introduction

A detection is only as good as the evidence that it works, and that evidence comes from running it against cases where the answer is known.

Every rule in this course has been judged that way, on the labeled month: attacks it should catch, look-alikes it should leave alone, and everything else it should stay quiet on. This lesson is about that method itself, and about what happens when a rule fails it and has to be tuned.

Attacks caught by three versions of one rule vssadmin only 2 of 6 attacks, 1 look-alike 2 of 6 Every route 6 of 6 attacks, 1 look-alike 6 of 6 Every route, prune excluded 6 of 6 attacks, no look-alikes 6 of 6 The rule a reader would write first catches a third of the attacks. Only a test set shows that, and only a look-alike shows the cost of fixing it.

The figure is the whole argument in miniature. A rule for one of the most common ransomware preparation steps, written the way most people would write it first, catches two of six attacks in its test set and flags a scheduled backup job. Nothing about reading the query reveals either problem. Running it against labeled cases reveals both in seconds.

1

What a Test Set Holds

attacks, look-alikes and where each came from

A test set is a named list of labeled cases for one behavior. Each case is a set of records in the corpus marked as an attack or as a look-alike, with when it happened and what it was. The set for removing a machine's ability to recover from ransomware holds eight:

Test set recovery-all: Recovery removed: production and emulation

label key

attack, 03/15

All shadow copies deleted and recovery disabled on the file server

attack, 03/15

Backups and recovery removed on the backup server by four other means

attack, 03/09

Atomic Red Team T1490 test 1: vssadmin delete shadows /all on the lab

attack, 03/09

Atomic Red Team T1490 test 2: wmic shadowcopy delete on the lab

attack, 03/09

Atomic Red Team T1490 test 4: bcdedit turns off boot recovery, and its cleanup on the lab

attack, 03/09

Atomic Red Team T1490 test 5: PowerShell deletes shadow copies through WMI on the lab

look-alike, 02/15

The backup job prunes its oldest shadow copy each week

look-alike, 03/04

An administrator grows the file server's shadow storage

Two of the attacks come from production: the file server on 15 March, the vssadmin command lesson 0.3 followed through three sources, and the backup server the same night, where recovery was removed by four other means.

Four come from the detection lab: Atomic Red Team tests run on purpose on 9 March to produce the same technique through different routes. The two look-alikes are real administration on the same servers: the backup job removing its oldest shadow copy each week, and an administrator growing shadow storage.

Each kind of case answers a different question. Production attacks prove the rule catches what actually happened. Emulated attacks prove it catches what an attacker could have done instead. Look-alikes prove it does not catch the work the estate does every day. A set missing any of the three can be passed by a rule that will fail in production, and Module 15 builds sets with all three on purpose.

The cases are labeled once, by the people who know what they are, and stored with the corpus. That is what lets every rule in the course be scored the same way, by the platform rather than by the author's reading of their own query, and it is what the course's own exercises have been doing since lesson 0.1.

A label is a claim too, and it can be wrong. A case marked benign that was really an attack teaches every rule tested on it to stay quiet.

The course's labels carry a title for every case and, where a case could be read either way, a note on why it is labeled as it is, so that anyone who disagrees can check the reasoning against the records rather than take the label on trust.

2

Positive and Negative Cases on One Rule

three versions, scored case by case

The first version of the rule looks for vssadmin deleting shadows, the command every write-up of ransomware mentions. The second adds every other route in the set. The third excludes the weekly prune. Each version scored on every case:

Test runThree versions of a recovery-inhibition ruletest set recovery-all, scored on the labeled corpus
Candidate rule
File servermust alert
Backup servermust alert
Atomic 1must alert
Atomic 2must alert
Atomic 4must alert
Atomic 5must alert
Weekly prunemust stay quiet
Resizemust stay quiet
Other rowsallowed: 0
Result
vssadmin onlyvssadmin delete shadows
Alerted
Missed
Alerted
Missed
Missed
Missed
Fired
Quiet
0
Fails
Every route+ wmic, bcdedit and WMI from PowerShell
Alerted
Alerted
Alerted
Alerted
Alerted
Alerted
Fired
Quiet
0
Fails
Every route, prune excluded+ !has "/oldest"
Alerted
Alerted
Alerted
Alerted
Alerted
Alerted
Quiet
Quiet
0
Meets
Each cell is one case. The first version catches the commands a reader expects; the cases it misses are the ones an attacker chooses.

The first version catches the file server and the matching Atomic test, and misses every other route: WMI from the command line, WMI from PowerShell, boot recovery turned off with bcdedit, and the backup server's four methods. It also flags the weekly prune, because the prune uses the same command with different arguments. Its recall is a third, which is what a rule written from the obvious example usually scores.

The second version catches everything, and still flags the prune. The third adds one condition, that a vssadmin deletion is not of the oldest copy only, and passes: every attack, no look-alikes, nothing else in the month. That condition is visible in the look-alike's own command line, and only because the look-alike was in the set did anyone have to look for it.

Without the prune in the set, the second version would have shipped and alerted every week on the backup server, and the likeliest response to that noise is an exclusion for the whole server, which would have hidden the attack on it on 15 March.

Testing case by case also shows what a fix costs. The exclusion in the third version is narrow: it leaves alone only a deletion that names the oldest copy.

An attacker who deletes copies one at a time with that same option would pass it, and the right response is not to drop the exclusion but to add that edge case to the set, so the next version of the rule is tested against it. A test set grows every time someone thinks of a new way past the rule.

3

Emulation

what an Atomic Red Team run leaves behind

Real attacks in a month of production data are few. Emulation adds more, by running published test procedures for a technique in a controlled environment and keeping the records they produce. The lab ran the Atomic Red Team tests for inhibiting system recovery on the morning of 9 March:

LabDeviceProcessEvents
| where Timestamp between (datetime(2026-03-09 10:40)
    .. datetime(2026-03-09 10:50))
| project Timestamp, InitiatingProcessFileName, FileName,
    ProcessCommandLine

13 process records from four tests in eight minutes. Each test ran from PowerShell, as the Atomic framework runs them, usually through a command shell; one test also ran its own cleanup, turning recovery back on. Those details matter when the records are used as test cases.

A rule that required cmd.exe as a parent would pass the Atomic runs and miss the backup server, where the attacker used PowerShell directly. A rule that alerted on any bcdedit recovery change would alert on the cleanup too.

Emulation broadens a test set; it does not replace production cases.

Atomic tests use the commands their authors chose, from a framework with its own parents and arguments, on a lab machine with nothing else happening. They prove a rule reaches routes the month's attackers did not take, and they are cheap to rerun after every change. They do not prove a rule is quiet in production, which is what the look-alikes are for.

Module 15 runs a test end to end and captures what it produced as a labeled case.

The lab records also show why emulated cases are kept apart from production ones. They live in their own tables, from a machine named as a lab host, so that a rule scored against the month is never credited with catching an attack that only happened in the lab, and so that a production rule never alerts on a test someone ran on purpose.

The test set joins the two only when a rule is being judged, and the board in section two shows each case's origin in its name.

4

Hits, Misses and Time to Detect

a rule measured on its own schedule

Scoring a query against a test set says whether it can find a case. Running it on its schedule says whether it does, and how quickly. Lesson 0.6 showed a rule whose query was right and whose schedule missed the attack; every measure in the course is therefore taken on the rule's real schedule. R6, the session-replay rule, measured that way:

R6, session replay, measured on its own schedule

every hour, one hour back

Hits

100% of the set's attacks: the replay of 25 February

False positives

0 look-alikes flagged, 0 unlabeled rows

Time to detect

first use from the second client 10:31, first alert 11:05: 34 minutes

Where the time goes

up to an hour waiting for the window to close, five minutes delay, and the run

What it does not show

the copied-agent replay in the second test set, section six

The replay was caught, nothing else alerted, and the alert came 34 minutes after the stolen session was first used. That number is a property of the schedule as much as the query.

An hourly rule with a one-hour lookback waits for its window to close before it can see an event at the start of it, and then adds the platform's delay. Running it every fifteen minutes with an hour's lookback would cut the wait and, as lesson 0.6 showed, raise the same alert several times unless it is suppressed.

Time to detect matters because some attacks finish quickly. A replayed session reading mail does its damage in minutes; on the file server in this month, encryption began fourteen minutes after the shadow copies were deleted.

Module 15 measures the delay for every rule in the library, and Module 18 reports it across the program alongside recall and false positives, because a rule that catches everything an hour late can still be too slow.

The measure has a blind spot, named in the last line above. A rule is scored on the test sets someone wrote for it, and R6 passes the one it was written for. Section six runs it on the second.

The five lines in that record are the measures every rule in the library carries: what it caught, what it flagged that it should not have, how long it took, why, and what it was not tested against.

Together they are a rule's report card, and the last line is the one most often left out. A rule's measures describe its test sets as much as the rule, and a reader needs to know which sets they came from before trusting them.

5

Tracing a False Positive to Its Cause

a rule that was noisy because of its window

R4 alerts when someone makes an Azure change that no one else with the same job title has made. Lesson 0.6 showed it raising far more alerts than one attack could explain. Replayed on its schedule over the month, on its deployed lookback and on a longer one:

R4, Azure change no peer has made, on two lookbacks

scheduler replay

Looks back one hour

11 alerting runs: the guest's Owner role, and 10 changes that are not attacks

Looks back fourteen days

2 alerting runs: the guest's Owner role, and 1 other

The cause

"what peers have done" is read from the same window as the change

The fix

a lookback long enough to hold the peers' history, not a higher threshold

The query defines "what peers have done" as the changes in the same window it is checking:

let Titles = IdentityInfo
| summarize arg_max(TimeGenerated, JobTitle) by AccountUPN
| project Caller = AccountUPN, JobTitle;
let PeerDone = AzureActivity
| where TimeGenerated < ago(1h)
| join kind=inner Titles on Caller
| distinct JobTitle, OperationNameValue;

With a one-hour window, almost nothing has happened in the previous hour, so almost every change looks like one no peer has made. The rule is not wrong about the guest's Owner role, which it catches either way. It is wrong about everything else, and the cause is a single setting rather than the logic.

That is the usual shape of a false positive. The tempting fix is a higher threshold, or an exclusion for the accounts that alert most, and either would have quietened R4 while leaving the cause in place.

Tracing each false positive to the reason it matched is slower and finds fixes that make a rule better rather than quieter: here a lookback long enough to hold two weeks of peer history. Module 16 opens with this kind of trace.

The same trace also produces a test. The 10 changes the one-hour version flagged are now known to be ordinary peer activity, and keeping a few of them as labeled look-alikes means a future change that shortens the lookback again fails its test instead of filling a queue. A false positive traced and fixed is worth more as a test case than as a closed ticket.

The one other alert left on the longer lookback deserves the same trace rather than a shrug. It might be a genuinely new change by someone in that role, which is what the rule is for, or it might be a job title recorded differently in the directory, which is the identity problem from lesson 0.3 in a new place.

Either answer improves the rule or the data, and neither is found by raising a threshold.

6

A Fix That Trades One Failure for Another

the condition the attacker copies

R6 alerts when one session is used from a second address with a second client. On the test set it was written against, it is perfect. The month also has a replay where the attacker presented the victim's own user agent, and two look-alikes where a person's session moved between networks:

Test runR6 against a replay that copied the victim's browsertest set session-replay-agent
Candidate rule
Copied agentmust alert
Personal laptopmust stay quiet
Travelmust stay quiet
Other rowsallowed: 0
Result
As deployednew address and new client
Missed
Quiet
Quiet
1
Fails
Address onlynew address, client ignored
Alerted
Fired
Fired
1
Fails
Neither version passes. The condition that keeps travellers quiet is the condition the copied agent defeats.

As deployed, R6 misses the copied-agent replay: the client is the same, so its condition fails. Drop the client condition and it catches the replay, and flags the traveller and the personal laptop, whose sessions also moved between addresses with the same browser. The condition that keeps ordinary travel quiet is exactly what the attacker copied.

This is the commonest result of testing a rule honestly: no version of it passes every set, and each fix moves the failure somewhere else. The way out is a different signal rather than a different threshold.

Module 12 looks for signals a copied user agent cannot carry across, and Module 16 tests the fix against both sets together. A rule that only ever met the first set would have shipped with the gap.

The practice at the end of this lesson starts from the address-only version and asks for the client condition, which passes the first set. Its last step runs the result on the second set as well, because seeing a fix fail elsewhere is the point.

It is tempting to read the board as a verdict on R6, and it is really a verdict on the test sets. Before the copied-agent case was labeled, R6 had a perfect record that described only the attack someone had thought of.

The second set did not make the rule worse; it made the rule's weakness visible. That is why the course keeps adding cases to sets long after a rule ships, and treats a new case that a rule fails as a success of the process rather than a failure of the rule.

7

An Exclusion That Opens a Bypass

a tuning change nobody tested

On 6 March an analyst tuned R6 in the portal to stop it alerting on people moving to their phones. The audit record holds the rule's query before and after the edit; the line that was added:

SentinelAudit
| where SentinelResourceName == "Session replayed from a new client"
    and tostring(parse_json(ExtendedProperties).CallerName) == "p.sharma@ne.com"
| extend P = parse_json(ExtendedProperties)
| extend After = tostring(parse_json(tostring(P.UpdatedResourceState))
    .properties.query)
| project TimeGenerated, Caller = tostring(P.CallerName),
    Added = extract(@"\n(\s+and LaterAgent !has [^\n]+)", 1, After)

The change looks reasonable and passes R6's test sets, which contain no mobile clients at all. Read as a test case rather than a filter, it is a bypass:

Why this exclusion is a bypass

review

What it says

and LaterAgent !has "Mobile"

What it meant

stop alerting when a user moves to a phone

What it does

stop alerting on any later client that calls itself mobile

Who controls that

the client: the user agent is text the browser or tool sends

What a test needs

an edge case: a replay presenting a mobile user agent

A user agent is text the client sends. An attacker replaying a stolen cookie can send any user agent, and an attacker who knows or guesses that mobile clients are excluded will send one.

The exclusion keys the rule's silence on a field the adversary controls, which is the defining mistake of tuning and the most common one. The same mistake turns up as exclusions on process names, file names, task names and window titles, all of which the attacker writes.

Exclusions should key on facts the attacker cannot choose: a managed device's identity, an address range the organization owns, an application ID issued by the tenant. Every exclusion should also arrive with the edge case that tests it, here a replay presenting a mobile agent, so that the change and its weakness are reviewed together.

Module 16 builds that review, and Module 17's pipeline tests every rule change against labeled data before it deploys. An exclusion added in the portal, like this one, skips both.

The analyst was not careless. The alerts were real noise, the change stopped them, and nothing in the portal asked what the change would let through. That is the case for putting tuning through the same process as writing a rule: the person tuning sees the noise, and only a test shows the cost.

Lesson 0.6 showed the same change from the audit side, as a write the pipeline never reconciled; here it is the same event seen as a test the rule was never given.

8

What Modules 15 and 16 Build On It

testing, then tuning

Module 15 tests: positive, negative and edge-case tests for one rule; building a labeled replay dataset; running an Atomic Red Team test and capturing what it produced; and measuring a rule's hits, false positives, misses and latency.

Module 16 tunes: tracing a false positive to its cause; thresholds and baselines that move with the estate; an exclusion that opens a bypass; and query cost, scan volume, joins and the scheduling budget every workspace has.

Cost is the one subject this lesson has not shown. Every rule reads data on every run, and a workspace has a finite budget for scheduled queries.

A rule that scans a month of sign-ins every fifteen minutes to compute a baseline can be correct and still be the reason other rules are throttled. Module 16 measures what each rule reads and moves expensive work, such as baselines and lookups, into summary tables computed once.

Both modules rest on one habit this lesson has shown five times: change a rule only with a test that shows what the change does. Recall, false positives, time to detect and cost are all measured, never estimated, and every change is scored against every set the rule has, not only the one it was written for.

The result is a library whose claims can be checked. Each rule carries the sets it was tested on, its scores on each, its measured delay and its cost, and each of those numbers comes from a run that anyone can repeat.

When a rule is tuned, the numbers change in a commit that shows why. That is what lets a detection team say what it can and cannot see, and be believed.

Practice

The exercise below is section six in reverse. Start from the address-only rule and make it pass the session-replay set.

Do this Fix one failure, then look for the next
  1. Run the starting query. Note which case the address-only rule flags and why it is not an attack.
  2. Find what separates them. Compare the user agents of the first and later requests in each session.
  3. Add the condition. Require a different client as well as a different address.
  4. Check the score. The replay caught, the traveller left alone.
  5. Run it on the second set. Score the same rule on session-replay-agent and write down what it now misses.
What you should end up with: a session replay rule that passes one test set, and a note of the case it fails in the other.

A rule is judged by the cases it is run against. Production attacks show it catches what happened, emulation shows it catches what could have, and look-alikes show it stays quiet on ordinary work. Measured on its real schedule it also has a time to detect.

Tuning is the same method in reverse: trace each failure to its cause, fix the cause, and test the fix against every set, because the fix for one case is often the failure in another.

The last lesson keeps rules in code, deploys them through a pipeline, and reviews the rules an AI assistant writes.

Resources