Reading width
Wide uses the full column for everything, text, diagrams, code, and exercises. Narrow keeps the standard reading width.
Text size
Scales the body text. Headings and code blocks keep their size.
In this section
0.10 Rules as Code and Rules Written by AI
Introduction
Every earlier lesson ended with a rule that worked: tested on labeled cases, measured on its schedule, tuned without opening a bypass. This lesson is about keeping it that way. A library of rules changes every week, edited by different people for different reasons, and increasingly drafted by AI assistants.
Without a process around those changes, each one is a chance to undo the work that made the rule good, and lessons 0.6 and 0.9 both showed a change like that.
The answer the course builds is detection as code. Every rule lives as a file in a repository; every change arrives as a pull request; automated checks read the file, run the query against labeled data and replay it on its schedule; a reviewer approves; a pipeline deploys; and a detection catches any change made anywhere else.
The same process is what makes an AI-drafted rule safe to use, because it judges the rule by what it does rather than by who wrote it.
A Rule Is a File
everything a rule needs, in one placeNorthgate's rules live in a repository, one folder per rule, each with a rule file. The session-replay rule from lessons 0.6 and 0.9, at rules/identity/session-replayed-new-client/rule.yml, with its query and its deployment settings folded:
title: Session Replayed From a New Client
id: 8c355665-4ada-465d-8ea9-9a7439a7f5b6
status: stable
description: Detects one sign-in session used from a second address and a second
client within eight hours.
references:
- https://learn.microsoft.com/entra/identity/monitoring-health/concept-sign-ins
author: Marcus Webb
owner: detection-engineering
date: '2026-02-12'
tags:
- attack.lateral-movement
- attack.t1550.004
query_language: kql
tables:
- SigninLogs
- AADNonInteractiveUserSignInLogs
fields:
- TimeGenerated
- ResultType
- SessionId
- UserPrincipalName
- IPAddress
- UserAgent
query: |
...
falsepositives:
- A phone moving from office Wi-Fi to a mobile network, with a browser that changes
its agent string on update
level: high
test_set: session-replay
preconditions: Interactive and non-interactive sign-in logs, with SessionId
limitations: A replay that copies the victim's user agent is not a new client
and is missed (lesson 12.2)
sentinel:
...
Most of the file is not the query. It names the rule, gives it a stable ID that survives renames, says who owns it and when it was written, and maps it to ATT&CK.
It lists the tables and fields the query depends on, which is how a missing column or a renamed field is caught before deployment. It names the test set the rule is judged on, states the preconditions it needs from the estate, and records the false positives and limitations someone already knows about.
The limitations line is worth reading twice. It says the rule misses a replay that copies the victim's user agent: the failure lesson 0.9 found on the second test set, written into the rule itself so that whoever relies on it knows. A file that admits what its rule cannot do is more useful than one that does not, and the course's standard requires the line.
The header follows Sigma's format, the portable rule format from Module 7, with the course's own fields added. A rule written in Sigma converts to KQL and SPL; a rule written in KQL, like this one, still uses the same header, so that every rule in the library can be read, checked and searched the same way regardless of language.
Keeping the rule in a file also gives it a history. Every change is a commit with an author, a date and a reason, and the version that ran on any given day can be recovered exactly.
When lesson 0.6 asked who changed a rule and what it said before, the audit log could answer only the first half; the repository answers both, for every rule, back to the day it was written.
What Every Change Runs
checks a machine can do before a person looksA change to a rule is a pull request, and every pull request starts a workflow before a reviewer sees it. The validation steps from Northgate's repository:
steps:
# actions/checkout v4.2.2
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
with:
fetch-depth: 0
# actions/setup-python v5.6.0
- uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065
with:
python-version: "3.12"
- run: pip install -r tools/requirements.txt
- name: Lint the workflows
run: actionlint
- name: Validate the rules
run: python tools/validate.py --changed "origin/${{ github.base_ref }}"
The workflow lints its own definition, validates every changed rule against the repository's schema, and checks the Sigma rules with Sigma's own tool. The actions it uses are pinned to exact commits rather than version tags, so that a compromised tag cannot change what runs; the comments name the version each commit is.
The repository also has a quality checker that reads a rule file against the course's rule standard, in five groups. On the session-replay rule:
rules/rule/rule.yml: PASS
ok Required metadata
ok ATT&CK mapping
ok Fields and data
ok Test coverage
ok Documentation
Every check passes. The checker confirms the metadata is complete, the ATT&CK tags are real techniques with tactics that match, the fields exist in the tables named, a test set is named, and the documentation lines are filled in. None of that says the rule works; it says the rule is complete enough to be tested.
These checks are cheap and they catch a large share of mistakes before anyone spends time on review. A field name that does not exist, a technique ID that was revoked, a test set nobody named: each would otherwise reach production and fail there, as lesson 0.6's rule with the missing function did. Module 17 builds the schema, the validator and the workflow, and runs them on the library.
The checks also set a floor for review. A reviewer who knows the file is complete and its fields exist can spend their attention on the questions only a person can answer: whether the rule detects the right thing, whether its exclusions key on fields the attacker controls, and whether its limitations are honest.
Automation that removes the routine checks makes the human review better, not optional. It also makes review faster, which matters more than it sounds: a pipeline that takes days to approve a change pushes people back to the portal.
Tests in the Pipeline
the bar a rule has to clearReading a file only goes so far. The next step runs the query against the labeled corpus and replays it on its schedule, the two measurements from lesson 0.9, and compares the result with the bar the rule must clear. The bar for the session-replay rule:
# Each rule's bar on its labeled test set, keyed by the rule's id.
8c355665-4ada-465d-8ea9-9a7439a7f5b6: # R6
recall: 1
lookalikes: 0
other: 0
Recall one, no look-alikes, nothing else: every attack in the rule's test set caught, every look-alike left alone, and no other rows in the month. The bar is keyed by the rule's ID, so it follows the rule through renames, and it lives in the repository beside the rules, so changing it is itself a reviewed change.
The schedule replay is the part most pipelines lack. A query that finds the attack when run over the whole month can still miss it on its real frequency and lookback, as lesson 0.6's leaked-key rule did.
Testing in CI runs the rule as the scheduler would, with each run seeing only its window and only what had arrived, so that a change to a lookback is tested as carefully as a change to the query.
Module 17 builds this test harness and finds three production faults with it that every file check had passed. The principle carries beyond the course: a detection pipeline that does not execute the rule on known data is checking its spelling, not its behavior.
The bar also protects against a quieter failure: a change that makes a rule worse without breaking it. An edit that narrows a condition can leave the query valid and the file complete while the rule stops catching one of its cases.
Without a test, that change merges and nobody notices until an attack is missed. With one, the pull request fails and shows which case was lost, before anything reaches production. The test that failed is the explanation, and the author fixes the rule or argues in the pull request that the bar should move.
Deploying, and Changes Made Elsewhere
the pipeline as the only writerAfter review and approval, a deployment workflow pushes the rule to the workspace. For that to mean anything, the pipeline has to be the only thing that changes rules, and a detection has to notice when it is not. Lesson 0.6 read the month's rule changes from SentinelAudit; this is the rule that checks them:
let Writes = SentinelAudit
| where OperationName == "Microsoft.SecurityInsights/alertRules/Write"
| extend P = parse_json(ExtendedProperties)
| project TimeGenerated, SentinelResourceId, SentinelResourceName,
Caller = tostring(P.CallerName), CallerIp = tostring(P.CallerIpAddress),
State = tostring(parse_json(tostring(P.UpdatedResourceState)).properties);
let Pipeline = Writes
| where Caller == "3d7b9e52-8a14-4c6f-b0e3-71f5c2a9d846"
| project SentinelResourceId, Synced = TimeGenerated, SyncedState = State;
Writes
| where Caller != "3d7b9e52-8a14-4c6f-b0e3-71f5c2a9d846"
| join kind=leftouter Pipeline on SentinelResourceId
| summarize Reconciled = countif(Synced > TimeGenerated and SyncedState == State)
by TimeGenerated, SentinelResourceName, Caller, CallerIp
| where Reconciled == 0
It finds every write to an analytics rule made by anyone other than the pipeline, and keeps only those the pipeline did not later overwrite with the same state.
The emergency change on 10 March is gone, because the pipeline redeployed the reviewed version that afternoon. The exclusion added to the session-replay rule on 6 March remains, because nothing ever reconciled it, and it is exactly the change lesson 0.9 showed to be a bypass.
Two things make this work. Deployments run as their own identity, an application whose ID the rule can recognize, rather than as a person's account. And the rule compares states rather than only counting writes, so an emergency fix that is then merged and redeployed is not an alert.
Module 17 builds the deployment workflow with approval gates and secrets kept out of the repository, and Module 18 runs the drift rule in the library.
There is a cost to this discipline, and it is speed. An analyst who can only change a rule through a pull request cannot silence a noisy rule at three in the morning. Teams handle that with an emergency path that is fast and recorded, like the change on 10 March, rather than by leaving the portal open, and the drift rule is what keeps the emergency path honest.
Every alert it raises has two good outcomes: the change is merged into the repository and redeployed, or it is reverted. Either way the repository and the workspace end the day describing the same rule.
A Rule an AI Assistant Drafted
what the file checks catchAI assistants are now a common way to start a detection: describe the behavior, get a rule back. On 16 March an engineer asked one for a rule for the certutil download from lesson 0.7, and opened a pull request with the result:
title: Certutil Used to Download a File
id: 4b7e2c90-1d3a-4f6e-8a52-9c0e7b1f3d64
status: experimental
description: Detects certutil.exe used to download a file from a URL, a common
living-off-the-land technique.
references:
- https://lolbas-project.github.io/lolbas/Binaries/Certutil/
author: Marcus Webb, drafted with an AI assistant
owner: detection-engineering
date: '2026-03-16'
tags:
- attack.command-and-control
- attack.t1086
query_language: kql
tables:
- DeviceProcessEvents
fields:
- Timestamp
- FileName
- CommandLine
query: |
DeviceProcessEvents
| where Timestamp > ago(1h)
| where FileName =~ "certutil.exe" and CommandLine contains "url"
| project Timestamp, DeviceName, AccountName, CommandLine
falsepositives:
- Administrators using certutil
level: high
test_set: lolbin-execution
preconditions: Defender for Endpoint process events
limitations: None known
sentinel:
display_name: Certutil used to download a file
severity: High
frequency: P1D
period: P1D
trigger:
operator: GreaterThan
threshold: 0
grouping: AlertPerResult
tactics:
- CommandAndControl
techniques:
- T1086
incident:
create: true
It reads well. The title and description are accurate, the reference is relevant, the false positives and limitations are filled in, and the query is short and plausible. The quality checker's verdict:
rules/draft/rule.yml: FAIL
ok Required metadata
FAIL ATT&CK mapping
T1086 (PowerShell) is revoked in ATT&CK 19.2
attack.command-and-control is not a tactic of the techniques tagged
(execution)
FAIL Fields and data
field CommandLine is not in DeviceProcessEvents
ok Test coverage
ok Documentation
Two faults, both of the kind assistants are known for. The ATT&CK tag is T1086, a technique ATT&CK revoked, which described PowerShell and has nothing to do with downloading a file; the tactic tag does not match it.
And the query uses a column called CommandLine, which DeviceProcessEvents does not have: the table's column is ProcessCommandLine, and as written the rule cannot run at all. Both are confident and specific, and both are wrong in a way that reading the file would not catch.
The checker found them because it checks the file against facts: the ATT&CK bundle the course vendors and the schema of the tables in the corpus. That is the right first defense against AI-drafted rules, and it costs nothing per rule. It is also not enough.
The checker reported every other group as passing, including documentation that claims no limitations, and the next section shows what the rule still gets wrong once it runs.
The draft also shows where an assistant's output is strongest and weakest. The prose parts, title, description, false positives, are fluent and mostly right, because they are the parts most like the text it learned from.
The parts that depend on facts about one environment, which columns this table has, which techniques are current, how this rule will be scheduled, are where it guesses, and its guesses look exactly as confident as its knowledge.
What Only Running It Shows
two faults no reading would findWith the column fixed, the draft runs. Scored on its test set, without its time filter, it catches the download and flags an analyst's revocation check, because contains "url" also matches -urlfetch: the fault lesson 0.7's practice fixed.
Replayed on its schedule, with the time filter, it never alerts at all. The rule runs once a day and its query keeps only the last hour, so each run reads one hour of the twenty-four, and the download fell in an hour no run read.
The full review, with the check that found each fault:
Four faults in the draft, and what found each one
reviewWith all four fixed, the rule catches the download on its daily schedule, at 00:05 on 4 March, and leaves both look-alikes alone. The fixed rule's time to detect is nearly fourteen hours, which is a separate question for whoever owns it: a download is an early step in an intrusion, and a daily rule may be too slow for it whatever its logic.
None of these faults is peculiar to AI. People write each of them, and lesson 0.9's exclusion was written by a person. What changes with AI drafting is the rate: more rules, written faster, by people who may not have checked the details an assistant fills in.
The answer is not to distrust the assistant's output more than a colleague's, but to run every rule through checks that do not depend on trusting anyone. Module 19 reviews an AI-drafted rule with five faults, on a different technique, the same way.
The review is also a good way to use an assistant well. Asked to draft a rule against a named table with its schema in hand, a named test set and a stated schedule, an assistant has the facts it would otherwise invent. The pipeline then checks the result. Used like that, it saves the routine typing and leaves the judgment where it belongs.
Detecting Abuse of an AI Service
keys, listings and calls from outsideAI services are also something to detect on. Northgate runs an Azure OpenAI resource for its support desk, called by an application with a key. On the night of 13 March someone listed that resource's keys from the management plane:
AzureActivity
| where OperationNameValue has "COGNITIVESERVICES"
and OperationNameValue has "LISTKEYS"
| summarize Listings = count(), FirstSeen = min(TimeGenerated),
LastSeen = max(TimeGenerated) by Caller, CallerIpAddress
The pipeline lists the keys every Monday from its own address when it rotates them. The other listing is a person, d.ogilvie, at 22:40, from 192.0.2.214, an address the account had signed in from for the first time nine minutes earlier. Twenty-four minutes later the model began answering calls from somewhere new:
AzureDiagnostics
| where ResourceProvider == "MICROSOFT.COGNITIVESERVICES"
and Category == "RequestResponse"
| summarize Calls = count(), FirstSeen = min(TimeGenerated),
LastSeen = max(TimeGenerated) by CallerIPAddress
1,658 calls from one outside network in under five hours, after a month in which every call came from one cloud address range. The resource masks the last part of each caller's address in its logs, which is why both appear with asterisks; the network is still enough to tell the two apart.
This is the same pattern as lesson 0.8's service principal: a credential obtained on one plane and used on another. Detecting it needs the management log, which records the listing, and the resource's own diagnostic log, which records the calls, and the second is not collected unless someone turns it on for each resource.
Module 19 builds the rules on both and on the month-end batch job that looks like the abuse, and treats an organization's AI services as part of the estate the detection library has to cover.
AI services raise the stakes of a stolen key in two ways.
The calls cost money, billed per request, so abuse at volume is a direct financial loss. And a model connected to internal data or tools can be asked to reveal or act on them, so a stolen key can be a route into whatever the application behind it can reach. Both make the listing on the management plane worth alerting on by itself, not only when calls follow.
What Modules 17 and 19 Build On It
a library that lastsModule 17 builds detection as code: a repository layout, a schema and review rules; validating rules in CI with a linter and the quality checker; testing queries in CI against labeled data, on their schedule; and deploying with GitHub Actions, approval gates and secrets kept out of the repository.
Module 19 takes on what comes after: anomaly detection with distributions, seasonality and peer groups; rules that survive evasion through renamed binaries, variants and logging gaps; reviewing a detection an AI assistant wrote; and abuse of an AI service.
The question Module 19 asks of the whole library is robustness: how much an attacker has to change to get past each rule. The course's answer, built up over this module, sorts the fields a rule can rely on by who controls them:
Fields a rule can key on, by who controls them
from lessons 0.7 to 0.9Every lesson in this module has had an example. The certutil rule held because it keyed on what the download does, and the session-replay rule failed where it keyed on a user agent the attacker copied. The exclusion on mobile clients failed for the same reason.
A library reviewed against that ladder, tested on labeled cases on its real schedule, and changed only through a pipeline that runs those tests, is a library whose claims can be trusted, and that is what the rest of the course builds.
Anomaly detection, the other half of Module 19, extends the same idea to behavior no one has written a rule for. Instead of a fixed condition, the rule compares each account or machine with its own history and with its peers, as lesson 0.9's peer-group rule did with job titles.
The finance account that downloaded most of its department's site at eleven at night in this month is the course's case: no single download was unusual, and the volume, hour and address together were. Baselines like that need the same testing as any rule, against look-alikes such as a team downloading a new library on its first morning.
Practice
The exercise below is the AI-drafted rule from section five. Fix it so that it runs, catches the download and leaves both look-alikes alone.
- Run the draft. Read the error: the first fault is the one that stops it running.
- Fix the column. Use the command-line column the table actually has.
- Remove the time filter. Let the schedule decide the window.
- Fix the match. Match the download's argument as a whole term.
- Check the score. The download caught, both look-alikes left alone.
A rule kept as a file carries everything needed to judge it: its owner, its tests, its preconditions and its known limits. Every change runs the same checks, file checks first and then the rule itself on labeled cases and on its schedule, and the pipeline is the only thing allowed to deploy. That process is what makes AI-drafted rules usable, because it judges each rule by what it does.
That completes the foundation. Module 1 starts the course proper: reading a detection, the life of one rule, and judging a detection's quality on rules Northgate already runs.