Detection-as-Code: What You Will Build and Keep

This course is not a lecture series. Every module produces an artifact, and every artifact is a file you commit to a repository in your own GitHub account. By the end you have a working detection-as-code pipeline you can demonstrate, deploy, or extend. Nothing is simulated. Nothing is handed to you pre-built.

Six deliverables make up the pipeline, and they are not independent. Each one feeds the next: the rules produce conversion output, the output feeds the test harness, the harness runs inside CI, CI gates deployment, and the rule metadata generates the coverage layer and rule-health report. The diagram shows how what you build connects into a single system.

Blue blocks are the engineering substrate (rules, tests, CI). Green is deployment. Orange is measurement. The arrows are the data flow: nothing in this pipeline is standalone.

The repository

The central deliverable is the repository itself. It is a Git repository on your GitHub account, structured with a folder layout that enforces the rule/test/documentation contract. When you commit a new detection, the folder structure tells you (and the CI pipeline) where the rule lives, where its test fixtures are, where the conversion output goes, and where the documentation sits.

Every rule is a YAML file. Its test fixtures sit in a tests/ folder alongside it. The folder layout is the contract: if a rule exists, its fixtures must exist too.

Module 2 teaches this structure: the naming conventions, the folder hierarchy, the metadata schema, and the design decisions behind the layout. By the end of Module 2, your repository exists and has its first committed rule. The folder structure is not arbitrary; it is the enforcement mechanism for the rule/test/documentation contract. A CI step can verify that every rule file has a corresponding test directory with at least one true-positive and one benign fixture. The structure makes the contract checkable by machines, not just by reviewers.

The Sigma rule set

Across the course you write Sigma rules covering a representative range of ATT&CK techniques. Each rule has a logsource, a detection condition, ATT&CK tags, a severity level, and false-positive documentation. The rules are authored by you, in your repository, against the Northgate Engineering detection backlog that provides the worked narrative.

The rules are not copied from SigmaHQ. They are written to teach the format, the modifiers, the logsource taxonomy, and the judgment calls that make a rule operational. You learn by writing, not by reading someone else's rules. The range is deliberate: you write rules for process creation, authentication events, registry modification, and network connection so that you see how the logsource abstraction and the modifier system work across different event types. By the time you finish, you have seen enough variety to author Sigma rules for any logsource the format supports.

Each rule also has committed metadata: the ATT&CK technique tags that power the coverage layer, the severity level that determines how the alert is triaged, and the false-positive documentation that tells the reviewer (and the future analyst who triages the alert) which legitimate activity looks similar. The metadata is not optional decoration; it is part of the rule's contract with the pipeline.

Module 1 teaches Sigma in depth. Module 4 teaches backend conversion: running sigma-cli to turn each rule into KQL, SPL, and Elastic query, and writing the field-mapping configuration that makes the conversion work.

The test harness

Each rule is committed with its test fixtures: a true-positive event and a benign-baseline event. A test script reads the rule, converts it, and runs the converted query against both fixtures. The test asserts two outcomes: the rule matches the true-positive event, and the rule does not match the benign event.

The true-positive fixture is a constructed event, not a captured one. You author it to contain exactly the fields and values the detection looks for, based on what the attack produces. For a Kerberoasting detection, the fixture is a TGS request event with the encryption type, service name, and ticket options that distinguish a Kerberoasting request from a normal service-ticket request. The benign fixture is a legitimate TGS request with the same structure but the values that a normal authentication flow produces. Together, the two fixtures form the minimum viable test: the detection fires when it should and stays silent when it should not.

The constructed-fixture approach is a deliberate design choice. Captured events from a live Atomic Red Team execution are valuable when you can get them, but they tie the test to a specific environment configuration. A fixture that includes your lab host's hostname, your test user's SID, and your environment's domain name is harder to share and harder to maintain than one that contains the minimum fields the detection requires. Constructed fixtures are portable, reviewable, and version-controlled. When a reviewer reads the PR, the fixture shows them exactly which fields and values the detection depends on, not the full noise of a production event with 40 fields of context.

The test harness is a script you write, not a third-party tool. It uses the pySigma conversion output and the committed fixtures. The output is a pass/fail result for each rule, and the CI workflow reads that result to block or allow the merge. Because the harness is a file in your repository, it is version-controlled and reviewable. When you add support for a new fixture format or a new assertion type, the change goes through the same PR process as a rule change.

Module 5 is the course's centre of gravity. You build the test harness, write the fixture assertions, learn how to construct representative events, and see how the test output integrates with CI.

The CI workflow

The GitHub Actions workflow is a YAML file in your repository. It runs automatically on every pull request and executes the lint-convert-test-block sequence described in Section 0.4. The workflow is not a template you paste; it is built step by step in Module 6, and you understand every line because you wrote it.

The workflow file lives at .github/workflows/detection-ci.yml in your repository. It uses the free tier of GitHub Actions, so no billing is involved. When you open a PR with a new or modified rule, the workflow fires, and the result appears as a status check on the PR. A failing check blocks the merge. The workflow log shows which step failed and why, so the analyst knows whether the problem is a malformed YAML file, a conversion error, or a fixture test failure.

The blocking behaviour is the key differentiator from a linting tool you run locally. A local lint pass is advisory: the analyst can ignore it and push anyway. A CI check that blocks the merge is enforced by the platform. The analyst cannot merge a failing PR without administrator override, and the override leaves a visible audit trail. This is the same pattern that software teams use to enforce code quality: the machine says no, and the human fixes the issue before trying again.

# You will build this file step by step in Module 6
name: Detection CI
on: [pull_request]
jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Install toolchain
        run: pip install sigma-cli pysigma-backend-microsoft365defender
      - name: Lint
        run: sigma-cli check detections/
      - name: Convert
        run: sigma-cli convert --target microsoft365defender detections/
      - name: Test
        run: python tests/run_fixtures.py

This is a simplified preview. The actual workflow you build in Module 6 handles multiple backends, parallel conversion, the fixture test harness, and the pass/fail gate. The point here is that the CI workflow is a file in your repository: version-controlled, reviewable, and yours.

Automated SIEM deployment

When a PR merges, a second workflow deploys the converted rule to the target SIEM. For Sentinel, this means packaging the KQL query as an ARM template analytics rule and pushing it via the Azure API. For Splunk, this means writing a savedsearches.conf stanza and deploying it through the Splunk REST API. The SIEM receives the rule without a human logging into a console.

The deployment stage also handles the configuration that surrounds the query: the alert severity, the entity mapping (which field is the hostname, which is the user), the suppression window, and the scheduling interval. These settings are defined in the rule's metadata or in the deployment configuration, not typed into a console form. When a rule is updated, the deployment stage updates the configuration alongside the query, so the two cannot drift apart.

Module 7 teaches deployment for both platforms as parallel tracks. You follow the track for the SIEM you have. If you have neither, you complete the pipeline through CI and defer the live-deployment step until you have a target.

The coverage layer

Each Sigma rule carries its ATT&CK technique in the tags field. A script reads every rule in the repository, extracts the tags, and generates a MITRE ATT&CK Navigator layer file. The layer shows a heatmap of which techniques your detections cover and which are gaps. You load the layer in the ATT&CK Navigator to see the picture.

This is coverage-as-code: the coverage answer is a generated artifact, produced from the same repository that holds the rules. When you add a rule, the coverage layer updates. When you retire a rule, the gap reappears. No spreadsheet. No manual audit. The coverage layer can be generated as part of the CI workflow or as a standalone script, and the output is a JSON file that the ATT&CK Navigator reads directly.

The gap analysis is the actionable part. When the coverage layer shows that you have five rules covering credential access but none covering lateral movement via remote services, that gap is a prioritisation input for the detection backlog. The gap is not a guess; it is derived from the actual rule inventory. When leadership asks "are we covered for initial access," you open the layer and show them. The answer is as current as the last merge to main.

Module 8 teaches coverage generation, DeTT&CT integration, gap analysis from the repo, and how to produce a coverage report that non-technical stakeholders can read.

The rule-health report

After deployment, pipeline data tells you which rules are firing, how often, and at what false-positive rate. A rule that has not fired in 90 days may mean the technique is no longer seen in your environment, or it may mean the detection is broken. A rule that fires 200 times a day is producing noise, not signal. A rule whose false-positive rate exceeds 80 percent is wasting analyst time rather than protecting it. The rule-health report surfaces these patterns so they can be triaged.

The report is not a dashboard you check occasionally. It is a systematic review mechanism: the team looks at the rule-health data on a regular cadence and triages the outliers. Rules that have gone silent are investigated. Rules that are too noisy are tuned or retired. Rules that have aged past their review date are re-evaluated against the current threat landscape. This is the feedback loop that keeps the pipeline from becoming a rule graveyard: a collection of detections that were once good but have not been maintained.

Module 10 teaches rule-health metrics, the maturity gradient (from rules-in-a-repo through to a full pipeline), the RACI for detection content, and the cross-team operating model that keeps the programme healthy.

Section 0.6 covers the lab environment and tooling: what you need installed before Module 1.