Your Sigma Rule Converts Cleanly and Still Never Fires. Here's the Test That Catches It.

Here is a detection rule. It is valid Sigma. It converts cleanly to KQL, to SPL, to Elastic, to every backend your team runs. It passes sigma check with no warnings. Your CI pipeline goes green and merges it. And it will never fire, not once, no matter how many times the attack it was written to catch runs through your environment.

The reason is a single character. Look closely:

title: LSASS Memory Access
logsource:
    category: process_access
    product: windows
detection:
    selection:
        TargetImge|endswith: '\lsass.exe'   # TargetImage, misspelled
        GrantedAccess: '0x1010'
    condition: selection

TargetImge should be TargetImage. That field does not exist in any Sysmon event ever produced. The rule is looking for a key that will never be present, so its selection can never match, so it is silent by construction. And nothing in your static toolchain will tell you, because from the toolchain's point of view nothing is wrong. TargetImge is a perfectly valid string. The YAML parses. The condition references a selection that exists. The convert step emits syntactically flawless output. Every check you run answers a question about the rule as text, and as text, the rule is fine.

This is not a contrived example. It is the single most common way a detection dies quietly, and the industry has measured how often it happens.

One rule in eight. And these are not neglected shops; they are organisations with mature detection programmes and real budgets. The typo above is exactly the "missing field" case that report describes, reproduced in miniature. The rule claims a technique is covered. The dashboard agrees. The technique is not covered.

Why every static check misses it

Conversion and linting answer one narrow, mechanical question: is this valid Sigma, and does it translate into valid syntax for the target backend. Both questions are about the rule on the page and nothing else. Neither one ever touches a real log line.

That is not a flaw someone forgot to fix. It is the correct scope for those tools. A rule-conversion utility has no business knowing your organisation's telemetry in advance; baking that knowledge in would tie the whole toolchain to one environment's schema and destroy the portability that makes Sigma worth using. sigma check validates structure. sigma convert translates syntax. Doing their jobs perfectly tells you the rule is well-formed, and well-formed is the property least likely to be wrong and least connected to whether the rule does its actual job.

Watch the broken rule sail straight through conversion. The typo survives into the emitted query, because the converter has no way to know TargetImge is not a real column:

// sigma convert -t microsoft_xdr lsass-access.yml   ->  emits clean KQL
DeviceEvents
| where ActionType == "ProcessAccess"
| where TargetImge endswith "\\lsass.exe"   // column does not exist: always empty
    and GrantedAccess == "0x1010"

// sigma convert -t splunk lsass-access.yml   ->  emits clean SPL
source="WinEventLog:Sysmon" EventCode=10
  TargetImge="*\\lsass.exe" GrantedAccess="0x1010"
``` TargetImge never appears in the data, so the search returns nothing ```

Both queries are valid. Both would run in their respective SIEMs without error. Both return zero rows for the rest of time. Run either one by hand during a test and you would see an empty result and, if you were not specifically looking for this, conclude the attack simply had not happened yet. That is the trap: an empty result from a broken rule is indistinguishable from an empty result from a correct rule that is patiently waiting. The dashboard cannot tell them apart. Neither can you, from the query alone.

The one check that catches it

The only thing that separates "silent because it works" from "silent because it is broken" is running the rule against a concrete event and checking whether its matching logic fires. That event is called a fixture: a single representative log record, shaped exactly as your telemetry produces it, with a known expected outcome.

For the LSASS rule, the fixture is one small JSON object holding the fields a real process-access event carries, with the values a genuine credential-dumping attempt produces:

// fixtures/true-positive_mimikatz-lsass.json
{
  "EventID": 10,
  "SourceImage": "C:\\Tools\\mimikatz.exe",
  "TargetImage": "C:\\Windows\\System32\\lsass.exe",
  "GrantedAccess": "0x1010"
}

The field is spelled TargetImage, the way real telemetry spells it, because the fixture is built from what the event actually looks like, not from what the rule claims to look for. That independence is the entire point. A fixture written by copying the rule would copy the typo too, match the broken rule, and prove nothing. A fixture built from the telemetry knows something the rule does not: the truth about the data.

Now run the rule's own selection logic against the fixture and assert the expected result:

$ python3 harness/evaluate.py lsass-access.yml fixtures/
FAIL  true-positive_mimikatz-lsass.json
      expected: MATCH   got: NO MATCH
      selection field 'TargetImge' not present in event

There it is. The fixture fails loudly, names the missing field, and stops the merge, all before the rule ever reaches production. The same broken rule that passed sigma check and converted to flawless KQL and SPL fails the one test that reads a real event. Fix the typo, re-run, and the fixture flips to PASS. The rule that was silent forever is now provably firing on the attack it was written to catch, and you know it on your own schedule instead of discovering it during an incident review.

Notice what this required, because the low cost is the whole argument. No SIEM. No test host. No live malware. One JSON file and a small harness that runs the rule's selection against it. The barrier to catching this entire class of failure was never technical difficulty; it was the habit of never once checking what the rule matches against. Static tools made checking the syntax free, so everyone does it. Almost nobody made checking the behaviour free, so almost nobody does it, and one rule in eight ships broken as a result.

A complete test is a pair, not a single fixture. The true positive proves the rule fires on the attack. A second, benign fixture, an event that looks close to the attack but should not match, proves the rule does not fire on legitimate activity. Together they pin the rule down from both sides: it catches what it must and ignores what it must. A rule proven only against the case it should catch has never been asked whether it also catches things it should not, and that second question is where noisy rules are born.

Shift the check left

There is a runtime version of this problem, and we have written about it before: detection rules that go silent in production and how to hunt them down in your live SIEM. That is the cure. The fixture is the prevention. A rule that ships with a committed fixture proving it fires never becomes a silent rule you have to go hunting for later, because the gap was caught at the one moment it was cheapest to fix: before merge, when the author had the rule open and the fix was one character.

This is what "detection as code" actually means past the slogan. It does not mean storing rules in Git and calling it modern. It means treating a detection the way you treat any other code that matters: it ships with a test, the test runs in CI, and CI blocks the merge if the test fails. The exit code from the fixture run is the gate. A human reads the failure and fixes the typo; the pipeline just enforces that no rule reaches production claiming a coverage it was never proven to provide.

What to do this week

1. Pick your single highest-severity detection rule, the one you would least want to be silently broken. Open the query it runs in production and read every field name against your actual schema. One of them being wrong is a 13% coin flip, and this is the rule where you least want to lose.
2. Write one true-positive fixture for it: a JSON event, built from what the telemetry actually looks like, not copied from the rule, with the field names your SIEM really uses. Atomic Red Team (atomicredteam.io) gives you the technique's real command to shape it from.
3. Run the rule's selection against that fixture and assert it matches. If it does not, you have found a broken rule that was counting as coverage. If it does, you have your first proven detection.
4. Add one benign fixture, an event close to the attack that should not match, and assert NO MATCH. Now the rule is pinned from both sides.
5. Wire the fixture run into CI so a nonzero exit blocks the merge. The Sigma project's rule repository and pySigma are the reference points for rule structure and conversion, and the MITRE ATT&CK pages anchor what each rule should be detecting.

A rule that passes every static check and a rule that actually works look identical right up until the moment the attack arrives, and only one of them fires. Static validation proves your detection is well-formed. A fixture proves it is a detection at all. Ship the second kind.