In this section

The Vocabulary of Coverage

Module 0

Operational Context

A purple-team program produces evidence, and evidence needs language that is precise, shared and unambiguous. Without it, teams default to "we have good coverage", which is unmeasurable, and "detection is handled", which is unfalsifiable. Neither survives a question from anybody who wants a number.

Six metrics, each with the query or calculation that produces it.

Figure PT0.3. The six metrics the course tracks. Each is populated per technique as you walk the course.

Learning Objectives

Separate the six metrics and know what each measures. Mean time to detect, validated coverage, detection quality, false-positive classification, remediation backlog and cadence compliance. This matters because they fail independently, so a single headline number always conceals at least one of them.
Understand why validated coverage differs from deployed coverage. This matters because deployed counts the rules you own and validated counts the ones you have seen fire, and the difference between those two numbers is the finding.
Read a false-positive classification as a work assignment. This matters because an environmental false positive needs an exclusion, a logic false positive needs a rewrite, and reporting them as one number sends the work to the wrong person.

Metric 1: MTTD, mean time to detect

MTTD is the time from the moment you execute the attack to the moment the first alert appears in the SIEM. Here's a worked example.

You run credential dumping at 14:32:00:

# Attack executed at 14:32:00
procdump.exe -ma lsass.exe C:\Windows\Temp\debug.dmp

The Sentinel analytics rule is scheduled to run every 5 minutes. At 14:37:12, the rule fires:

// The alert that fired
SecurityAlert
| where TimeGenerated == datetime(2026-04-22T14:37:12Z)
| project TimeGenerated, AlertName, AlertSeverity

TimeGenerated           AlertName                        AlertSeverity
─────────────────────── ──────────────────────────────── ─────────────
2026-04-22T14:37:12Z    LSASS Access - Credential Dump   High

MTTD = 14:37:12 - 14:32:00 = 5 minutes 12 seconds.

That number is a fact. The judgment depends on what happens next. If automated isolation triggers on the alert and completes in 30 seconds, total exposure is 5 minutes 42 seconds. If the alert goes to a queue that isn't triaged for four hours, the MTTD is irrelevant: the response time dominates.

You can measure MTTD for any technique with this KQL pattern:

// Measure MTTD, time between technique execution and first alert
let AttackTime = datetime(2026-04-22T14:32:00Z);
SecurityAlert
| where TimeGenerated > AttackTime
| where AlertName has "LSASS" or AlertName has "Credential"
| summarize FirstAlert = min(TimeGenerated)
| extend MTTD = FirstAlert - AttackTime
| project AttackTime, FirstAlert, MTTD

AttackTime              FirstAlert              MTTD
─────────────────────── ─────────────────────── ──────────────
2026-04-22T14:32:00Z    2026-04-22T14:37:12Z    00:05:12

The course records this per technique, per variant, per SIEM. By Module 14, you'll answer "how fast do we detect credential dumping" with a number, not a feeling.

Two cautions about this number, because it is the one most often quoted and most often quoted wrongly.

The mean hides the shape. A team with a mean of eight minutes may be detecting most techniques in two and three techniques in an hour, and the three are where the incident comes from. Report the distribution or at least the worst case alongside the mean, because an attacker does not experience your average.

And detection time only means something next to response time. Four minutes to detect is excellent if containment is automated and irrelevant if the alert queues until morning. So the pair to report is time to detect and time to act, and where the second is unknown the first should be presented as a ceiling on your performance rather than a measure of it.

Metric 2: Validated coverage percentage

Deployed coverage is the count of rules in your SIEM. Validated coverage is the subset that have been tested against the actual technique in the last 90 days. Here's a worked example.

Your threat model scopes 61 ATT&CK techniques (the same 61 the course covers). After your first quarter of purple-team work:

Program Coverage. Q1 Assessment
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Techniques in scope:                          61
Techniques with deployed rules:               45  (74% deployed)
Techniques tested in last 90 days:            18  (29.5% validated)
Techniques with broken rules (found by test): 7   (15.6% of deployed)
Techniques with no rule at all:               16  (26.2% uncovered)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Validated coverage: 18 ÷ 61 = 29.5%

The 74% deployed number is the one most teams report. The 29.5% validated number is the one that's true. The 7 broken rules, rules that exist, show as active, but don't fire, were invisible until testing surfaced them.

The 90-day window is load-bearing. Vendor telemetry shifts, attacker tool evolution, environment divergence, and tuning drift all have a realistic chance of breaking a rule within 90 days. A rule validated last week almost certainly works. A rule validated six months ago is a hypothesis.

The calculation is the same idea in any platform, and worth running against your own estate rather than reading. It answers which rules have ever fired, which is not the same question as which rules exist.

// Which deployed rules have produced an alert in the window, and which never have.
// The second list is your validated-coverage gap.
let window = 90d;
let fired = SecurityAlert
| where TimeGenerated > ago(window)
| summarize last_fired = max(TimeGenerated), alerts = count() by AlertName;
fired
| project AlertName, last_fired, alerts
| order by last_fired asc

// Defender XDR: custom detection rules that have generated alerts
AlertInfo
| where Timestamp > ago(90d)
| summarize last_fired = max(Timestamp), alerts = count() by Title, DetectionSource
| where DetectionSource == "Custom detection"
| order by last_fired asc

| rest /servicesNS/-/-/saved/searches
| search is_scheduled=1 alert.track=1
| table title cron_schedule
| join type=left title
    [| search index=_internal sourcetype=scheduler status=success earliest=-90d
     | stats max(_time) as last_fired count as runs by savedsearch_name
     | rename savedsearch_name as title]
| fillnull value="never" last_fired
| sort last_fired

A rule with no firings in ninety days is not necessarily broken, and it is necessarily unvalidated. That distinction is the whole of this metric: the query above gives you the candidates, and only firing the technique tells you which of them work.

Metric 3: Detection quality score

Not all detections are equal. A rule that fires but produces 200 false positives per week is worse than a rule that fires cleanly. Detection quality captures this using three components, each scored 0–5.

Here's a worked example for the LSASS credential dumping rule:

Detection Quality Score. T1003.001 LSASS Memory
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
#
Component 1: Coverage (does the rule fire?)
  Catches 5 of 6 known variants (Mimikatz, procdump,
  comsvcs.dll, reflective loader, DCSync).
  Misses NanoDump (documented in remediation backlog).
  Score: 4/5
#
Component 2: Telemetry richness (what does the alert include?)
  Alert includes: SourceImage, TargetImage, GrantedAccess,
  SourceUser, DeviceName, TimeGenerated, ProcessId.
  Triage analyst can identify the tool, the target, and the
  user without pivoting to another table.
  Score: 5/5
#
Component 3: Tuning maturity (has FP noise been managed?)
  Two environmental FPs tuned out (VeeamAgent, SCCM client).
  One benign TP documented (IT admin procdump for crash dumps).
  No blanket exclusions. Tuning reviewed within last 30 days.
  But NanoDump gap is open, accepted risk, not tuned.
  Score: 3/5
#
Detection Quality Score: (4 × 5 × 3) ÷ (5 × 5 × 5) × 100 = 48/100
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Interpretation: functional detection with a coverage gap
and room for tuning improvement. Priority: close the
NanoDump gap to raise coverage to 5/5, which lifts the
score to 60/100. Full tuning review lifts it further.

A score of 48 means "working but needs attention." A score of 80+ means "solid, maintain it." A score below 20 means "broken or absent, prioritize." The quarterly coverage assessment uses these scores to decide where to spend the next cycle's effort.

Metric 4: False-positive classification

Three categories. Different fix for each. Here's a concrete example of all three from the same LSASS detection rule.

Environmental FP: the backup service.

Your rule fires on this Sysmon Event 10:

{
  "EventID": 10,
  "SourceImage": "C:\\Program Files\\Veeam\\Backup\\VeeamAgent.exe",
  "TargetImage": "C:\\Windows\\System32\\lsass.exe",
  "GrantedAccess": "0x1010",
  "SourceUser": "NT AUTHORITY\\SYSTEM"
}

The rule correctly detects LSASS access. The source is a backup agent performing legitimate credential validation. The rule is right. The environment produces legitimate activity that looks like the attack. Fix: exclude VeeamAgent.exe by full path and hash, scoped narrowly:

// Environmental FP exclusion. Veeam backup agent
| where InitiatingProcessFileName != "VeeamAgent.exe"
    or InitiatingProcessFolderPath !startswith
       "C:\\Program Files\\Veeam\\Backup\\"

Verify the exclusion doesn't suppress a real attack that masquerades as VeeamAgent.exe by checking the path: an attacker dropping a binary named VeeamAgent.exe in C:\Windows\Temp\ wouldn't match the folder path filter.

Rule-logic FP: the overly broad PowerShell rule.

A PowerShell execution detection rule fires on every PowerShell.exe process start:

// Too broad, fires on ALL PowerShell
DeviceProcessEvents
| where FileName =~ "powershell.exe"

This produces hundreds of alerts per day. IT automation, login scripts, SCCM. The rule logic is too broad. Fix: tighten the query to match encoded commands and download cradles specifically:

// Tightened, matches attack patterns, not all PowerShell
DeviceProcessEvents
| where FileName =~ "powershell.exe"
| where ProcessCommandLine has_any (
    "-enc", "-EncodedCommand", "FromBase64String",
    "Net.WebClient", "DownloadString", "Invoke-Expression",
    "IEX", "bypass", "-nop"
  )

Retest after tightening to confirm the real attack variant still fires.

Benign TP: the IT admin running procdump.

Your rule fires on this event:

{
  "EventID": 10,
  "SourceImage": "C:\\Tools\\procdump64.exe",
  "TargetImage": "C:\\Windows\\System32\\lsass.exe",
  "GrantedAccess": "0x1FFFFF",
  "SourceUser": "NORTHGATE\\admin.jmorris"
}

The detection is correct. The activity is real. The user (admin.jmorris) is an IT administrator running procdump to collect a crash dump for a support ticket. This is a benign true positive: the alert is doing exactly what it should. Fix: acknowledge, classify, close. Do not tune it out. The same pattern from NORTHGATE\t.ashworth (a finance user) is the real attack.

Every Tuning Loop element in every technique sub classifies the expected false positives into these three categories and gives you the specific fix for each.

The reason to classify rather than count is that the two categories point at different people and different fixes, and a single false-positive rate hides which one you have.

An environmental false positive means the rule is correct and your estate happens to produce matching telemetry legitimately. A backup agent reads LSASS; a deployment tool spawns interpreters from Office. The fix is a narrow exclusion by hash, path, parent or account, and it belongs to you. It takes minutes and it is safe, provided the exclusion is specific enough that an attacker cannot simply run from the excluded path.

A logic false positive means the rule is wrong. It matches a pattern broader than the technique, and no exclusion list will fix it because the next false positive will come from somewhere else. The fix is a rewrite, and the honest interim position is that the technique is uncovered.

Reporting them together as a percentage tells the reader neither thing. Reporting them separately turns the number into a work assignment: this many exclusions to write, this many rules to rewrite, and this many rules to retire because the technique cannot be detected with the telemetry you have.

Metric 5: Remediation backlog

Every purple-team cycle produces gaps. The backlog tracks them. Here's a populated entry:

Remediation Backlog Entry
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Technique:    T1003.001. LSASS Memory
Gap:          NanoDump variant not detected by current rule
Root cause:   NanoDump uses GrantedAccess 0x0040 (PROCESS_DUP_HANDLE)
              which bypasses the 0x10/0x1FFFFF filter in the Sigma rule.
              Sysmon Event 10 fires but with a different access mask.
Priority:     HIGH (NanoDump is actively used in 2026 ransomware
              campaigns. M-Trends 2026 reports it in 23% of
              credential-access incidents)
Effort:       MEDIUM (requires Sysmon config update to capture
              GrantedAccess 0x0040 + new Sigma rule variant)
Status:       OPEN
Assigned:     Week 14 technique cycle
Created:      2026-04-22
Last review:  2026-04-22
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

The entry names the gap, the root cause, the priority (with citation), the effort to fix, and the status. The quarterly review presents the backlog to leadership. The maturity signal isn't zero gaps, it's gaps identified, prioritized, tracked, and communicated.

The number to watch is not the size of the backlog but its velocity: gaps closed divided by gaps found, per quarter. Below 1.0 the backlog grows, which is normal for the first two or three quarters of a new program and a problem after a year. Above 1.0 you are catching up. Reporting the ratio rather than the count also protects the practice from its own success, because a team that gets better at finding gaps would otherwise appear to be getting worse.

Metric 6: Program cadence compliance

Binary per week. Did you run the test or didn't you? Here's what the tracker looks like:

Cadence Tracker. April 2026
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Week    Technique Tested         Daily  Weekly  Monthly
──────  ────────────────────────  ─────  ──────  ───────
W14     T1003.001 LSASS Memory    5/5    ✓       n/a
W15     T1059.001 PowerShell      4/5    ✓       n/a
W16     T1055.001 DLL Injection   5/5    ✓       n/a
W17     (production incident)     2/5    ✗       ✓ (chain)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Monthly cadence: April chain emulation completed W17
Weekly compliance: 3/4 (75%). W17 missed due to incident
Daily compliance: 16/20 (80%)

Week 17 was missed because of a production incident. The tracker records it honestly. The quarterly review notes the miss and documents whether a reduced-scope test (a single atomic, 15 minutes) would have been feasible. Cadence compliance is about sustainability, not perfection.

Compliance is worth measuring because the cadence is the first thing sacrificed when the SOC gets busy, and its absence is invisible for a quarter. A missed weekly exercise produces no alert and no gap in any report, so the only way it surfaces is a metric that counts what was scheduled against what happened.

The program template, what it actually looks like

All six metrics live in a single Excel workbook. Here's the structure with sample data from one technique:

Tab 1: Coverage Matrix
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Technique      Env      Sentinel  Defender XDR  Splunk  Last Test
─────────────  ───────  ────────  ────────────  ──────  ──────────
T1003.001      Windows  PASS      PASS          PASS    2026-04-22
T1003.001      AD       PASS      PASS          MISS    2026-04-22
T1003.002      Windows  PASS      PASS          PASS    2026-04-15
T1003.003      AD       PASS      FAIL          N/A     2026-04-08
T1059.001      Windows  PASS      PASS          PASS    2026-04-23
T1059.001      Linux    PASS      N/A           PASS    2026-04-23
#
Tab 2: MTTD Log
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Technique      Variant       Sentinel  XDR    Splunk
─────────────  ────────────  ────────  ─────  ──────
T1003.001      mimikatz      4s        2s     8s
T1003.001      procdump      4s        3s     9s
T1003.001      comsvcs.dll   5s        2s     MISS
T1003.001      NanoDump      MISS      MISS   MISS
#
Tab 3: FP Classification
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Technique      FP Source            Type           Fix Applied
─────────────  ───────────────────  ─────────────  ──────────────────
T1003.001      VeeamAgent.exe       Environmental  Path+hash exclusion
T1003.001      SCCM CcmExec.exe     Environmental  Parent proc filter
T1003.001      IT admin procdump    Benign TP      Close, don't tune
T1059.001      Login scripts        Rule-logic     Tightened to -enc
#
Tab 4: Detection Quality Scores
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Technique      Coverage  Telemetry  Tuning  Score  Trend
─────────────  ────────  ─────────  ──────  ─────  ─────
T1003.001      4/5       5/5        3/5     48     ↑
T1059.001      5/5       4/5        4/5     64     →
T1078.004      2/5       3/5        1/5     5      NEW

By Module 14, every row has data from your own lab. The template is the artifact you take to your team, populated, evidence-backed, defensible.

What "coverage" looks like before you validate

Coverage is not a single number, it is a posture built from the detection capabilities you have turned on, and most of it is assumed rather than proven. Toggle the capabilities below against Northgate Engineering, which starts with Defender XDR and Sentinel but zero validated detections, and watch endpoint, identity, and network coverage move. Note the difference between the capabilities you can buy or enable and the one that actually proves a detection fires: purple-team validated rules. That gap between "enabled" and "validated" is the whole reason this course exists, and the rest of the modules close it technique by technique.

Three failure patterns in coverage reporting are worth recognizing by name, because you will be asked to produce at least one of them.

Counting rules. A number derived from how many detections exist in the SIEM measures your configuration, not your capability, and it moves in the wrong direction under pressure: a team that deletes forty broken rules has improved its actual coverage and reduced its reported coverage. Any metric that punishes cleanup is measuring the wrong thing.

The heatmap with no quality dimension. A technique shaded green because a rule exists, sitting next to one shaded green because a rule exists, fires reliably and triggers automated containment, tells the reader those two are equivalent. They are not, and the picture gives no way to tell. That is why detection quality is a separate metric here, and why breadth and quality are reported as two numbers throughout this course.

A percentage with no denominator. Coverage of what? ATT&CK Enterprise is over six hundred techniques and sub-techniques, most irrelevant to your estate, so a percentage against that is meaninglessly low. A percentage against the techniques in your industry's threat reporting is defensible. A percentage against the techniques you happened to test is circular.

The version that survives scrutiny states all three. "Of the 40 techniques we assess as relevant to our environment, 34 have a deployed detection, 21 have been validated in the last quarter, and 8 trigger automated containment." Nobody can misread that, and every number in it is checkable, which is the point.

The six metrics in this sub exist so that you can write that sentence about your own estate by the end of the course, with the numbers filled in from exercises you ran rather than from an inventory you queried.

← Previous Next →

Reading width

Text size