Four Things That Go Wrong

Introduction

The four outputs of this job each fail in a specific way, and the six modules after this one are arranged around preventing those four failures rather than around the products that produce the outputs.

This section names them properly and puts numbers against them, because all four share a property that makes them hard to notice: none produces an error. A detection that cannot fire, a hunt asking the wrong question, a verdict with nothing behind it and an artifact that was never collected all look, from a console, exactly like everything working.

You will finish able to recognize each of the four in your own estate, and to say which module of this course addresses it.

Each failure has a module. Knowing which is which turns a vague sense that something is wrong into a specific place to look.

They are also ordered by how expensive they are to fix once found, cheapest first, and that order is not the order they feel urgent in. Urgency tracks visibility, and these are ranked by consequence, which is why a team following its own sense of urgency tends to work them backwards. A broken detection is an afternoon. A hunt with the wrong question is a conversation. A team that does not record its reasoning is a habit change measured in months. And evidence that was not collected on a machine that no longer exists is not fixable at all, only preventable for next time.

The fourth carries a marker the others do not, and it is the reason readiness sits before triage in this course. A bad detection can be rewritten and a poor hunt re-run; an artifact that was never collected from a machine that no longer exists is simply gone.

The first three also compound in a specific direction that is worth seeing now. A detection that never fires means an intrusion is found later, by a hunt. A hunt with the wrong question means it is found later still, by an incident. And an incident worked without a defensible record means the same intrusion teaches the organization nothing, so the detection that would have caught it never gets written. The loop closes badly and each failure makes the next one likelier.

A detection rule that cannot fire looks identical to one that has simply found nothing. Both sit in the library, both are enabled, and the platform reports no error against either.

There is no state in any console for demonstrated. A rule is enabled or disabled, and enabled is doing an enormous amount of work in most people's heads: it is read as working, and it means configured. Every argument in this section follows from the gap between those two words.

This is measured rather than anecdotal. Analysis of more than 13,000 production detection rules across Splunk, Sentinel, QRadar and other platforms put broken rules that will never fire at 13 per cent of active production deployments, down from 18 per cent in the previous year's dataset. On a library of a hundred rules that is thirteen you are counting and not getting.

Take a moment on the direction of that figure, because it is encouraging and it is not a reason to relax. Eighteen per cent falling to thirteen means the industry is getting better at noticing, which almost certainly reflects more teams testing rather than fewer rules breaking. Breakage is a property of a changing environment and it does not decline on its own.

Assume your own library sits near that figure until you have measured it. It is not a comment on the people who wrote the rules, and the study covers exactly the platforms most organizations run. A team that believes it is the exception without having checked is describing a hope.

The number also understates the position on an endpoint estate specifically. Those studies count rules in a SIEM, where a broken rule is usually a broken query. On endpoints you have two rule sets, your custom detections and the vendor's built-in ones, and you can only test the first. The second is a black box you are trusting, which is reasonable and is worth stating out loud when somebody asks what your coverage is.

You can test the vendor's set indirectly, which is the useful workaround. Executing a technique and seeing what fires tells you whether anything caught it, without needing to know which rule did or how it works. That is a coarser answer than you get for your own rules and it is the one that matters operationally, because the question is whether the estate detects the behavior rather than which component did.

// Custom detections by how recently they fired
AlertInfo
| where Timestamp > ago(365d)
| where DetectionSource == "Custom detection"
| summarize Alerts = count(), LastFired = max(Timestamp) by Title
| sort by LastFired asc

That query finds the rules that fired. The rules you actually want are the ones absent from its output entirely, which means the useful version of this reading is the comparison between your rule list and that result, and the rule list does not live in the same place.

That awkwardness is worth naming rather than working around, because it explains why this check is so rarely done. Nothing in the platform presents rules that have not fired, since a rule with no alerts produces no rows anywhere, and building the list requires exporting the rule definitions and diffing them against the alert history by hand. It is fifteen minutes of unglamorous work that almost nobody does, which is precisely why the thirteen per cent figure survives.

There are three distinct reasons a rule never fires, and they need different fixes. It queries data that is not there. It references a field that has changed under it. Or it is correctly detecting something that has not happened. Only the third of those three is an acceptable state to be in, and nothing in a console distinguishes them from one another.

Establish which one you have by working backwards from the data rather than forwards from the rule. Strip the rule to its table and time range and run that alone: no rows means the data is not arriving and the rule was never going to work. Add the conditions back one at a time and the clause that takes the result to zero is the one to look at, which is usually a field name or a value that no longer matches what the platform emits.

Watch for the clause that takes the result to zero being correct, which happens more often than expected. A condition matching a signer name that changed when a vendor re-signed their software, or a path that moved in an operating system update, is a rule that is still asking the right question about a world that has moved. Those are quick to fix and impossible to find without taking the query apart.

That takes a few minutes per rule and it is the single most useful diagnostic technique in this course. It is also the reason the hunting module comes before the detection module: the skill of taking a query apart is the same skill in both, and it is easier to learn on a question you are exploring than on a rule you are trying to defend.

That is why the next two sections are about testing rather than about writing. A library nobody has executed against is a library of hypotheses, however carefully each one was reasoned.

The same body of research found something more uncomfortable than broken rules. The organizations studied were already ingesting enough telemetry to cover around ninety per cent of ATT&CK techniques, and their actual detection coverage was twenty-two per cent. Across the ten techniques seen most often in real intrusions, the average deployment covered four.

That is not a data problem, which is the part worth sitting with. The telemetry is arriving and being paid for; what is missing is detection logic written against it.

It reframes what a detection engineering backlog is competing for. The usual assumption is that better coverage needs more data, which is expensive and requires somebody else's budget. If the data is already there, coverage needs engineering time instead, which is cheaper and is entirely within the team's own control. That is a considerably easier thing to ask for and a considerably harder thing to defer.

The four-out-of-ten figure is the one to quote if you only quote one. It says that on the techniques adversaries actually use most, the average deployment detects fewer than half, and it does so on estates already collecting the telemetry that would cover them.

Read the last block. A coverage figure computed against the techniques you already cover cannot report a gap, by construction, and it will keep reporting a healthy number as the estate changes underneath it.

The honest replacement is not a bigger percentage. It is a smaller, harder claim: a named list of techniques, each with the rule that covers it, the date that rule was last demonstrated to fire, and the data source it depends on. That fits on a page, it cannot be inflated, and every row in it is falsifiable.

It also survives the question that breaks a percentage, which is what would happen if somebody did this to us today. A percentage cannot answer that. A list with demonstration dates against it can answer it technique by technique, including with the word no.

Being able to say no is the part that earns trust. A coverage report with no gaps in it reads as a report nobody stress-tested, and one that names three uncovered techniques with a reason for each reads as a team that knows its own estate. The second gets believed on the rows where it says yes.

The exercise below is the scenario at the top of this section as an artifact. Every rule is enabled, the platform reports no errors, and the reported coverage is accurate about what it counted.

Read the rule detail block before the firing counts. The counts invite a conclusion and the detail is what supports one, and doing it in that order is the habit this whole section is arguing for.

The two rules that look wrong and are not carry the lesson worth keeping. A rule firing twice in a year is either rare and valuable or broken, and the firing count alone cannot tell you which. It is the detail beside the number that separates them, which is why a review that reads only the counts will retire the wrong rules.

Retirement is still worth doing, which is the other side of that. A rule that has fired only false positives for a year is training your analysts to disregard the platform, and one that has never fired and cannot is occupying attention and audit surface for nothing. The discipline is to retire on a stated reason rather than on a count, and to record the reason where the next person will find it.

The cure for the first failure is executing the behavior deliberately and confirming an alert appears. That sounds heavier than it is: open-source libraries of technique simulations exist for exactly this, with over a thousand small contained tests mapped to individual ATT&CK techniques, available as PowerShell for Windows and shell scripts elsewhere.

That ordering saves the most common wasted afternoon, which is rewriting logic for a rule whose data was never arriving.

Each test does one thing, on one host, and leaves a known trace. You run it, you look for your alert, and you have converted an assumption into a fact in about ten minutes.

The middle column is what makes this worth the afternoon. A rule that produces no alert and no rows has a data problem that belongs to whoever owns collection. A rule that produces no alert and plenty of rows has a logic problem that belongs to you. Those go to different people, and without the raw row count you cannot tell which without guessing.

Record the passes as carefully as the failures. A rule demonstrated to fire on a named date is the only rule in your library about which you can make a positive statement, and a coverage claim built from demonstration dates is a different artifact from one built from rule counts.

Do this on a schedule rather than once, because the failure mode being defended against is drift. Quarterly is the commonly cited minimum and the useful trigger is a change rather than a date: an agent version, an operating system update, a schema change or a policy edit are each a reason to re-run the affected tests.

Start with the rules that matter rather than with the whole library. The ten techniques most common in real intrusions are documented and short, and validating your coverage of those first gives you a defensible answer to the only coverage question anybody senior actually asks, which is whether you would see the ordinary case.

Agree it with whoever owns the endpoints before you start. Executing attacker techniques on production machines without telling anybody is a good way to generate a genuine incident about yourself, and the conversation is easy because the exercise produces evidence they benefit from too.

Pick the machine carefully as well. A dedicated test host is cleanest and it is also the least representative, because it usually has a different policy set from the fleet. Running against one ordinary managed workstation, with the owner's agreement, tests the configuration that actually exists rather than the one you would have chosen.

The third and fourth failures are less measurable and no less common.

A decision nobody can defend is usually a correct decision recorded badly. An analyst saw something at the time, concluded correctly, and wrote three words in a ticket. Six weeks later the view they used has aged out, the conclusion cannot be reconstructed, and the only honest answer to why the ticket was closed is that somebody looked at it. That is a habit problem rather than a skill problem, and the fix is a record made during the decision.

The cost lands in three places and none of them is the original ticket. The same alert type arrives next month and the next analyst repeats the whole investigation, because the last one's reasoning is not there to inherit. An auditor asks how alerts of this class are handled and the honest answer is that it varies. And when one of those closures turns out to have been wrong, there is no way to tell whether the reasoning was flawed or the evidence was, which are different problems with different fixes.

What a good record looks like is shorter than people fear. What you looked at, what it showed, and what would have changed your mind. Three lines, written while the view is still open, and the third one is the one that makes it defensible rather than merely present.

The third line also protects the analyst. A closure that states the condition under which the verdict would have been different is a decision made against stated criteria, and if it turns out to be wrong the criteria were wrong rather than the person. That distinction matters a great deal the first time a closed ticket is revisited in an incident review.

Evidence that was never collected is not a security failure at all. Rebuilding a compromised machine quickly is correct behavior from a service desk measured on getting a user working, and it destroys the only copy of what happened. Nobody in that chain did anything wrong, which is exactly why asking them to behave differently does not work and why the answer is collection that triggers on a signal they already generate.

Telemetry is not a substitute for it, though it is frequently treated as one. The device tables record what the sensor saw and they do not contain the file, the memory, the registry hive or the scheduled task definition, so an investigation working from telemetry alone can establish that something ran and cannot examine what it was. That distinction is invisible until somebody asks a question that needs the artifact.

This is also the failure that is hardest to notice, because its symptom is an investigation that concluded less than it might have. Nobody records what an investigation could not establish, so the cost never appears in a metric and the readiness gap that caused it stays open for years.

Start recording it and the gap closes quickly. A single line at the end of each investigation naming what you could not establish and why turns an invisible cost into a list, and four entries on that list saying the machine was rebuilt is a business case that writes itself.

None of the four requires permission, a budget or a change window, which is what makes them the right first week of work on an estate you have just inherited.

Four failures, four checks, and each returns a number rather than an impression.

Notice what all four checks have in common. Each replaces a belief with a measurement, and in every case the belief was reasonable and held by competent people. That is the shape of this discipline: the failures are not caused by carelessness, they are caused by systems that report success in the same way whether they are succeeding or not.

The next section introduces the estate every module of this course works against, so the readings above have somewhere concrete to happen.