Detection, Hunting and Triage

Introduction

Detection, hunting and triage appear in the same job adverts, are done by the same people and are treated in conversation as three words for looking at security data. They are three different activities with different inputs, different time positions and different definitions of success.

Confusing them is expensive in a specific way. A hunt judged by the standards of a detection looks like a failure whenever it finds nothing, which is most of the time and is frequently the correct outcome. A detection judged by the standards of a hunt gets rewritten every time it produces a false positive. And triage judged by either gets measured on speed, which is the one thing it should not be optimized for alone.

The confusion also shows up in hiring and in workload planning. A role advertised as threat hunting that is actually a triage queue is one of the more common disappointments in this field, and it happens because whoever wrote the advert genuinely believed the two were the same activity performed with different amounts of enthusiasm.

You will finish able to say which of the three a given piece of work is, what its honest outcomes are, and what it owes the other two.

The second ending is the one worth changing habits over. Finding nothing establishes two facts at once, that the behavior is not present today and that a query exists which would have found it, and only the second is durable.

The cleanest way to tell the three apart is when they happen relative to the thing they are about.

A detection runs continuously and fires at the moment something matches. It is automated, it is reactive, and it operates on known patterns. Triage happens after something has fired, and its subject is a specific alert on a specific asset. Hunting happens without anything having fired at all, on the working assumption that something is already present and has not triggered anything.

Endpoints dominate the middle of those three by volume, which is worth knowing before you plan anybody's week. Published figures put the large majority of alerts across a typical SOC as coming from endpoint detection, so on an endpoint estate specifically the triage load is the load, and any plan that assumes hunting time will be found around the edges of it is a plan that has not looked at the numbers.

That concentration cuts both ways. It means the triage burden is heaviest exactly where this course operates, and it means the tuning improvements available here have more leverage than anywhere else in a security program: a rule narrowed on the endpoint side moves a larger share of total alert volume than the equivalent work on any other surface.

Three properties separate hunting from ordinary monitoring, and all three have to be present. It is human-led, so the query comes from somebody's reasoning rather than from a rule engine. It is hypothesis-driven, so you decide what you are looking for before you look. And it assumes compromise, so the job is to prove or disprove that something is already inside rather than to wait for confirmation.

Drop any one of those and you have something else. A query somebody runs because a dashboard looked odd is investigation. A query with no hypothesis is browsing, and it produces the four-thousand-row result from EO0.1, which is a volume nobody can read and a question nobody asked.

Browsing is not a criticism and it is worth doing deliberately when you are new to an estate. Looking at what a table contains, what the ordinary distributions are and which processes dominate is how you acquire the sense of normal that every later hunt depends on. The failure is only in calling it a hunt afterwards, because it has no hypothesis to have refuted and no ending to have reached.

Baselining sits between browsing and hunting and deserves its own name. Establishing what the twelve normal scheduled task command lines on your estate are is not a hypothesis test and it is not aimless: it is the work that makes every subsequent hunt in that area fast, and it produces an artifact other people can use. Teams that skip it re-derive normal every time they hunt, which is most of the cost of hunting.

The published frameworks all converge on the same loop and it is worth knowing one of them by name. The most useful contribution of the current best-known framework is not its phases but its insistence that there are distinct kinds of hunt: one that tests a stated claim, one that establishes what normal looks like so deviations become visible, and one driven by a model or a risk score. The second of those is the one teams skip, and it is the one that makes the other two cheaper.

This is the part that gets hunting programs canceled. A hunt that finds nothing is reported as a hunt that achieved nothing, and after two quarters of that somebody reasonably asks why the team is spending senior hours on it.

There are exactly three ways a hunt can honestly end, and all three are results.

The second ending is the one worth changing your habits over. Finding nothing establishes two facts at once: the behavior is not present today, and a query exists that would have found it. The second fact is durable and the first is not, which is exactly the argument for converting the query into something that runs by itself.

There is a caveat that stops this becoming a rule-generating machine. Not every hunt query makes a good detection: a query that returns a distribution rather than a match, or one that needs a human to look at rarity, does not convert directly and has to be narrowed first. That narrowing is detection engineering and it is a real piece of work, which is why the honest version of this ending is that the hunt produces a detection candidate and the candidate goes into a queue.

The distinction matters when somebody asks why last quarter's six hunts produced two rules rather than six. Two of the four were probably not convertible, and saying so is a better answer than pretending the ratio should be one to one.

The third ending is the one that feels worst and is frequently the most valuable. A hunt that discovers you cannot see a class of activity has found a gap that was invisible before it, and unlike a finding it will still be true next month.

There is one more thing a third-ending hunt should produce, and it is the reason to be precise about the wording. Record what you were unable to establish, not only what was missing: "could not determine whether any host loaded an unsigned DLL from a signed binary's directory" is a sentence a manager can act on, where "image load table unavailable" is a sentence only you understand.

Write it as a ticket with an owner rather than as a line in a report. A visibility gap recorded in a hunt document is read once by the person who already knew about it; the same gap as a ticket against whoever owns collection is a thing with a state, and gaps with states get closed. This is the single most common place a hunting program leaks value, and the fix costs two minutes at the end of the hunt.

Four such tickets over a year is also the strongest argument available for a retention or routing change, for exactly the reason EO0.3 gave: it is a list of questions the organization wanted answered and could not be, which is a different kind of document from a request for more logging.

Keep them in one place for that reason, rather than scattered across a ticketing system by date. A standing list of what this estate cannot currently answer is a small artifact that takes minutes to maintain and answers a question every security team is eventually asked, which is what we would not be able to tell you.

Every link in that loop is cheap to build and none of them builds itself. A team running the three activities in separate meetings has all three and none of the compounding.

Take one behavior, scheduled tasks running from user-writable directories, and watch what changes as it moves between the three activities.

// HUNTING: wide, exploratory, grouped so a human can read the shape
DeviceEvents
| where Timestamp > ago(90d)
| where ActionType == "ScheduledTaskCreated"
| summarize Devices = dcount(DeviceName), Runs = count()
          by TaskAction = tostring(parse_json(AdditionalFields).TaskName)
| sort by Devices asc

That is a hunt. It is wide, it returns everything, and it is sorted so the rarest rows appear first because rarity is the signal a human is reading for. It would be a poor detection: it fires on nothing, because it is not looking for a match, it is looking at a distribution.

Notice what the ninety-day window is doing. A hunt wants the longest range the data supports, because rarity is only meaningful against a large enough population and a week of data makes ordinary things look rare. That is also why retention is a hunting constraint before it is an investigation constraint: at thirty days of interactive data, a hunt asking what is unusual has a month to judge unusual against.

// DETECTION: narrow, specific, and it either matches or it does not
DeviceEvents
| where Timestamp > ago(1h)
| where ActionType == "ScheduledTaskCreated"
| extend Action = tostring(parse_json(AdditionalFields).TaskName)
| where Action has_any (@"\Users\", @"\AppData\", @"\Temp\")
| where InitiatingProcessAccountName !in ("system", "svc_deploy")
| project Timestamp, DeviceName, Action, InitiatingProcessAccountName, InitiatingProcessCommandLine

That is the same behavior as a detection. The window shrank to the rule's cadence, the grouping disappeared because a rule alerts on rows rather than on distributions, the conditions narrowed to the specific shape worth waking somebody for, and the known-good accounts from the hunt's baseline became an exclusion.

Four changes, and every one of them came out of the hunt. That is the relationship between the two activities made concrete: the hunt supplied the baseline that became the exclusion list, and without it the detection would have shipped either too broad or with exclusions somebody guessed at.

It is also why a detection written without a preceding hunt tends to be tuned into uselessness over its first six months. The exclusions accumulate one incident at a time, each added under pressure by whoever was on shift, and nobody ever sees the distribution the hunt would have shown them in an afternoon.

Triage is neither of those. It is the third query, scoped to one device and one time window, and it is written after the alert arrives because its question is what else happened around this.

Triage queries are also the ones most worth keeping and least often kept. The same alert type arrives repeatedly and the same three or four queries answer it every time, so writing them once into the detection's response guidance saves every future analyst the reconstruction. That is the detection-to-triage link in the loop below, and it is nearly free to build while a rule is being written.

One thing all three share is worth stating. Each of them begins by establishing that the data can answer the question, and each of them is worthless without that step. The hunt checks that benign rows come back, the detection is validated by executing the technique, and the triage query is read against what the estate collects on that host. Skip it in any of the three and you get a confident answer about nothing.

The three queries together are the clearest demonstration of why these activities need different skills. The hunt needs judgment about what is normal. The detection needs precision about what is not. The triage query needs speed and a clear question. Somebody strong at one is not automatically strong at the others, and teams that assume otherwise put their best hunter on the queue and wonder why hunting stopped.

The exercise below is the scenario at the top of this section as a document. Read the outputs column before the results column, because the outputs are where the difference between the three activities becomes visible. A results column of six zeros tells you almost nothing on its own; the outputs column tells you which of the six were hunts and which were afternoons.

It is also a document type you will be asked to produce rather than only to read. Most hunting programs report quarterly, most of those reports look roughly like this one, and the difference between a report that gets a program funded and one that gets it questioned is entirely in whether the outputs column has anything in it.

The cadence note underneath that report is the finding nobody wrote down. Nine of eighteen planned hunts ran across three quarters, and the reason recorded both times was analyst time on the alert queue. That is the constraint on almost every hunting program: the methodology is not the bottleneck, the hours are, and hunting competes for exactly the people triage needs.

There are only three honest responses to that and none of them is a better framework. Reduce the triage load so the hours exist, which is what the tuning work in EO0.4 is for. Reduce the hunt count to what the team can actually execute, and report six planned as three planned so the number means something. Or accept that hunting will not happen here and stop putting it in a plan, which is at least truthful.

The worst option is the one most teams take, which is to keep planning six and executing two. It produces a report that looks like a program, a team that feels behind, and a quarterly conversation about commitment rather than about capacity.

The three are not parallel tracks. Each produces the input another needs, and a team that runs them separately gets less than the sum.

The third row is the one that decays fastest and the one EO0.4 was really about. A false positive routed to the tuning layer removes the symptom and leaves the rule broad; the same false positive routed back to the rule makes the rule narrower and permanently better. Both close the ticket, and only one improves anything.

The fourth row is the one that gets skipped under pressure and costs the most. An incident is a technique demonstrated to work on your estate, which makes it the best-evidenced hunting hypothesis you will ever get, and the window in which anybody has the energy to act on it is short. Running the estate-wide version of the same query before the incident is closed is a fifteen-minute job that occasionally finds the second host.

That adjacency is the practical argument for doing both. A team that hunts in one tool and writes detections in another loses the conversion step to friction, because shipping a finding means translating it. Where the query language is shared, a hunt that ends well is a saved query and a form.

Every framework in this field converges on the same loop, and none of them addresses throughput, because a methodology can describe a good hunt and cannot manufacture the senior hours one consumes.

So the constraint is arithmetic rather than intellectual. The people who can hunt well are the people who triage well, there are not many of them, and the queue arrives whether or not the hunt plan does. That is why the tuning work in EO0.4 is not housekeeping: an hour removed from the triage load is an hour that can become a hunt, and it is the only reliable source of hunting time that does not require hiring somebody.

Read the first column against the third. Volume is the number that gets reported and hours is the number that decides whether hunting happens, and they are not the same number. A team that reduces volume by suppressing more can hold hours flat and lose coverage at the same time, which is the trade EO0.4 was warning about seen from the other side.

Measure the hours rather than assuming them. Most ticketing systems will give you time-in-state per alert type, and the top three alert types by total analyst time are almost never the top three by count. That list is where a tuning effort should start, and it is a different list from the one the volume query in EO0.4 produces.

The two lists diverge because cost per alert varies enormously. A thousand alerts that an analyst closes in ninety seconds each cost less than eighty that require pulling a device timeline apart, and only the second kind is worth an engineering afternoon. Volume is the visible number and time is the expensive one.

The other honest lever is scope. A hunt that covers the whole estate over ninety days is expensive; the same hypothesis tested against the servers only, or against a fortnight, is a fraction of the work and answers most of the question. Running four narrow hunts beats planning six wide ones and running one.

The honest response to that ratio is to change the plan, the capacity or the queue, rather than the expectation. Six planned and two executed is arithmetic rather than a commitment problem.

Three activities, three endings, and one hypothesis of your own to take through them.

One last thing about the three, which the rest of the course assumes. They are not seniority levels. Triage is not the junior version of hunting and hunting is not what you graduate to; they are different work with different outputs, and a team needs somebody doing each of them well. The best triage analyst on a team is frequently the person who most improves its detections, because they are the one who knows which alerts are worthless and why.

The next section takes the measures a team is judged on, and asks which of them survive contact with the three activities above.