Why Identity Evidence Is Ambiguous

Introduction

Identity telemetry has a property that makes it different from most security data: the malicious event and the legitimate one are frequently the same record. You will finish this section able to say why that is structural rather than a logging deficiency, and what a detection has to do about it.

The first instinct is to look for the field that separates them, and there is not one. Same operation, same actor, same role, same permission, and the same kind of object with the same properties.

This is not a gap in Microsoft's logging. It is a consequence of the attack being the legitimate action. An attacker who has compromised an Application Administrator does not need to do anything unusual to establish persistence, because adding a credential to an application is what an Application Administrator is for. The record is identical because the action is identical.

One consequence is that the phrase "suspicious activity" does very little work here. Suspicion implies a departure from expected behavior, and most of these events are expected behavior performed by the wrong person.

That property runs through most of identity security. A sign-in from a country the user has never visited is a business trip or a compromise. A consent grant is a productivity tool or a mailbox exfiltration. A new device registration is a phone upgrade or an attacker satisfying MFA in their own right. In every case the record is complete, accurate and insufficient.

Contrast it with the security data most people learn on. A malware detonation writes a file, spawns a process and reaches out to an address, none of which a normal user does. An exploit attempt is malformed by construction. Endpoint and network telemetry mostly separates into things that happen and things that should not, which is why signature approaches work there.

Identity has almost none of that. The operations an attacker performs are the operations the platform exists to provide, executed through the documented interface with a valid authorization. There is no malformed sign-in.

The rest of this section is about what to do with that, and the answer is not to give up on detection. It is to stop expecting a rule to deliver a verdict and start designing rules that deliver a question worth somebody's time.

If the difference is intent, and intent is not in the log, then the resolution has to come from somewhere that records intent. In practice there are four places, and none of them is a security system.

The four are worth ranking by how conclusive they are, because a detection built on the weakest of them is a detection that produces arguments.

A change record is the strongest and the least available. If the engineer raised a ticket, the ticket resolves the ambiguity completely and permanently, and the detection becomes a join between the audit log and a change management system. id04 builds exactly that rule, and its first finding in most estates is how many legitimate changes have no record at all.

It is also the only one that survives an attacker who has compromised the person who would otherwise vouch for the change, which is worth remembering when the compromised account is an administrator.

The owner confirming is a phone call, and it is the fallback when there is no change process. It resolves the case reliably and does not scale past a handful of alerts a week, which is a constraint on how noisy the rule can be.

The expiry pattern is weaker and available in the data. A rotation replaces a credential nearing expiry, so a new secret appearing while the existing one has eighteen months left is a different shape from one appearing three weeks before expiry. That is inference rather than proof.

The session is the fourth, and it is why id04's detections keep joining back to sign-in data. A credential added from a session that Identity Protection flagged as risky is a different proposition from one added from the engineer's usual laptop.

Notice that three of the four live outside anything a security team controls, and the fourth is a risk score with its own false positive rate. That is the honest position: identity detection depends on the quality of systems the detection engineer does not own and frequently cannot influence.

This changes what a detection rule looks like structurally. A pattern-matching rule says: find events with these properties. An identity detection frequently has to say: find events with these properties that do not have a corresponding record somewhere else.

That second shape is harder in three specific ways and each of them shows up in id04.

The other system has to be queryable. A change management platform with an API is a join; a change process that lives in email is not, and the honest answer in that case is that the rule cannot be built and the alert has to go to a human who checks manually.

The join has to tolerate timing. A credential added at 14:00 and a ticket raised at 16:00 is still a legitimate rotation, so the window is a threshold with all the properties section 0.3 described, and somebody has to choose it.

The other system also has to be trusted, which is a different question from being queryable. A change platform anybody can write to after the fact is not evidence, because an attacker with access to it can create the record that clears their own change.

And the absence of a record has to mean something. In an organization where half of legitimate changes are undocumented, a rule keyed on missing change records fires constantly and correctly, which per section 0.1 is the shape that trains an analyst to ignore it.

There is a fourth difficulty that is organizational rather than technical. A rule of this shape produces a finding whose meaning is "somebody did not follow the process", which lands differently from "somebody attacked us". The first alert most of these rules produce is about a colleague, and the conversation that follows determines whether the rule survives its first month.

That is worth anticipating rather than discovering. A join-based rule needs an agreement about what happens when it fires on a legitimate but undocumented change, and the agreement is easier to reach before the rule exists than afterwards.

That is worth establishing before building rather than after. The practice exercise at the end of this section is a twenty-minute test of whether the joins your future rules will depend on are available at all, and it is worth doing before id04 rather than during it.

The two audit entries from the scenario are below, side by side with everything the tenant knows about each. One of them is the attacker.

Work the columns rather than the rows. The two audit entries are identical, so everything that separates them sits in the material around them: what the existing credential's expiry was, whether the session that made the change had been flagged, whether the account had done anything else unusual that day, and whether anybody raised a ticket.

The point of that exercise is not that it is impossible. It is that everything you used to reach an answer came from outside the two records, and that a rule seeing only the audit log would have to treat them identically or flag them both. An analyst can do what the rule cannot, which is go and look at four other things, and that asymmetry is why some identity detections are deliberately designed to raise a question rather than assert a finding. Designing for that explicitly, rather than pretending the rule is conclusive and then being surprised when analysts close its output, is most of what separates a detection estate people trust from one they route around.

**The eleven-day gap matters as well. Two credential additions in a fortnight against the same application is itself a pattern, and neither entry contains it: seeing it requires looking at the application's credential history rather than at either event.

It is also worth noticing what you did not need. Neither entry had to be matched against a threat intelligence feed, a signature, or a known-bad address, and no amount of that would have helped, because the attacker was using a legitimate administrative account through the documented interface.

Flagging both is a defensible design.** A tenant with three credential additions a month can afford a rule that fires on all of them and asks a human, and that is frequently the right answer for high-consequence low-volume operations. It stops being right at a hundred a month, which is where the join becomes necessary rather than merely better.

There is an escape from the ambiguity and it is worth knowing because it changes what to build, and because it is the single most useful reframing in this course. Some questions are about events, which are ambiguous, and some are about current state, which frequently is not.

"Was this credential addition legitimate" is an event question and it is hard. "How many application credentials in this tenant are older than two years, held by applications with Graph write permissions, and owned by nobody" is a state question, and it has an exact answer that nobody can argue with.

Connect-MgGraph -Scopes "Application.Read.All"

Get-MgApplication -All |
  ForEach-Object {
    $app = $_
    $app.PasswordCredentials | ForEach-Object {
      [pscustomobject]@{
        App     = $app.DisplayName
        Added   = $_.StartDateTime
        Expires = $_.EndDateTime
        AgeDays = [int]((Get-Date) - $_.StartDateTime).TotalDays
      }
    }
  } | Sort-Object AgeDays -Descending | Select-Object -First 5

# App                      Added       Expires     AgeDays
# ---                      -----       -------     -------
# sp-northgate-deploy      2021-03-02  2026-03-02     1819
# LegacyReportingBridge    2021-11-14  2027-11-14     1562
# ...

The difference is worth naming precisely, because it changes what you build rather than merely how you feel about it. An event question asks whether a thing that happened was legitimate, and legitimacy is a property of intent, which is not recorded. A state question asks whether a thing that currently exists should, and that is a property of policy, which is written down.

State questions also have the advantage of being answerable at any time, by anybody, without waiting, and of being auditable in a way an event rule is not: anybody can rerun the query and get the same answer. An event rule tells you about the moment it fires and is silent about the five years before it was written. The query below finds credentials created long before anybody was watching, which no event rule can do because the events are outside retention or predate the workspace entirely.

Nothing in that output is ambiguous. That output requires no interpretation and no join. It is the same finding the event-based rule was reaching for, arrived at from the other direction, and per id01 it is frequently the faster route to a real problem in a tenant you have just inherited.

The first habit is to ask, of any detection idea, what would make it unambiguous, and to notice when the answer is outside the identity estate. That question is the difference between a rule that produces findings and one that produces questions.

The second is to prefer a state question where one exists. id01 opens with an inventory rather than a detection for exactly this reason: counting the credentials that should not exist is unarguable, and watching for the moment somebody creates one is a rule with a threshold, a join and a false positive rate.

There is a third habit, less obvious, which is to notice when the ambiguity is the finding. A tenant where nobody can tell whether a credential addition was legitimate does not have a detection problem, it has a change management problem, and the detection engineer is frequently the first person in a position to say so with evidence.

That is an uncomfortable finding to deliver and it is often the most valuable thing a detection program produces in its first year. It also has a straightforward fix that costs nothing technically and a great deal politically, which is why it usually needs the evidence a detection estate can supply.

Neither habit removes the need for event detection. Some things only ever appear as events, a token replay most obviously, and id04 spends most of its length there. But a great deal of what a new detection engineer wants to write as a rule is better answered by a query run once a month, and knowing which is which saves building the wrong thing.

Twenty minutes, and the answer determines what kind of detections are possible where you work.

The next section turns to the population this ambiguity hurts most, which is the identities with no person attached and therefore nobody to ask.