Measuring Identity Security Posture

Introduction

A feature list says what you deployed. A metric says what changed. This section is four numbers that move when a control works and stay still when it does not. You will finish able to baseline your own tenant, and to say why the score Microsoft gives you is not one of the four.

Metrics are continuous. They tell you how much of your environment is protected, how well the protection is working, and where the gaps are. A feature list that says "MFA enabled" is not the same as a metric that says "62% of sign-ins use phishing-resistant methods, 34% use push notification, and 4% use SMS." The feature is the same in both cases. The security posture is completely different: the 34% using push are vulnerable to AiTM, and the 4% on SMS are vulnerable to both AiTM and SIM swapping.

Metrics also make the investment case. When you deploy phishing-resistant authentication (EI2) and the resistant percentage moves from 3% to 45% in a quarter, that's a measurable improvement you can present to leadership. When the CA evaluation rate goes from 73% to 98%, you've eliminated 25 percentage points of implicit trust surface. These numbers justify continued investment in ways that "we enabled another feature" never will.

Four metrics capture the state of an identity security program. Each maps to a specific area of the course, a specific set of attack patterns, and a specific set of controls. Together, they give you a dashboard that answers the CISO's question with numbers instead of feature names.

Four numbers, each measuring a different one of 0.4's entry points.

A feature list says Conditional Access is deployed, MFA is enabled, PIM is configured. Every line is true, none can be falsified, and the list reads identically on a tenant that would survive an intrusion and one that would not.

A metric can be wrong, and that is the point. "82 per cent of administrator sign-ins used a phishing-resistant method last week" is a claim about behavior that either holds or does not, and it moves when a control starts working.

  1  Phishing-resistant coverage    of sign-ins, not registrations
  2  CA evaluation rate             share NO policy evaluated
  3  Standing privileged accounts   permanent, not eligible
  4  Applications holding consent   with no named owner

The falsifiability point is the whole argument. A feature list is a set of statements that cannot be wrong, and a metric is a statement that can, which is why one of them tells you whether last quarter's work changed anything.

The distinction is worth a test rather than a definition, because "metric" gets applied to both.

A feature list entry cannot be falsified: owning a feature is not a claim about what it does. "9 per cent of sign-ins matched no Conditional Access policy" can be wrong, can move, and somebody can be asked why it is that number.

So the test for a real metric is whether it could get worse without anybody changing a setting. Coverage falls when you hire. Evaluation rate falls when a new application arrives outside policy scope. A number that only moves when you touch a control is a configuration report wearing a percentage.

Each of the four names a population rather than a percentage. A coverage figure resolves to people without a key and an evaluation rate to sign-ins nobody assessed, which is what makes them actionable.

This is the percentage of successful sign-ins that used phishing-resistant authentication methods. FIDO2 security keys, passkeys (device-bound or synced), or certificate-based authentication. It directly measures your exposure to AiTM credential phishing and MFA fatigue, the two most damaging identity attack techniques in production environments today.

The calculation is straightforward: divide the number of sign-ins that used a phishing-resistant method by the total number of successful sign-ins. The result is a single number that captures the most important dimension of your authentication security posture.

// Metric 1: Phishing-resistant authentication coverage
SigninLogs
| where TimeGenerated > ago(30d)
| where ResultType == "0"
| extend AuthMethod = tostring(AuthenticationDetails[0].authenticationMethod)
| extend IsPhishingResistant = AuthMethod in ("FIDO2 security key", "Passkey (Microsoft Authenticator)", "X.509 Certificate")
| summarize
    TotalSignIns = count(),
    PhishingResistant = countif(IsPhishingResistant),
    Phishable = countif(not(IsPhishingResistant))
| extend ResistantPercent = round(100.0 * PhishingResistant / TotalSignIns, 1)

The target is 100% for administrative accounts and as close to 100% as operationally possible for all accounts. In a typical pre-course environment, this number is below 5%, almost nobody uses phishing-resistant methods unless they've been explicitly deployed. The gap between your current percentage and 100% is your exposure to AiTM: every sign-in using push notification, TOTP, or SMS is a sign-in that an AiTM proxy can intercept.

Track this monthly. After EI2 (authentication method deployment) and EI4 (authentication strength in Conditional Access), this number should show measurable improvement with each reporting cycle. The trajectory matters as much as the absolute number: a tenant moving from 3% to 40% in 90 days demonstrates program progress even though the target hasn't been reached.

Audit the report below before going further. Every figure in it is accurate.

A number without a denominator describes whichever population happened to be measured. Per 0.9 that is the whole difficulty of answering the CISO's question, and it is the argument this course returns to in every module.

This is the percentage of successful sign-ins that were evaluated by at least one Conditional Access policy. Sign-ins without CA evaluation bypass your entire Zero Trust enforcement layer, they represent implicit trust that should have been explicitly verified.

// Metric 2: Conditional Access evaluation rate
SigninLogs
| where TimeGenerated > ago(30d)
| where ResultType == "0"
| extend CAPoliciesApplied = array_length(ConditionalAccessPolicies)
| summarize
    TotalSuccessful = count(),
    WithCA = countif(CAPoliciesApplied > 0),
    WithoutCA = countif(CAPoliciesApplied == 0)
| extend EvaluationRate = round(100.0 * WithCA / TotalSuccessful, 1)

The target is 99% or higher. Common causes of gaps include applications excluded from CA scope (legacy line-of-business applications, the Azure Management portal, custom applications that were added to exclusion lists during troubleshooting and never removed), user groups not targeted by any policy (service accounts, guest users, contractor accounts, break-glass accounts that are excluded by design but represent a known and monitored exception), and authentication flows like device code that aren't explicitly blocked in CA policy conditions.

When this metric shows 73%, the 27% gap is exactly where an attacker will target: the path of least resistance through your Zero Trust enforcement. An attacker who discovers that a specific legacy application isn't covered by CA can authenticate to that application's API and potentially access data through it without triggering any of your verification requirements. After EI3, EI4, and EI8, this number should approach 100% for interactive sign-ins. Service principal coverage requires Workload Identities Premium licensing and is addressed in ARC406.

To identify exactly which applications and user groups are creating gaps, extend the query to group by AppDisplayName and UserType where CAPoliciesApplied == 0. The resulting list is your CA gap report: each entry is a specific application or user category that bypasses your Zero Trust enforcement entirely.

The fourth metric is the one that grows without anybody deciding it should.

What to count is sign-ins where the Conditional Access result was "not applied", as a share of all sign-ins.

This is not a denial rate, and confusing the two inverts the reading. A blocked sign-in means a policy worked. A sign-in no policy evaluated is not a decision at all, and it reports as a clean success in every other view you have.

The unevaluated population comes from three places: legacy protocols, which bypass Conditional Access rather than fail it, per ei02; service principals and workload identities, per ei10; and exclusion groups that were temporary.

The honest target is not zero. Break-glass accounts should appear here by design, per ei03. The target is that every entry is one you can name.

This is the metric most tenants have never measured, and the one where the first measurement is usually a surprise.

The distinction in the second block is what makes this metric worth more than a denial count. A denied sign-in is a control working and an unevaluated one is a control that was never consulted, so an estate can have excellent policies and a substantial population they never see.

This measures how many accounts have permanent administrative role assignments versus eligible (PIM-managed) assignments. Standing privilege is the attack surface: an attacker who compromises a permanently assigned Global Administrator gets those privileges immediately, with no additional challenge, no time limit, and no audit trail beyond the initial sign-in.

The math is straightforward. If your organization has 6 permanent Global Administrators and each account has credentials that could be compromised, you have 6 permanent targets for privilege escalation. With PIM, convert 4 of those to eligible assignments and keep 2 as permanent break-glass accounts. The standing exposure drops from 6 to 2. The eligible accounts still have access when they need it, they activate the role with additional MFA, provide a justification, and receive the assignment for a defined time window (typically 1-8 hours). If any of the 4 eligible accounts are compromised while the role is not active, the attacker gets a standard user account.

// Metric 3: Recent admin role assignments (90-day window)
// Each permanent assignment is standing privilege exposure
AuditLogs
| where TimeGenerated > ago(90d)
| where OperationName == "Add member to role"
| extend RoleName = tostring(TargetResources[0].modifiedProperties[1].newValue)
| where RoleName has_any ("Global Administrator", "Security Administrator",
    "Exchange Administrator", "SharePoint Administrator",
    "Privileged Role Administrator")
| extend AssignedUser = tostring(TargetResources[0].userPrincipalName)
| extend AssignedBy = tostring(InitiatedBy.user.userPrincipalName)
| project TimeGenerated, RoleName, AssignedUser, AssignedBy
// Review: each row is a role assignment in the last 90 days
// For current state, use Graph: Get-MgDirectoryRoleMember

This query shows role assignments made in the last 90 days, useful for identifying recent changes and assignments that may have been made during incidents and never revoked. The Midnight Blizzard breach exploited exactly this pattern: emergency access assigned during an incident, never cleaned up, eventually discovered by an attacker. For a complete picture of current standing assignments, the production approach uses the Graph API: Get-MgDirectoryRoleMember for each administrative role, filtered to distinguish permanent assignments from PIM-eligible assignments. EI6 teaches the full privileged access audit.

The target is 2 permanent Global Administrator accounts (break-glass accounts, monitored with Severity 1 alerts on any sign-in) and zero permanent assignments for all other administrative roles. Every other admin role should be PIM-eligible, activated on demand with justification, time-limited, and logged. Track this monthly. Any increase in permanent assignments should trigger an immediate review: was this an emergency that needs a follow-up governance action, or was this an unauthorized change that needs investigation?

This measures the number of applications with high-privilege permissions. Mail.ReadWrite, Files.ReadWrite.All, Directory.ReadWrite.All, full_access_as_app, or equivalent, and specifically how many of those applications lack a defined owner. Applications without owners are ungoverned, nobody reviews their permissions, rotates their credentials, or monitors their activity.

The target is zero high-privilege applications without an accountable owner and a documented justification for the permissions they hold. The Midnight Blizzard breach succeeded because a dormant, over-privileged application existed without lifecycle management. Consent phishing succeeds because users can grant permissions without administrator review. Both attacks exploit permission sprawl.

# Metric 4: Applications with high-privilege permissions and no owner
$apps = Get-MgApplication -All
$highPrivApps = foreach ($app in $apps) {
    $perms = $app.RequiredResourceAccess.ResourceAccess.Id
    $owners = Get-MgApplicationOwner -ApplicationId $app.Id
    if ($perms -and $owners.Count -eq 0) {
        [PSCustomObject]@{
            AppName = $app.DisplayName
            PermissionCount = $perms.Count
            Owners = 0
        }
    }
}
$highPrivApps | Sort-Object PermissionCount -Descending
# Each ownerless app with permissions is an ungoverned attack surface

This PowerShell snippet identifies application registrations that have API permissions but no defined owner. The full permission audit in EI9 goes deeper, matching specific Graph API permission GUIDs to high-privilege operations like Mail.ReadWrite.All (GUID e2a3a72e-...) and Directory.ReadWrite.All (GUID 19dbc75e-...). The metric you track is the count of ownerless, high-privilege applications, trending toward zero over time.

Track this quarterly at minimum. Unlike the sign-in metrics that measure real-time posture, application permission sprawl changes slowly, applications are created, permissions are granted, and they persist indefinitely unless actively managed. A quarterly audit that identifies new ownerless applications and new high-privilege grants is the governance rhythm that keeps this metric under control.

The number is applications holding permissions in the tenant, split by whether a user or an admin consented, and by whether the permission is delegated or application-level.

It only grows, because nothing expires. Per 0.4 a consent grant survives a password reset and a session revocation, so it also survives the departure of the person who granted it and the retirement of the application that needed it.

Two cuts matter: application-level permissions, which act with no user at all, per ei10, and grants nobody can name an owner for, which is 4.8's register problem landing on a different object.

  Mail.ReadWrite            every mailbox in reach
  Files.ReadWrite.All       every file
  Directory.ReadWrite.All   the directory itself
  full_access_as_app        acts as the application, no user

Count grants carrying any of those four, then count how many have an owner somebody can name. The second number is the one that goes in the baseline.

It is a posture metric rather than a backlog because it measures accumulated decisions rather than current activity, and it is the only one of the four that gets worse if the security team does nothing at all. Count the ones with no owner. That is the number.

The last cut is the one that turns a long list into work. An application permission with no nameable owner cannot be defended or removed with confidence, so counting those rather than counting grants gives a figure that shrinks only when somebody makes a decision.

In addition to these four operational metrics, Microsoft provides Secure Score: a percentage-based posture assessment available in the Microsoft 365 Defender portal. Secure Score evaluates your tenant configuration against Microsoft's recommended security settings and provides a normalized score with specific improvement actions ranked by impact.

Secure Score is useful as a starting point and a conversation tool for leadership. It identifies common configuration gaps, accounts without MFA registered, legacy authentication not blocked, PIM not enabled, admin accounts without phishing-resistant authentication, and provides step-by-step remediation guidance. For organizations starting their identity security journey, working through the Secure Score recommendations is a structured way to address the most impactful configuration gaps first.

However, Secure Score measures configuration state, not operational effectiveness. A Conditional Access policy that exists but has scope gaps gets credit in Secure Score but doesn't provide the protection the score implies. A PIM configuration that has eligible roles but nobody uses the activation workflow gets credit even though standing access hasn't actually been reduced. An MFA registration campaign that registered 100% of users but deployed push notification instead of phishing-resistant methods gets full credit even though AiTM exposure hasn't changed.

Entra ID Recommendations, available in the Entra admin center under Identity → Overview → Recommendations, provide more targeted guidance specific to your tenant. These recommendations are context-aware, they analyze your actual directory objects, policy configurations, and authentication patterns to surface issues like "3 applications have unused permissions that should be removed" or "5 users assigned Global Administrator haven't signed in for 60 days." Because they reference specific objects in your tenant, they're more immediately actionable than Secure Score's generic guidance. Check Entra Recommendations monthly alongside your KQL metrics.

Secure Score is a weighted list of recommended configurations, scored against how many you have applied. It is genuinely useful as a prompt, because it surfaces controls you did not know existed.

It is not a measurement of your tenant's behavior. It scores configuration, so it is a feature list with a number attached, and a feature list cannot be wrong.

Where that bites is specific. The score rises when you enable a policy in report-only, per ei08, because the policy exists: it does not distinguish evaluating from enforcing. And it does not know which method satisfied your MFA requirement, per 0.4, which is the entire difference between stopping AiTM and not.

Use it as a prompt rather than a baseline. Track the four metrics beside it, because those move when behavior changes and the score moves when configuration does.

The report-only case is the clearest illustration. A score that rises when a policy is created rather than when it enforces is measuring intent, which is exactly the gap between the four metrics here and the number a vendor dashboard offers.

If you have access to a Sentinel workspace or Advanced Hunting in Microsoft 365 Defender, run these four metrics now. Record the numbers in a document you'll update monthly. These are your baseline: the starting point from which every module in this course produces measurable improvement.

A typical pre-course baseline looks like this. Metric 1 (phishing-resistant coverage) is below 5%, most organizations haven't deployed FIDO2 or passkeys beyond a small pilot group, if at all. Metric 2 (CA evaluation rate) is 60-80%. Conditional Access exists but has gaps in application coverage, user group targeting, and legacy authentication handling. Metric 3 (standing privilege) shows 4-8 permanent Global Administrators instead of the target 2, accounts accumulated from incident responses, onboarding shortcuts, and "temporary" assignments that became permanent. Metric 4 (application permission sprawl) reveals 20-50 applications with high-privilege permissions, many without assigned owners or any record of why those permissions were granted.

These numbers are not unusual. They reflect the state of most M365 environments before systematic identity security work begins. The value of the baseline is not in judging the numbers, it's in having a measured starting point. When you complete EI2 and Metric 1 moves from 3% to 45%, that's demonstrable progress you can report to leadership. When EI6 moves Metric 3 from 6 permanent GAs to 2, that's a quantifiable reduction in attack surface.

If you don't have a lab environment yet, Section 0.10 walks you through the setup. The queries will be waiting when your environment is ready. The metrics become meaningful after EI1, when you learn to read sign-in logs fluently, and become actionable starting in EI2, when you deploy your first authentication method changes. By EI9, all four metrics should be at or near their targets.

Measure before you change anything. The first number is worth more than the second, because it is the only one taken before you knew what you were looking for.

Record the query, not just the result. Per ARC402 9.10 the query text survives and the reasoning does not, and a percentage with no query behind it cannot be recomputed the same way in six months.

Record the window and the denominator. "82 per cent" of what, over how long. Per ARC402 2.8 a percentage whose denominator nobody wrote down is a number that will be recomputed differently and both results defended.

Date it, because these move. An undated baseline is a claim about an unknown moment.

And expect the second metric to be the surprise. Most tenants have never counted unevaluated sign-ins, and the first count is usually larger than anybody expects.

// EI0.9: share of successful sign-ins no CA policy evaluated
SigninLogs
| where TimeGenerated > ago(30d)
| where ResultType == 0
| summarize Total = count(),
    Unevaluated = countif(ConditionalAccessStatus == "notApplied")
| extend Share = round(todouble(Unevaluated) / Total * 100, 1)
// Record the window and the denominator with the number, not after

Four numbers, four queries, one date. That is the baseline, and it fits on a page.

Step 3 is the one that decides whether the baseline is comparable to anything. A percentage without its denominator and window recorded is not a measurement, it is a number, and the second person to compute it will get a different answer and have no way to tell which is right.

Identity security posture is not a feeling, it is the sum of the controls you have enforced and the attack surface each one removes. Toggle the controls below and watch how each, phishing-resistant MFA, Conditional Access, Identity Protection, PIM, and application governance, reduces the surface for credential theft, token replay, consent phishing, and privilege escalation. A default tenant leaves every technique in this course undefended; a complete architecture closes them.

  Phishing-resistant rollout  metric 1, slowly     ei02
  Blocking legacy auth        metric 2, sharply    ei02
  A new CA policy             metric 2, in scope   ei03
  PIM on privileged roles     metric 3, at once    ei06
  Consent restriction         metric 4's GROWTH    ei09

Three of those rows have a caveat that decides whether you will be disappointed.

Metric 1's denominator is people, so it falls again every time you hire, and a program that stops at 60 per cent drifts backwards without anybody doing anything wrong.

Blocking legacy authentication is the only control here that moves a metric by removing traffic rather than adding coverage. The legacy population sits in the unevaluated count until it is blocked, at which point metric 2 improves because those sign-ins stop happening.

And read the last row carefully. Consent restriction moves metric 4's growth rate rather than metric 4. Existing grants stay until somebody removes them, so four of these change a number and one changes a trend.

The distinction in that last row is the one to set expectations with. Restricting consent stops the accumulation and removes nothing already accumulated, so metric 4 keeps its value after the control lands, and the work of reducing it is a separate exercise with its own owner.

The baseline is four queries and an hour, and its value is entirely in being taken before you change anything.

A feature list cannot be wrong, which is why it is not a measurement: owning Conditional Access is not a claim about what it evaluates. The test for a real metric is whether it can get worse without anybody changing a setting, and all four here do: coverage falls when you hire, evaluation rate falls when an application is onboarded, and permission sprawl grows on its own. Each of the four traces back to an identity list somebody can act on, where a single posture score traces to nothing, and Secure Score measures adherence to Microsoft's recommendations rather than your exposure. Take the baseline before you deploy anything, with the denominator and the query beside each number.