See It Done: The Endpoint Assessment

Introduction

This is the assessment from the rest of this module, done once, on one estate, in the order the readings were actually taken.

It is written as a walkthrough rather than as a method because the interesting part is not the technique. Every reading below appears earlier in this module and none of them is difficult. What is worth watching is where the answers contradicted each other, which number turned out to be wrong twice, and the one request that had to be refused.

You will finish able to run the same sequence on your own estate and, more usefully, to recognize the three points at which it goes wrong for reasons that have nothing to do with security.

The request was for a single figure. That is a reasonable thing to ask for and it has no honest answer, so the first decision in this assessment was made before any reading was taken: produce the profile, and produce alongside it the one-sentence version somebody can put in a response document, derived from the profile rather than instead of it.

Response document, security section, as submitted

  Endpoint telemetry covers 712 of 865 devices, with the
  difference identified and being closed. Preventive controls
  are partially enforcing, with a promotion program under way.
  Evidence retention is our known gap and has a dated plan
  against it.

  [account team draft, replaced]
  The estate is fully covered by enterprise endpoint protection.

Both versions are true. The second would not have survived the client's follow-up question, and the first names a gap before anybody has to find it, which is the only version that improves your position by being read carefully.

That distinction sounds like presentation and is not. A single number cannot be checked, so a client who receives one has to trust it, and a client who is going to audit you later will discover which parts of it were generous. A profile with five numbers and the reading behind each is slower to read and survives the follow-up, which is the only property that matters in a contract renewal.

The account team's objection was that a profile looks worse than a score. It does, and it is worth being direct that this is true rather than pretending otherwise: five numbers with two of them low reads as a weaker position than a single respectable average. The argument that carried it was that the client's own security team would ask for the evidence, and a score with no derivation would produce exactly the meeting nobody wanted.

There is a second argument worth having ready, because the first only works with people who expect to be audited. A single score cannot be improved deliberately. Told the estate is at 1.6, nobody can name what would move it, so the number produces a feeling rather than a plan, and next quarter's version is produced the same way and compared to the last as though the comparison meant something.

The profile was also cheaper to produce than the score, which surprised everybody including the person who made it. Deriving one number requires deciding how to weight five layers against each other, and that argument has no resolution and consumes a day. Reporting five requires no weighting at all.

The first reading was the device count, because every figure that follows is a fraction of it. Four systems hold a number and all four disagree, which is expected, and the size of the disagreement was not.

Northgate, counted four ways, first pass

  asset register            902
  Entra ID device objects   871      -31
  Intune enrolled           843      -28
  Defender reporting        779      -64

  gap, register to Defender 123      13.7 per cent of the estate

First conclusion: 123 machines are unwatched.
That conclusion was wrong.

Thirteen per cent unwatched is a serious finding and it went into the first draft as one. It was wrong, and the reason it was wrong is the most useful thing in this whole assessment, because it is a property of how the tooling works rather than a mistake anybody made.

The instinct at this point is to start chasing the 123 individually, which is a week of work and finds nothing. The next section is what should be done instead, and it takes an afternoon.

It is worth naming why the four systems disagree rather than treating it as an anomaly, because the disagreement is structural. The register is updated when something is bought and rarely when something is disposed of, so it drifts upward permanently. Entra holds an object for anything that ever joined the directory, enrolled or not. Intune holds what enrolled and is still enrolled. Defender holds what is reporting now. Each answers a different question correctly and none was designed to answer yours.

That also tells you which gaps to expect. Register against Entra is disposals. Entra against Intune is devices that joined and never enrolled. Intune against Defender is the one that matters, because those are managed machines producing no telemetry, and it is the only one of the three that is a security finding rather than a records exercise.

Take the counts on the same day, which sounds obvious and was not done first time. Two of the four figures at Northgate were a week apart because they came from different exports, and in a fleet of 865 a week is enough movement to produce a gap that has no cause. Redo any count you cannot date.

When a device is reinstalled or renamed, a new entity is created for it and the old one remains, moving to inactive after it stops reporting. Off-boarded devices behave the same way: they stay in the list and go inactive after seven days. So an estate that has done any rebuilding at all accumulates entries for machines that no longer exist, and every one of them sits in the denominator making the coverage figure worse.

// Separate the ghosts from the genuinely unwatched
DeviceInfo
| summarize arg_max(Timestamp, OnboardingStatus, DeviceName) by DeviceId
| extend QuietDays = datetime_diff('day', now(), Timestamp)
| summarize Entities = count() by DeviceName
| where Entities > 1

That query answers a narrow question: which device names hold more than one entity. Every row is a machine that was rebuilt or renamed, and the older entity is a record rather than a device. At Northgate it returned 41 names, which accounts for a third of the gap immediately and costs one query rather than a week of tickets.

Twenty-three is a much better finding than 123 because it is actionable. Fourteen at one site is a conversation with one team, six contractor laptops is a policy question that was already open, and three unexplained is a small enough number that somebody can go and look at each one. A hundred and twenty-three would have produced a program; twenty-three produced three tickets.

The three unexplained are worth a sentence because of how they resolved. Two were machines rebuilt and re-enrolled under a name differing by one character, which the duplicate query missed because it matches on the name. The third was genuinely a device with a failed sensor install, and it is the only true finding in the original 123. One machine, from a figure that would have justified a project.

None of which is an argument for ignoring the gap. It is an argument for spending the afternoon before spending the quarter, and for stating in the pack how the number was reconciled, because the next person to run this will otherwise start from 123 again.

The lesson to carry is that the first count is a starting position rather than a result. Reconcile before reporting, because a coverage figure quoted from an unreconciled register is wrong in a direction that makes you look worse and, when somebody eventually corrects it, makes the rest of your assessment look careless.

With the denominator settled, the next question was whether the 779 reporting devices are actually protected, and they are not all the same. Sensor health is not a binary: a device can be active, or misconfigured, or inactive, and misconfigured splits further into impaired communications, where the sensor runs but cannot reach the service, and no sensor data, where it communicates but reports only partially.

The breakdown that came back split the reporting population four ways, and only the first row is a device where a reading can be trusted without qualification.

Thirty-one impaired is a network finding rather than a security one, and it turned out to be a proxy configuration at two sites, which is the single most common cause. The six reporting partial data are the more interesting group: they appear in the inventory, they contribute to coverage, and a hunting query returns incomplete results for them without saying so.

That gives the real denominator for everything after this point. Not 902, not 865, not 779, but 712 devices where a reading can be trusted, and every percentage in the pack was computed against that and stated with the figure beside it.

The thirty inactive devices needed a decision rather than a fix, and that sort of decision quietly does not get made. Some are ghosts already accounted for, some are machines genuinely off for a season, which happens in an organization with field staff, and some were off-boarded and never removed. Each needs a different action, and the useful step was routing the list to whoever owns device lifecycle rather than holding it as a security backlog.

One more property matters for anybody writing a hunting query later. Those six devices reporting partial data return partial results, and a query does not indicate that it received an incomplete answer for some hosts. A hunt that comes back clean across the estate is clean across the devices reporting fully, and the difference is invisible unless somebody checked this table first.

The policy export said eighteen attack surface reduction rules were configured. The devices said something more complicated, and the sample was twenty machines chosen across sites, images and hardware generations rather than at random.

PS C:\> $p = Get-MpPreference
PS C:\> $p.AttackSurfaceReductionRules_Actions | Group-Object |
            Select-Object Name, Count

Name Count
---- -----
1        3
2       13
0        1

PS C:\> (Get-MpComputerStatus).AMRunningMode

Normal

Seventeen of eighteen present on the workstations, three enforcing, thirteen auditing and one explicitly disabled. The disabled one is worth its own line in a report, because a rule set to zero is a decision somebody took and a rule absent from the list was never configured, and those two look identical in a policy summary.

The servers were different and this is where the assessment found the thing worth the whole exercise. Northgate runs a third-party antivirus product on its file and application servers, so Defender there is not the active engine.

PS C:\> (Get-MpComputerStatus).AMRunningMode

Passive

PS C:\> $i = [array]::IndexOf($p.AttackSurfaceReductionRules_Ids,
            "56a863a9-875e-4185-98a7-b882c64b5ce5")
PS C:\> $p.AttackSurfaceReductionRules_Actions[$i]

1

The vulnerable driver rule reads block on those servers and cannot enforce, because attack surface reduction is delivered through Defender Antivirus and requires it to be the active engine. Every compliance view showed the rule as enforcing across the estate. On the twelve machines where kernel access matters most, it was doing nothing, and nothing anywhere reported that.

It is worth being precise about why this is not the third-party product's fault, because the first reaction in the room was to blame it. The other product is doing its job; attack surface reduction is a Defender Antivirus feature and needs that engine active to act. The finding is not that the wrong antivirus is installed, it is that a coverage claim was made across a population where the mechanism behind it cannot operate, and nobody had asked what the rule requires in order to work.

That question generalizes and became a line in the pack. For each control claimed, what has to be true for it to act. Cloud protection needs reachability, block at first sight needs sample submission, attack surface reduction needs the engine active. Each is a single reading and none appears as a warning beside the setting.

Four of the five layers can be read from a device. Readiness partly can, and the part that matters cannot, which took a while to accept during this assessment rather than working around.

PS C:\> $pol = "HKLM:\SOFTWARE\Policies\Microsoft\Windows\PowerShell"
PS C:\> $sb  = "$pol\ScriptBlockLogging"
PS C:\> (Get-ItemProperty $sb -EA SilentlyContinue).EnableScriptBlockLogging

PS C:\> $a = "HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System\Audit"
PS C:\> (gp $a -EA 0).ProcessCreationIncludeCmdLine_Enabled

1

Command line capture on, script block logging unset. That is the partial state from ES0.4 and it is the commonest one: process events carry what was run and a script executed inside one of those processes leaves only whatever the engine flagged by itself.

The part with no device reading is whether the evidence has ever been used. An estate can have every logging setting correct and have never once tried to answer a question with it, and that is not a configuration state, it is an organizational one. The question asked instead was what the last incident produced, and the answer was the scenario at the top of ES0.3: detected within the hour, contained the same afternoon, no conclusions.

That answer is worth more than any configuration reading for this layer, and it is available in every organization that has had an incident, which is all of them. Read the last post-incident report and count the questions it could not answer. Where it says the scope could not be established, or persistence could not be ruled out, or it is unknown which files were accessed, those are readiness findings with a date and a business consequence already attached, which is a considerably better artifact than a registry value.

Where there has been no incident to read, the substitute is to ask the question in advance and write down the answer. Given a compromised laptop this afternoon, what could you establish about the last thirty days on it. Twenty minutes with the people who would actually do it produces a list of things nobody can answer, which is the same finding arriving before the incident rather than after.

The readiness bar is marked because it is the only one where the level describes a decision rather than a state, and where being at one on the day of an incident cannot be corrected for that incident afterwards.

The assessment was then reviewed by somebody who had not taken the readings, which is the step most assessments skip and the one that catches the errors that survive self-checking.

Here is the same data, presented as it was handed over. Read it as the reviewer did: not to check the arithmetic, but to find what a client's security team would ask about first.

Classify it before deciding what to do about it, because an assessment with an error and an assessment with an omission go back to different people. An error is the assessor's to fix. An omission that everybody's template shares is a finding about the template, and the second is more valuable because it does not recur.

What the review actually surfaced at Northgate was the denominator. Two figures in the draft were computed against 865 and three against 779, nobody had noticed, and every one of them was individually defensible. That is the failure mode of a document assembled over several days by one person, and it is why the final pack states the denominator beside every percentage rather than in a footnote.

The reviewer's second finding was smaller and more embarrassing. Four of the five scores had been written from the readings and one from memory of what the readings said, and that one was a level too generous. Reading the raw output back against every score before submitting is a ten-minute check and it caught something that would have been challenged.

Both findings share a cause worth naming. Neither is a security failure and neither is caught by knowing more about endpoints. They are the ordinary failure modes of assembling a document, and the counter is a reviewer holding the raw readings rather than a more careful author.

Pick the reviewer for the right property as well. The useful reviewer is somebody who understands the estate and did not take the readings, which usually means a peer rather than a manager. A manager reviews the conclusions, which are the part least likely to be wrong, and a peer reviews the derivation, which is where both of these findings lived.

The deliverable was three pages and the shape is worth copying. Five scores with one line of evidence each, a named lowest layer with a concrete next action, and one paragraph the account team could put in the response, derived from the five rather than written separately.

Nothing in that paragraph is flattering and nothing in it is false, which is the property that made it usable. The account team's version had said the estate was fully covered by enterprise endpoint protection, which was also true and would not have survived the client's follow-up question.

The evidence line beside each score is the part that does the work, and it is one sentence by design. A score with a paragraph attached is not read, and a score with nothing attached is not believed, so one line naming the specific reading is the only version that survives being skimmed by somebody who will come back to it if a number looks wrong.

Worth noticing what the pack does not contain. No maturity level for the estate, no percentage without a denominator beside it, no adjective, and no timeline. The lowest layer carries a next action rather than a date, because a date in a client response becomes a commitment somebody else has to meet and the work had not been scoped or approved.

The three pages also took longer to agree internally than to produce, and that is the honest shape of this work. The readings were an afternoon. The conversation about whether an honest profile could go to a client took considerably longer, and it was won on the argument that the client's own security team would derive the same numbers within a day of being given access.

Keep the working document as well as the pack, and keep it with the raw output in it. The pack is what went out and the working document is what lets somebody re-derive it next year, or restate this year's position against criteria that have since changed. Every assessment that has to be rebuilt from nothing is rebuilt from nothing because this file was not kept.

The assessment took a few days and three of those were spent on things that were not security work at all. That is normal and worth planning for rather than being surprised by.

The fourth failure is also the cheapest to prevent, because every denominator you might use comes out of one query.

// Every denominator this assessment could have used, in one result
DeviceInfo
| summarize arg_max(Timestamp, OnboardingStatus, SensorHealthState) by DeviceId
| summarize Onboarded = count(),
            Reporting = countif(Timestamp > ago(7d)),
            Healthy = countif(SensorHealthState == "Active")

Run this once, at the start, and quote whichever figure you use with the other two beside it. The reviewer's finding at Northgate was that two figures in the draft used one denominator and three used another, and nobody had noticed because each was individually correct.

The fourth is the one that would have done the most damage and the least visibly. An assessment that is wrong is corrected. An assessment whose figures are individually right and mutually inconsistent is not corrected, it is discounted, and the discount applies to everything the team produces afterwards.

The second, reading the policy rather than the devices, is avoided by a different habit and it is the harder of the two to adopt. A policy export can be produced from a chair and a device reading needs access to machines, so under time pressure the export is what gets used, and every figure derived from it describes what was asked for rather than what is in force. Every significant finding in this assessment came from the difference between those two, which is the argument for building the device sample into the method rather than treating it as verification at the end.

That is also the practical answer to how long an assessment like this takes. Long enough to read twenty devices, which is an afternoon on a well-managed estate and rather longer on one where getting access to twenty machines across four sites is itself the finding.

One last thing this assessment produced that was not asked for and turned out to matter most. The reconciliation, the sensor health breakdown and the twenty-device sample are all repeatable, so the next run is a comparison rather than a fresh exercise. The first time this is done it is an assessment. Done a second time against the same method, it becomes the only evidence anybody has about whether the estate is improving, and that is worth more to the organization than the document that was requested.

That completes the module. The next module takes the lowest layer on your profile and starts closing it, beginning with the architecture the controls in this course actually act on.