0.1 Northgate's Estate

Introduction

Every detection in this course is written for one company, Northgate Engineering, a fictional manufacturer based in Manchester whose month of activity is the course's corpus. A rule is only as good as its fit to the estate it runs in.

A threshold that suits a hundred accounts fails at ten thousand, a rule for a Windows event is silent where the event isn't collected, and an exclusion for an administrator only makes sense if you know who the administrators are.

So before any rule, this lesson reads the estate itself: who works there, which machines and clouds it runs, what the logs from each look like, and who holds the accounts that matter most. Nothing here is described from a diagram. Each fact comes from a query against the same data every later lesson uses, so the map you finish with is one you can check.

The figure is the shape of the estate. The sections fill it in, one source at a time.

You have joined Northgate's information security team as its detection engineer. The team is small, and most of the people you'll work with sit in two departments, Information Security and Information Technology. Their accounts, titles and managers are in IdentityInfo, the table Sentinel builds from Entra ID and Active Directory, which keeps a row each time an account's details change.

Ask it for the latest row per account in those two departments:

IdentityInfo
| summarize arg_max(TimeGenerated, *) by AccountUPN
| where Department in ("Information Security", "Information Technology")
    and (JobTitle has_any ("CISO", "Director", "Architect", "SOC", "Administrator"))
| project AccountUPN, AccountDisplayName, JobTitle, Department, Manager
| order by Department asc, JobTitle asc

In Splunk the same directory is a lookup, ne_identities, which holds one row per account, so there is no history to collapse:

| inputlookup ne_identities
| search department IN ("Information Security", "Information Technology")
    (title="*CISO*" OR title="*Director*" OR title="*Architect*"
    OR title="*SOC*" OR title="*Administrator*")
| table upn display_name title department manager
| sort department title

Both return the same eight people, with their titles and the managers they report to:

account        name             title                   department
r.okafor       Rachel Okafor    CISO                    Information Security
t.ashworth     Tom Ashworth     L1 SOC Analyst          Information Security
p.sharma       Priya Sharma     L1 SOC Analyst          Information Security
m.webb         Marcus Webb      Security Architect      Information Security
p.greaves      Phil Greaves     IT Director             Information Technology
s.mitchell     Steve Mitchell   Systems Administrator   Information Technology
t.quinlan      Tamsin Quinlan   Systems Administrator   Information Technology
k.dhillon      Kieran Dhillon   Systems Administrator   Information Technology

These are the people the course's lessons work for and with. Rachel Okafor, the CISO, decides which threats the detection program covers first and approves the rules that go to production.

Marcus Webb, a security architect who reports to her, writes and reviews rules beside you, and his changes are the ones you'll read in the repository in Module 17. The two L1 SOC analysts, Priya Sharma and Tom Ashworth, triage the alerts your rules raise, so every rule's investigation guide is written for them.

On the IT side, Phil Greaves is the IT Director and Rachel Okafor's manager. His three systems administrators, Steve Mitchell, Tamsin Quinlan and Kieran Dhillon, run the servers, the domain and the configuration tooling. Their work matters to a detection engineer more than anyone else's in the company, because administration and attack use the same tools.

When a rule fires on a server, the first question is often whether it was one of them.

Two columns in these rows do work later. Manager is the approval path: an alert about Rachel Okafor's own account goes to Phil Greaves, not to her, and an investigation guide that says whom to call reads this column rather than a list someone has to keep current.

JobTitle is something a rule can read, which is useful and fragile at once, because a title changes when a person moves role and a rule keyed on it changes with it, silently.

A rule that counts accounts needs to know how many there are to count. Ten failed sign-ins across ten accounts is a password spray at a company of eighty-five people and a quiet Monday at a company of eighty thousand. Count the accounts in each department, from the latest row per account:

IdentityInfo
| summarize arg_max(TimeGenerated, Department) by AccountUPN
| summarize Accounts = count() by Department
| order by Accounts desc

The Splunk lookup gives the same counts directly, because it holds one row per account:

| inputlookup ne_identities
| stats dc(upn) as accounts by department
| sort - accounts

The result is a small company, with no department larger than fifteen people:

department               accounts
Engineering                    14
Finance                        13
Sales                          12
Information Technology         11
Operations                     11
Marketing                      11
Risk & Compliance               9
Information Security            4

Northgate has 85 accounts in eight departments, every one a member account in Entra ID. Engineering and Finance are the largest, and they hold what an attacker usually wants: drawings and designs on one side, payments and supplier banking details on the other.

Lesson 19.1's bulk download case is a Finance account taking a library's documents in a single night, and it stands out partly because Finance is small enough that its usual day is easy to describe.

The directory also changes. IdentityInfo holds 171 rows for these 85 accounts, because a new row is written whenever an account's details change, and one account changed department during the month.

A rule that enriches an alert with a department should read the row that was current when the alert happened, not the latest one, which lesson 4.3 builds. For a count of the company today, the latest row is right.

Size changes how every later rule is tuned. At this scale a threshold of ten accounts is a large share of the company, a single unusual user is visible by eye, and peer groups are small, which is why lesson 8.4 measures a peer group before trusting it.

The same rules on a larger estate would need different numbers, and the course says so wherever a number depends on Northgate's size. The smallest department, Information Security, has four accounts, and every one of them is on the team in section 1.

Northgate's Windows machines are onboarded to Microsoft Defender for Endpoint, and DeviceInfo holds a row per machine each time Defender reports its state. Take the latest row per machine and count them by type:

DeviceInfo
| summarize arg_max(Timestamp, *) by DeviceName
| summarize Devices = count() by DeviceType, OSPlatform, MachineGroup

The Splunk side reads the device lookup, ne_devices, built from the same Defender inventory:

| inputlookup ne_devices
| stats dc(device) as devices by device_type

Northgate has 71 Windows machines: 65 Windows 11 laptops, one for most of the people, and six Windows Server 2022 machines. The latest row per machine matters here for the same reason it did for accounts: DeviceInfo is a snapshot Defender writes repeatedly, 2,730 rows in the month, so counting rows instead of machines would describe a company forty times its size.

Laptops are named after their users, so ne-okafor-lt is Rachel Okafor's, and that convention is how many lessons find the person behind an event on a machine.

Not every machine matters equally, and Northgate records which matter most in a Sentinel watchlist, Watchlist_AssetCriticality, which Splunk holds as the ne_assets lookup. Count it by tier:

Watchlist_AssetCriticality
| summarize Hosts = count(), Functions = make_set(Function) by Tier
| order by Tier asc

The Splunk form reads the same watchlist from the ne_assets lookup:

| inputlookup ne_assets
| stats count as hosts values(function) as functions by tier
| sort tier

Each tier, with how many hosts it holds and the functions they serve:

Tier 0, 5 hosts
  Backup server
  Domain controller
  Perimeter firewall
  Remote access VPN
Tier 1, 10 hosts
  Application server
  File server (engineering drawings)
  Log collector (syslog and CEF to Sentinel)
  Monitoring
  PostgreSQL (Bristol replica)
  PostgreSQL (manufacturing)
  SQL Server (ERP)
Tier 2, 2 hosts
  Public web server

Tier 0 holds the five hosts that control everything else: the two domain controllers, the backup server, and the two devices every connection into the company passes through, the perimeter firewall and the remote access VPN. An attacker who controls any of them controls the domain, the way in or the means of recovery.

Tier 1 is the business's servers, Windows and Linux, including the log collector that forwards syslog and firewall records to Sentinel. Tier 2 is the two public web servers. The laptops aren't in the watchlist; they are in DeviceInfo and untiered, because none of them administers anything.

Tiers decide severity. The same command on a laptop and on a domain controller is not the same alert, and lesson 4.3 joins this watchlist to an alert, adding the tier and owner of what was reached, so severity can rise on Tier 0 without a second rule for it.

Northgate's Linux servers don't run Defender. They send syslog through a collector into the Syslog table, and so does one appliance, the VPN gateway, because it writes its own events in the same format. Count events and distinct programs per host:

Syslog
| summarize Events = count(),
    Programs = dcountif(ProcessName, isnotempty(ProcessName))
    by Computer
| order by Events desc

In Splunk the same events are sourcetype syslog, the servers in index linux and the gateway in index network, with the host in host and the program in process:

(index=linux OR index=network) sourcetype=syslog
| stats count as events dc(process) as programs by host
| sort - events

Nine hosts, and the watchlist names what each one does:

host                 events  programs  function
WEB-NGE-MCR-01         5728         5  Public web server
WEB-NGE-MCR-02         3335         6  Public web server
SRV-NGE-MCR-APP02      1231         5  Application server
SRV-NGE-MCR-APP01      1217         5  Application server
NE-FGT-VPN01           1140         1  Remote access VPN
SRV-NGE-MCR-DB01        954         5  PostgreSQL (manufacturing)
SRV-NGE-MCR-DB02        928         4  PostgreSQL (manufacturing)
SRV-NGE-BRS-DB01        916         4  PostgreSQL (Bristol replica)
SRV-NGE-MCR-MON01       608         4  Monitoring

The eight Linux servers run the business's own applications: two web servers in front of an application tier, PostgreSQL for manufacturing with a replica in the Bristol office, and a monitoring server. The busiest is the first web server, because a web server logs every request it handles.

NE-FGT-VPN01 writes its events in CEF, and its records are how a person working from home appears in the corpus before they reach anything else. The query counts only named programs because 509 of the gateway's rows carry none: from 3 March the collector agent upgraded itself and stopped filling the field, the schema drift lesson 4.5 catches before it silences a rule.

Two consequences follow for detection. A Linux rule reads text, not structured fields. A failed SSH password is a line of sshd output, and every Linux rule in Module 11 starts by extracting the field it needs from SyslogMessage.

And what a Linux host records depends on what it's configured to record. sshd and sudo log by default, but a process starting or a file changing is logged only when auditd has a rule for it, which is why lesson 11.5 writes the audit rules the other Linux detections depend on.

The VPN gateway shares the table and not the meaning. Its events are tunnels and authentications, not commands, and a rule for one shouldn't match the other, which is the reason the Linux rules in this course always name the hosts or programs they read.

Every person signs in through Entra ID, and so do the applications that act on the company's behalf. The sign-ins are split across three tables: interactive sign-ins, the token refreshes and background sign-ins an application makes for a signed-in user, and service principals signing in as themselves. Count each:

union withsource = Source SigninLogs, AADNonInteractiveUserSignInLogs,
    AADServicePrincipalSignInLogs
| extend Who = coalesce(UserPrincipalName, ServicePrincipalName)
| summarize Rows = count(), Identities = dcount(Who), Apps = dcount(AppDisplayName)
    by Source

Splunk keeps all three in one sourcetype, azure:monitor:aad, with a category field saying which is which:

index=azuread sourcetype="azure:monitor:aad" category=*SignInLogs
| eval who=coalesce(user, sp_name)
| stats count as rows dc(who) as identities dc(app) as apps by category

The three tables, side by side, with how many identities each one records:

table                              rows  identities  apps
SigninLogs                         2591          85     6
AADNonInteractiveUserSignInLogs   13111          85     5
AADServicePrincipalSignInLogs      2008           4     0

The non-interactive table is the largest by far, because one interactive sign-in is followed by many refreshes, and a stolen session usually shows up there rather than in an interactive sign-in. The six applications people sign in to are Azure Active Directory PowerShell, Azure Portal, Microsoft Office, Microsoft Teams, Office 365 SharePoint Online and OfficeHome.

The four service principals are Northgate's own automation: ne-backup-graph-reader, ne-hr-sync, ne-servicedesk-connector and ne-terraform-pipeline. Each signs in from where it runs, on a schedule, which makes a service principal one of the most predictable identities in the estate and one of the most valuable to an attacker who adds a secret to it.

Every one of them signs in on every day of the month, from one or two addresses, so a sign-in from anywhere else is unusual on its face, which lesson 12.4 builds a rule on.

Northgate also runs one Azure subscription and one AWS account. Their control-plane logs, with Azure OpenAI's resource logs, are three more tables:

union
    (AzureActivity
    | summarize Rows = count(), Detail = dcount(ResourceProviderValue)
    | extend Cloud = "Azure, control plane", Unit = "resource types"),
    (AzureDiagnostics
    | summarize Rows = count(), Detail = dcount(Resource)
    | extend Cloud = "Azure OpenAI, resource logs", Unit = "resources"),
    (AWSCloudTrail
    | summarize Rows = count(), Detail = dcount(EventSource)
    | extend Cloud = "AWS, CloudTrail", Unit = "services")
| project Cloud, Rows, Detail, Unit

How much of each cloud the corpus shows, and across how many services:

source                         rows  breadth
Azure, control plane             302  7 resource types
Azure OpenAI, resource logs     9166  1 resources
AWS, CloudTrail                 2069  9 services

The Azure subscription is mostly storage, which makes up 242 of its 302 records, and holds one Azure OpenAI resource, OAI-NE-SUPPORT, whose own logs record every request to its model; lesson 19.4 reads both. The AWS account works in two Regions, eu-west-2 for nearly everything, and its trail records EC2, S3 and IAM most often.

The handful of records in us-east-1 are IAM, STS and console sign-ins, not activity in a second Region: AWS records events from global services such as IAM and STS in us-east-1, so a rule that filters CloudTrail to the account's working Region drops every IAM change, the events lesson 12.6 depends on.

Cloud logs record intent as API calls, so a cloud rule reads an operation name and its parameters where a Windows rule reads a process.

Four more sources carry most of what an attack looks like on its way in and its way out: messages arriving, documents read, connections leaving, and names looked up. One query counts each and how widely it reaches:

union
    (EmailEvents
    | summarize Rows = count(), Breadth = dcount(RecipientEmailAddress)
    | extend Source = "EmailEvents", Of = "recipients"),
    (OfficeActivity
    | where OfficeWorkload == "SharePoint"
    | extend Library = extract(@"(sites/[^/]+/[^/]+)", 1, OfficeObjectId)
    | summarize Rows = count(), Breadth = dcount(Library)
    | extend Source = "OfficeActivity, SharePoint", Of = "libraries"),
    (CommonSecurityLog
    | summarize Rows = count(), Breadth = dcount(SourceIP)
    | extend Source = "CommonSecurityLog", Of = "source addresses"),
    (ASimDnsActivityLogs
    | summarize Rows = count(), Breadth = dcount(Dvc)
    | extend Source = "ASimDnsActivityLogs", Of = "DNS servers")
| project Source, Rows, Breadth, Of

The mail, counted the same way in Splunk from Defender's email events:

index=defender category="AdvancedHunting-EmailEvents"
| stats count as messages dc(recipient) as recipients
    dc(src_user_domain) as sender_domains

All four, in rows and in how widely each one reaches:

source                        rows  breadth
EmailEvents                    9375  91 recipients
OfficeActivity, SharePoint     8567  9 libraries
CommonSecurityLog             14502  99 source addresses
ASimDnsActivityLogs           13141  2 DNS servers

EmailEvents records every message Defender for Office 365 processed, 9,375 of them, to 91 recipients from 247 sender domains, which is where a phishing case starts.

Most are inbound, 8,407 against 968 outbound, and the inbound ones are where a rule looks for a message from a sender Northgate hasn't heard from before. OfficeActivity holds SharePoint file access across nine document libraries and 39 Exchange mailbox events, among them the inbox rules a mailbox attack leaves behind.

The firewall is a Palo Alto Networks PA-3220, and its traffic records arrive in CommonSecurityLog in the CEF format, which Module 4 parses and normalizes. DNS queries come from the two domain controllers, NE-DC01 and NE-DC02, which are also Northgate's DNS servers, through the normalized ASimDnsActivityLogs table.

That table is where a beacon resolves its server and where a tunnel hides its data, and Module 13 reads both. Because the domain controllers answer every internal lookup, a laptop that resolves its own names elsewhere, or a name nobody else has asked for, stands out against the ninety clients asking the same two servers.

These sources overlap on purpose. One connection to a malicious domain can appear as a DNS query, a Defender network event from the laptop and a firewall record at the edge, each with different fields and different coverage.

Choosing which one a rule reads is one of the first decisions in Module 3. The firewall sees every machine but no process, Defender sees the process but only on Windows, and DNS sees the name but not whether anything connected.

The accounts with the most power are the ones rules most often need to name, either because their use is what an attack looks like or because their use is ordinary and must not alert. Northgate keeps them in a second watchlist, Watchlist_PrivilegedAccounts, held in Splunk as ne_privileged:

Watchlist_PrivilegedAccounts
| project Account, Kind, OwnerUPN, Purpose
| order by Kind asc, Account asc

The Splunk lookup returns the same rows, one per account:

| inputlookup ne_privileged
| table account kind owner_upn purpose
| sort kind account

Seven accounts, each with a person who owns it and a stated purpose:

account        kind     owner              purpose
a-kdhillon     admin    k.dhillon@ne.com   Server administration
a-smitchell    admin    s.mitchell@ne.com  Server administration
a-tquinlan     admin    t.quinlan@ne.com   Server administration
svc-ansible    service  k.dhillon@ne.com   Configuration management
svc-backup     service  s.mitchell@ne.com  Nightly backup
svc-intranet   service  t.quinlan@ne.com   Intranet application pool
svc-sql        service  t.quinlan@ne.com   SQL Server services

The three admin accounts belong to the systems administrators, one each, separate from their everyday accounts, so s.mitchell reads mail and a-smitchell administers servers. The four service accounts run the backup job, SQL Server, Ansible configuration management and the intranet's application pool, and each has an owner who can say whether a change to it was planned.

The admin accounts are used narrowly, which is what makes them worth watching. In the month each signs in to Windows servers between 15 and 19 times, and only to the same four member servers, srv-nge-mcr-app03, srv-nge-mcr-bkp01, srv-nge-mcr-fs01 and srv-nge-mcr-sql01.

An admin account signing in to a laptop, or to anything outside that set, is unusual for this estate before any rule says so, and a rule can only know that because somebody recorded who the administrators are.

This watchlist does two jobs in the course. It raises severity: an attacker using a-smitchell is worse than one using s.mitchell. And it explains look-alikes: svc-ansible runs encoded PowerShell over WinRM as part of its normal work, which makes it the look-alike lesson 6.1 has to separate from a malicious cradle.

A rule that excludes these accounts by name is easy to write and easy for an attacker to use, so from Module 5 onward the course excludes the behavior, the host and the account together. An exclusion that names all three still lets svc-ansible do its job, and stops an attacker who borrows its name from doing anything else.

A map of the estate is only useful to a detection engineer if it says which logs each important machine sends. Join the asset watchlist to every source that names a host, Defender's process events, Sysmon in the Event table, the Security log and Syslog, and list what reaches each tiered host:

let Seen = union withsource = Table DeviceProcessEvents, Event, SecurityEvent, Syslog
| extend Host = tolower(tostring(split(coalesce(DeviceName, Computer), ".")[0]))
| summarize Tables = make_set(Table) by Host;
Watchlist_AssetCriticality
| extend Host = tolower(Hostname)
| join kind=leftouter Seen on Host
| project Tier, Hostname, Function, Tables
| order by Tier asc, Hostname asc

The Splunk form collects the same four sources by host, appends the watchlist's hosts so that a host sending nothing still appears, and joins the tiers:

(index=defender category="AdvancedHunting-DeviceProcessEvents")
    OR (index=endpoint
        sourcetype="XmlWinEventLog:Microsoft-Windows-Sysmon/Operational")
    OR (index=wineventlog sourcetype="WinEventLog:Security")
    OR ((index=linux OR index=network) sourcetype=syslog)
| eval host=lower(mvindex(split(coalesce(DeviceName, host), "."), 0)),
    table=case(index="defender", "DeviceProcessEvents", index="endpoint", "Event",
        index="wineventlog", "SecurityEvent", true(), "Syslog")
| stats values(table) as tables by host
| append [| inputlookup ne_assets | fields host]
| stats values(tables) as tables by host
| lookup ne_assets host OUTPUT tier function
| where isnotnull(tier)
| table tier host function tables
| sort tier host

Every tiered host, with the sources that carry its events:

tier    host                 sources
Tier 0  fw-nge-mcr-01        none
Tier 0  ne-dc01              Defender, Sysmon, Security
Tier 0  ne-dc02              Defender, Sysmon, Security
Tier 0  ne-fgt-vpn01         syslog
Tier 0  srv-nge-mcr-bkp01    Defender, Sysmon, Security
Tier 1  srv-nge-brs-db01     syslog
Tier 1  srv-nge-mcr-app01    syslog
Tier 1  srv-nge-mcr-app02    syslog
Tier 1  srv-nge-mcr-app03    Defender, Sysmon, Security
Tier 1  srv-nge-mcr-db01     syslog
Tier 1  srv-nge-mcr-db02     syslog
Tier 1  srv-nge-mcr-fs01     Defender, Sysmon, Security
Tier 1  srv-nge-mcr-log01    none
Tier 1  srv-nge-mcr-mon01    syslog
Tier 1  srv-nge-mcr-sql01    Defender, Sysmon, Security
Tier 2  web-nge-mcr-01       syslog
Tier 2  web-nge-mcr-02       syslog

The Windows servers send all three of their sources, and the Linux servers and the VPN gateway send syslog. Two hosts send nothing at all: fw-nge-mcr-01, the perimeter firewall in Tier 0, and srv-nge-mcr-log01, the log collector. The firewall's records of other machines' traffic are in CommonSecurityLog, but nothing records what happens to the firewall itself, such as a configuration change or an administrator signing in to it.

The collector forwards every Linux and firewall record Sentinel receives and reports nothing of its own, so when it stops, the only sign is that those records arrive late or not at all, which is the fault lesson 4.5 measures.

Add both columns to the map. A host's tier says how much an attack on it would matter, and its sources say whether any rule could see one.

The detection program's first priorities come from where they disagree: a Tier 0 host with no telemetry, or a host every investigation depends on that can fail without anyone noticing. Neither is fixed by writing a rule, and recording them now means no later rule is credited with coverage it can't have.

Practice

The map is this lesson's task: the estate in one page, with the query behind each count, so it can be checked and kept current.

Next, lesson 0.2 reads the evidence itself: the corpus month, the tables and their Splunk views, and the label key that says which activity is an attack.