0.2 The Evidence This Course Reads

Introduction

A detection engineer's claims are claims about data: this rule catches that attack, stays quiet on this administration, and costs the queue so many alerts a week. Each claim is only as good as the evidence behind it.

In a production workspace the evidence is incomplete in a particular way: you can see what the rules alerted on, but not what they missed, because nobody labeled the attacks that went unnoticed.

This course's evidence is built to close that gap. It is a month of Northgate's activity, generated with attacks and the ordinary work that resembles them placed in it on purpose, and a key that records each one: where it is, which rows it left and whether it is an attack.

That key is what lets a lesson say a rule caught a case rather than that it looks right. This lesson reads the month, its Splunk view, the key and the faults.

The figure is the course's evidence in two rows: what a rule reads, and what it is measured against.

Every lesson reads the same window: from 14 February to 15 March 2026, ending at 12:00 UTC on the 15th. That end is the corpus's anchor, and the consoles in the lessons treat it as the present, so a query that asks for the last thirty days means the thirty days before the anchor, whenever you run it:

print Now = now(), MonthAgo = ago(30d), WeekAgo = ago(7d)

Now is the anchor, and thirty days back is 13 February, just before the window opens. A rule that reads the last hour, run in a lesson, reads the hour before noon on 15 March. That is how every result in the course stays the same from one year to the next.

The window also limits what a rule can know. A rule that compares today with the previous fourteen days has no full history until 28 February, so a lesson that tests a baseline rule either starts it there or says what the first two weeks miss.

The month itself is 21 event tables. Count each one's rows and the first and last event it holds:

union withsource = Table SecurityEvent, Event, DeviceProcessEvents,
    DeviceNetworkEvents, DeviceFileEvents, DeviceRegistryEvents,
    DeviceEvents, Syslog, SigninLogs,
    AADNonInteractiveUserSignInLogs, AADServicePrincipalSignInLogs, AuditLogs,
    AzureActivity, AWSCloudTrail, ASimDnsActivityLogs,
    CommonSecurityLog, EmailEvents, OfficeActivity,
    SentinelHealth, SentinelAudit, AzureDiagnostics
| summarize Rows = count(), Earliest = min(TimeGenerated), Latest = max(TimeGenerated)
    by Table
| order by Rows desc

The tables, largest first, each with its first and last event:

table                              rows  first             last
DeviceProcessEvents               17374  02-14 00:09  03-15 11:53
Syslog                            16057  02-14 00:00  03-15 11:50
CommonSecurityLog                 14502  02-14 00:13  03-15 11:50
DeviceNetworkEvents               14295  02-16 05:50  03-15 11:40
ASimDnsActivityLogs               13141  02-14 00:13  03-15 11:50
AADNonInteractiveUserSignInLogs   13111  02-14 00:15  03-15 11:52
DeviceFileEvents                  11400  02-16 07:42  03-15 02:49
EmailEvents                        9375  02-14 00:09  03-15 11:56
AzureDiagnostics                   9166  02-16 06:10  03-14 03:39
Event                              8699  02-14 00:52  03-15 11:53
OfficeActivity                     8606  02-16 07:31  03-13 17:51
SecurityEvent                      7005  02-14 00:52  03-15 11:53
SentinelHealth                     4606  02-16 09:20  03-15 11:50
SigninLogs                         2591  02-16 06:14  03-13 22:31
AWSCloudTrail                      2069  02-14 00:02  03-13 04:54
AADServicePrincipalSignInLogs      2008  02-14 00:28  03-15 11:58
AzureActivity                       302  02-14 00:00  03-15 09:02
DeviceRegistryEvents                218  02-14 03:28  03-15 10:55
AuditLogs                            78  02-16 08:52  03-13 15:24
SentinelAudit                        10  02-16 08:57  03-10 16:31
DeviceEvents                          3  02-26 11:41  03-08 02:20

154,616 events in all. The endpoint, syslog, firewall and DNS tables are the largest because they record continuously, every process, connection and lookup. The identity and cloud control-plane tables are smaller because they record decisions: a sign-in, a role assignment, an API call.

SentinelHealth and SentinelAudit are different in kind. They are Sentinel's records of its own rules, when each ran, whether it succeeded and who changed it, which Modules 9, 17 and 18 read to tell a silent rule from a broken one.

Beside the month sit five reference tables with no event time, IdentityInfo, DeviceInfo, ThreatIntelIndicators, Watchlist_AssetCriticality and Watchlist_PrivilegedAccounts, which lesson 0.1 read, and three Lab tables, a separate test workspace that Module 15 uses to run attack simulations without mixing their events into production.

The first and last times in the table above aren't all the same, and the differences are the first lesson in reading evidence. Most tables run to within minutes of the anchor. Six stop earlier: DeviceEvents on 8 March, SentinelAudit on 10 March, AWSCloudTrail on 13 March, AuditLogs on 13 March, OfficeActivity on 13 March and SigninLogs on 13 March.

Reading those ends correctly is the first skill the evidence asks for. Three are simply sparse: DeviceEvents holds 3, SentinelAudit holds 10 and AuditLogs 78 events in the month, so their last event is just whenever the last one happened. Others are the calendar. 14 and 15 March are a Saturday and a Sunday.

Northgate's interactive sign-ins fall from about 127 on a working day to about 24 on a weekend day, nobody opens SharePoint at the weekend at all, and on this last weekend before the anchor nobody did either, so SigninLogs and OfficeActivity end on Friday because the people stopped. One table is not the calendar. Look at CloudTrail day by day from 10 March:

AWSCloudTrail
| where TimeGenerated >= datetime(2026-03-10)
| summarize Events = count(), Latest = max(TimeGenerated)
    by Day = bin(TimeGenerated, 1d)
| order by Day asc

The Splunk form counts the same days and adds the time each day's last event was indexed:

index=aws sourcetype="aws:cloudtrail" earliest="03/10/2026:00:00:00"
| bin _time span=1d
| stats count as events max(_indextime) as last_indexed by _time
| sort _time

The last days of the trail, one row per day with its last event:

day        events  last event
03-10        82  23:04:00
03-11        77  23:04:24
03-12        73  23:00:43
03-13        22  04:54:10

The trail records 73 to 82 events a day and then 22 on 13 March, the last at 04:54:10. That last event is a StopLogging call, made with t.quinlan's leaked access key from 198.51.100.140.

After it, the account's activity continues and nothing records it. CloudTrail doesn't follow the calendar, either: it recorded 586 events across the eight earlier weekend days in the window, every one of them, because servers and pipelines call AWS whether anyone is at work.

A table that ends because people went home and a table that ends because an attacker stopped it look the same in a count; they differ in what the last event was and in whether the source should have kept writing.

The label key records the second kind as a gap instance, s30-trail-stopped: "CloudTrail from the production account stopped by the attacker", from 13 March until the anchor. A gap has no rows, because its evidence is the absence, which is why Module 18 builds rules that watch sources rather than events.

Every lesson pairs its KQL with SPL, and the Splunk side reads a projection of the same month, built by the same generator in the same run: the same events, with Splunk's field names, under the sourcetypes the Splunk add-ons for each product assign. Count four of them both ways:

union withsource = Table DeviceProcessEvents, Syslog, AWSCloudTrail, CommonSecurityLog
| summarize Rows = count() by Table
| order by Table asc

In Splunk each table becomes a sourcetype in an index, which this search counts:

(index=defender category="AdvancedHunting-DeviceProcessEvents")
    OR sourcetype=syslog OR sourcetype="aws:cloudtrail" OR sourcetype="pan:traffic"
| stats count as rows by index, sourcetype
| sort sourcetype index

Splunk's side of the count, by index and sourcetype:

index     sourcetype                rows
aws       aws:cloudtrail             2069
defender  ms:defender:eventhub      17374
network   pan:traffic               14502
linux     syslog                    14917
network   syslog                     1140

The rows match table for table, with one difference in shape: Syslog is one table in Sentinel and one sourcetype in two indexes in Splunk, because the Linux servers' events go to linux and the VPN gateway's to network. The full mapping, every table with a Splunk view, to its index and sourcetype:

table                           index          sourcetype
SecurityEvent                   wineventlog    WinEventLog:Security
Event                           endpoint
                                XmlWinEventLog:Microsoft-Windows-Sysmon/Operational
DeviceProcessEvents             defender       ms:defender:eventhub
DeviceNetworkEvents             defender       ms:defender:eventhub
DeviceFileEvents                defender       ms:defender:eventhub
DeviceRegistryEvents            defender       ms:defender:eventhub
DeviceEvents                    defender       ms:defender:eventhub
Syslog                          linux and network syslog
SigninLogs                      azuread        azure:monitor:aad
AADNonInteractiveUserSignInLogs azuread        azure:monitor:aad
AADServicePrincipalSignInLogs   azuread        azure:monitor:aad
AuditLogs                       azuread        azure:monitor:aad
AzureActivity                   azure          azure:monitor:activity
AWSCloudTrail                   aws            aws:cloudtrail
ASimDnsActivityLogs             dns            MSAD:NT6:DNS
CommonSecurityLog               network        pan:traffic
EmailEvents                     defender       ms:defender:eventhub
OfficeActivity                  o365           o365:management:activity
AzureDiagnostics                azure          mscs:azure:eventhub

Two things in the mapping matter for writing rules. The four Entra ID tables land in one sourcetype, azure:monitor:aad, in the azuread index, and a Splunk search tells them apart by a category field rather than by where it reads, so a sign-in rule that forgets the category also reads every audit event.

The Defender tables share one sourcetype too, ms:defender:eventhub in the defender index, and a search selects a table by its category, which is the table's name after AdvancedHunting-.

SentinelHealth and SentinelAudit have no Splunk view, because they describe Sentinel itself, and Splunk keeps the equivalent records of its own scheduled searches, which lesson 9.5 reads. The Splunk view holds 150,060 events, the same events as the KQL tables plus the Lab copies.

Selecting by category is what keeps a Splunk rule both fast and correct, and the course's SPL always does it: a search that names only the sourcetype reads every Defender table and has to filter them apart afterwards, and a search that forgets the category matches a field name that happens to exist in two tables, such as FileName, in both.

Field names are where a rule translated between languages most often breaks, because the same value is called different things and nothing warns you: a search for a field that doesn't exist matches nothing and reports no error. Take one interactive sign-in, q.pemberton's at 02:47 on 7 March:

SigninLogs
| where UserPrincipalName == "q.pemberton@ne.com"
    and TimeGenerated == datetime(2026-03-07T02:47:39Z)
| project TimeGenerated, UserPrincipalName, IPAddress, AppDisplayName,
    ResultType, ResultDescription, Location

The same event in Splunk, selected by the user and the time:

index=azuread sourcetype="azure:monitor:aad" category=SignInLogs
    user="q.pemberton@ne.com" _time="2026-03-07T02:47:39.000Z"
| table _time user src_ip app signature_id reason src_country

The two rows, column by column, Sentinel's name beside Splunk's:

Sentinel             Splunk         value
TimeGenerated        _time          2026-03-07T02:47:39.000Z
UserPrincipalName    user           q.pemberton@ne.com
IPAddress            src_ip         203.0.113.58
AppDisplayName       app            OfficeHome
ResultType           signature_id   50074
ResultDescription    reason         Strong Authentication is required.
Location             src_country    GB

Every value is identical; only the names differ. UserPrincipalName is user, IPAddress is src_ip, and ResultType, the Entra ID result code, is signature_id. Code 50074 isn't a failure in the usual sense: it means the password was right and the sign-in stopped to ask for multi-factor authentication.

That matters for the rule in section 8, because a spray that guessed a password correctly and stopped at MFA is a success hiding in a failure code.

Most of the Splunk names follow the Common Information Model, the field names Splunk's add-ons map sources to, so user and src_ip mean the same thing across sign-ins, firewall records and Linux logs.

Sentinel keeps each source's own names and normalizes through separate ASIM parsers, which Module 4 builds. Neither approach is free: one name for everything makes a rule portable, and the original names keep detail a common model drops. When a lesson's KQL and SPL disagree in shape, the field mapping is the first place to look.

Time has two names on each side as well. TimeGenerated and _time are when the event happened, which is what a rule's window reads. The time the record reached the workspace is separate: _TimeReceived in Sentinel and _indextime in Splunk, on every row in this corpus.

The gap between them is the ingestion delay that lessons 4.5 and 6.5 measure, and it is why a rule can run correctly over the right hour and still miss what happened in it.

The label key is a file beside the corpus that records every instance placed in the month. It holds 108 instances from 31 scenarios: 59 attacks with rows, 44 look-alikes, ordinary activity built to resemble an attack, and five gaps, where a source went silent, three of them because of an attack. Here is one attack:

{
  "id": "s07-ssh-guessing",
  "scenario": "s07",
  "label": "malicious",
  "title": "Password guessing over SSH succeeds, then sudo to root",
  "technique": "T1110.001",
  "tactic": "Credential Access",
  "entities": {
    "host": "WEB-NGE-MCR-02",
    "ip": "203.0.113.198"
  },
  "start": "2026-03-07T01:03:08.000Z",
  "end": "2026-03-07T01:27:29.001Z",
  "rows": {
    "Syslog": 221
  }
}

The fields are what a test needs. The label says what the instance is. The technique and tactic say what an attack is an example of, 34 ATT&CK techniques across 11 tactics in all.

The entities say who and where, here a host and the address the guessing came from. The start and end bound it in time, and rows says how many events it left in each table, 221 lines of Syslog for this one.

Look-alikes are the half that production data never labels and the half that makes a rule hard to write well. Beside the guessing sits s07-mistyped-password: "An administrator mistypes her password, then uses sudo", SRV-NGE-MCR-DB02 on 11 March, 10 rows. Failed SSH passwords followed by a success and sudo, which is exactly what the attack looks like, done by somebody entitled to do it.

Every test set in the course pairs its attacks with at least one such neighbor, chosen because it is what a first draft of a rule would also catch. That choice is what makes a test informative. A test set of attacks alone is passed by a rule that alerts on everything, and only the look-alike shows whether the rule understood the difference.

The key is how the course grades without opinion. A rule's output row matches an instance when the row names the instance's entities and falls within its time window, allowing an hour before it and a day after for rules that summarize.

A row that matches an attack is a detection, a row that matches a look-alike is a false positive with a name, and a row that matches nothing is noise the key didn't anticipate, which is often the most interesting kind.

Instances are grouped into test sets: each set holds the attacks a rule of one kind should catch and the look-alikes it should leave alone. The SSH set pairs the guessing with the mistyped password. Here is the first version of an SSH rule, from lesson 1.3, which alerts when one source both fails and succeeds against a host within an hour:

Syslog
| where ProcessName == "sshd"
| extend Src = extract(@"from (\S+) port", 1, SyslogMessage),
    Outcome = extract(@"^(Failed|Accepted) (password|keyboard-interactive)",
        1, SyslogMessage)
| summarize Failures = countif(Outcome == "Failed"),
    Successes = countif(Outcome == "Accepted")
    by Computer, Src, bin(TimeGenerated, 1h)
| where Failures > 0 and Successes > 0

The same rule in Splunk extracts the address and the outcome from the message text:

index=linux sourcetype=syslog process=sshd
| rex field=message "from (?<src>\S+) port"
| rex field=message "^(?<outcome>Failed|Accepted) (password|keyboard-interactive)"
| bin _time span=1h
| stats count(eval(outcome="Failed")) as failures
    count(eval(outcome="Accepted")) as successes by host src _time
| where failures > 0 AND successes > 0

Both return two rows, 212 failures and a success from 203.0.113.198 on WEB-NGE-MCR-02, and 4 failures and a success from 10.0.1.194 on SRV-NGE-MCR-DB02. Scored against the set, the attack and its look-alike side by side:

The board reads the way every test in the course reads. The rule alerted on the guessing, which it must, and fired on the mistyped password, which it must not; there were no rows on anything else.

The two cases differ by two orders of magnitude in failures, and the rule's condition, any failure and any success, can't see the difference. Lesson 1.3 adds the threshold and the source, and the board turns green. The last column, other rows, counts output that matches no case in the set.

The default standard is the strictest one: catch every attack, fire on no look-alike, and return nothing else. Lesson 5.5 writes acceptance criteria that relax it where a rule has a reason to, a summary row allowed beside each detection, say, and states the reason in the rule's specification.

Scoring on a set rather than on the whole month is deliberate, and it is how every detection lesson in the course ends. It asks a narrow question, does this rule separate this attack from this administration, which a rule can be designed to answer. The month-wide question, what did all the rules raise and what did they miss, is lesson 0.3's.

Attacks and look-alikes are what a rule should find and ignore. Faults are what goes wrong with the evidence on the way to the rule: a sensor that stops, a field that changes, a record that arrives late or twice. The key records six, each with the tables it touches:

sysmon-gap             Event
identity-variants      one account, in every table that names it
firewall-duplicates    CommonSecurityLog
forwarder-late         Syslog, CommonSecurityLog
cef-processname-drift  Syslog
trail-stopped          AWSCloudTrail

Each is something production does to detections without warning. The Sysmon gap is six hours of silence from one laptop while Defender kept reporting it. The duplicates are two hours of firewall records written twice.

The late forwarder is the collector stopping, as lesson 0.1 warned it could, so its records arrive hours after their own timestamps. The CEF drift is the gateway's empty program name. The stopped trail is the attacker's StopLogging in section 2. And identity variants is one person written several ways. See it across four sources:

union withsource = Table SigninLogs, OfficeActivity, SecurityEvent,
    DeviceProcessEvents
| extend Who = coalesce(UserPrincipalName, UserId, TargetUserName, AccountName)
| where Who has_any ("j.okonkwo@ne.com", "jokonkwo")
| summarize Rows = count() by Table, Who
| order by Table asc

The Splunk form normalizes the account field before matching both spellings:

(index=azuread sourcetype="azure:monitor:aad" category=SignInLogs)
    OR (index=o365 sourcetype="o365:management:activity")
    OR (index=wineventlog sourcetype="WinEventLog:Security")
    OR (index=defender category="AdvancedHunting-DeviceProcessEvents")
| eval who=lower(coalesce(user, TargetUserName, AccountName))
| search who="j.okonkwo@ne.com" OR who="jokonkwo" OR who="jokonkwo@*"
| eval source=coalesce(category, sourcetype)
| stats count as rows by source who

One person, counted by each spelling the four tables use for them:

table                 as written in the row             rows
DeviceProcessEvents   jokonkwo                           129
OfficeActivity        j.okonkwo@ne.com                   233
SecurityEvent         jokonkwo                            36
SecurityEvent         jokonkwo@CORP.NE.COM                41
SigninLogs            j.okonkwo@ne.com                    23

Entra ID and Microsoft 365 write j.okonkwo@ne.com. Active Directory and the endpoint tables write jokonkwo, which isn't the start of the UPN, as it is for most people at Northgate, and the domain controllers' service ticket events, 4769, write jokonkwo@CORP.NE.COM, the account name with its Kerberos realm.

A rule that joins a sign-in to endpoint activity by the part of the UPN before the at sign finds nothing for Joseph Okonkwo, and nobody notices, because a join that finds nothing returns nothing.

Lesson 4.4 resolves one user across five sources, and this person is why it's needed. Each fault in the key carries its window and the hosts, tables or accounts it touches, so when a rule misses a case in a later lesson, the first check is whether a fault overlaps it.

A miss caused by a fault is a finding about the data, and the fix belongs to whoever owns the source rather than to the rule.

The key is evidence, but it is also a document somebody wrote, and the habit this course asks for applies to it as much as to a rule: check a claim against the data before building on it. The distributed spray's note says "23 addresses from 203.0.113.50, two accounts each". Count it:

SigninLogs
| where TimeGenerated between (datetime(2026-03-07T01:00:00Z)
        .. datetime(2026-03-07T07:40:00Z))
    and IPAddress startswith "203.0.113."
| summarize Accounts = dcount(UserPrincipalName), Attempts = count() by IPAddress
| summarize Addresses = count(), AccountsPerAddress = make_set(Accounts),
    Attempts = sum(Attempts)

The Splunk form counts the same addresses and attempts:

index=azuread sourcetype="azure:monitor:aad" category=SignInLogs src_ip="203.0.113.*"
    earliest="03/07/2026:01:00:00" latest="03/07/2026:07:40:00"
| stats dc(user) as accounts count as attempts by src_ip
| stats count as addresses values(accounts) as accounts_per_address
    sum(attempts) as attempts

The result is one row, which summarizes every address in the window:

addresses 23   accounts per address [2]   attempts 47

Twenty-three addresses, every one trying two accounts, and 47 attempts, which is the 47 rows the key says the instance left in SigninLogs. The note holds. It also explains why this spray is hard: a rule that counts failed accounts per address sees two at a time and never reaches a threshold of ten, which is the miss lesson 0.3 finds on the schedule and lesson 12.1 fixes.

The note's second half says the right password for q.pemberton stopped at MFA, which section 4 showed as result code 50074 from 203.0.113.58.

Every lesson works this way: a figure on the page comes from a query that ran when the page was built, and the query is beside it, so you can rerun it and see the same answer. The key is held to the same standard, and only partly by machine.

The validator that runs with every corpus build checks that each instance left rows, that its entities appear in them, and that every test set holds an attack and a look-alike. It can't read a note, which is a sentence a person wrote, so a note is checked the way this section checked one: by a query, before anything is built on it.

A production equivalent of the key, a record of past incidents with their entities and times, deserves the same check before anyone scores a rule against it.

Practice

Tracing an instance is this lesson's task: from the key to its rows in both languages, its test set and any fault that would change what a rule sees.

Next, lesson 0.3 puts five of Northgate's rules on the schedule for the month and judges every alert they raise against the label key: which fired correctly, which fired on administration, and which attacks nothing caught.