In this section

Network Detection Engineering Foundations: Suricata Signatures, Behavioral Rules and Testing

Module 0

Introduction

An investigation that ends with a report has answered one question once. The same intrusion, or the next one like it, will happen again, and the work of noticing it should not have to be repeated by hand.

Detection engineering is the step that turns a finding into something that runs on every day's traffic, and the hard part is not writing the rule but making sure it finds what it should, leaves everything else alone, and says what it cannot see.

From a finding to a detectionFindingwhat the case showedRulesignature or behaviorTestmalicious and benignsetsTunethresholds, exceptionsRecordowner, misses, review

Each stage tests the one before it: a rule that is never run on benign traffic, or never written down, is a guess.

This lesson takes findings from earlier in the course and builds detections from them: a signature for the beacon from Module 8, and the course's three behavioral rules for beacons, DNS channels and uploads. It tests each against the malicious packs and against m11, a twenty-minute window of the lab's approved software that no detection may flag. Module 11 teaches each step in depth.

1

What to Detect

The parts of a finding that last

A finding contains many facts, and some make better detections than others. The beacon in Module 8 had a server name, two addresses, a TLS fingerprint, a jittered schedule, a fixed exchange size and a browser claim its handshake contradicted. Each could be a rule. They differ in how long they stay true.

What a detection can key on, and how long it lasts

A name or address

Exact, cheap, and changed in a minute

A fingerprint

Shared by every program on the same library

A behavior

A schedule, a fixed exchange, a ratio: costly to change

The name and the addresses are the cheapest to match and the cheapest for the program's operator to change; the beacon itself changed its visible name in the same window, moving to a fronted site.

The fingerprint is shared by every program built on the same TLS library, including the chat client on the same workstation. The behavior, a schedule with a fixed exchange to a destination nothing owns, is the most expensive to change, because changing it means changing how the program works.

The same ranking applies to every finding in this course. The DNS tunnel in Module 5 had a zone name, which is cheap to change, and names at the length limit, which are the tunnel's way of working.

The uploads in Module 10 had a destination, which is cheap to change, and a ratio of bytes sent to bytes received, which is what an upload is. In each case the durable feature is the one that describes what the activity does rather than where it goes.

That is not an argument against matching names. A name rule is precise, fast and easy to explain, and while the name is in use it catches every session. It is an argument for knowing what each rule is worth, and for pairing a cheap, exact rule with a durable, behavioral one so that the second catches what the first loses when the operator moves.

The choice also depends on what the network keeps. A Suricata signature runs on packets as they pass and needs a sensor running Suricata. A behavioral rule runs over logs after the fact, needs those logs to hold the right fields, and can be run again on history.

Module 11's first lesson starts from findings in earlier modules and decides, for each, which of the network's records a detection should key on. The question it asks of every finding is the same: if the operator changed the cheapest thing they could, which records would still show the activity?

2

A Signature

Exact, and exactly as good as the name

A Suricata signature for the beacon's name, matching the TLS server name field exactly. The documentation for the tls.sni keyword says it matches the Server Name Indication, and bsize sets the buffer's length so that a longer name containing this one does not match. Loaded and tested:

$ cat > /tmp/nd09.rules <<'EOF'
alert tls any any -> any any (msg:"ND0.9 beacon name"; tls.sni; \
  content:"api.cloud-metrics.example.org"; bsize:29; sid:9000901; rev:1;)
EOF
suricata -T -S /tmp/nd09.rules -l /tmp 2>&1 | tail -1
i: suricata: Configuration provided was successfully loaded. Exiting.

Run on the Module 8 capture and on the benign window:

$ for p in m08 m11; do rm -rf /tmp/s-$p; mkdir /tmp/s-$p
  suricata -r ../$p/lan.pcap -S /tmp/nd09.rules -l /tmp/s-$p -k none \
    --set 'vars.address-groups.HOME_NET=[10.20.0.0/16]' > /dev/null 2>&1
  echo "== $p"; jq -r 'select(.event_type == "alert")
    | [.src_ip, .tls.sni] | @tsv' /tmp/s-$p/eve.json | sort | uniq -c; done
== m08
     23 10.20.30.45	api.cloud-metrics.example.org
== m11

Twenty-three alerts on the Module 8 capture, every one of the cloud-metrics sessions, and none on the benign window. As a test that is a clean result: everything it was written for, and nothing else.

The signature's precision comes from two choices. It matches one buffer, the server name, rather than searching the whole packet, and it fixes the length, so a longer name that contains this one cannot match by accident. Both make it fast as well as exact.

It also missed six sessions in the same capture. When the program moved to the fronted name, its handshakes named www.example.net, and the signature, keyed on the old name, had nothing to match. A signature for www.example.net would fire on every visit to a popular site, so there is no signature fix.

The rule did exactly what it was written to do; the operator did something else. The test above would never have shown that, because the benign window contains no fronting. A signature's misses come from the attacker's next move, which no test set holds until someone builds it; the record in section 6 is where they are written down instead.

That is the general limit of indicator signatures, and the reason they remain useful anyway. Within their window they are the most precise detections a network can run, and they cost almost nothing to write once a case has produced the indicator.

Module 11's lesson on Suricata signatures writes signatures on several buffers, the server name, the DNS query, the URI and the JA4 fingerprint, and tests each on the same sets, so that the trade between precision and durability is measured rather than argued.

3

A Behavioral Rule

The schedule, not the name

The course's ndrule.py holds three behavioral rules that run over Zeek's logs. Its beacon rule flags a client and server name pair that repeats at least a minimum number of times, with an interval spread inside a band and a fixed exchange: the bytes the client sends barely vary.

The pack's ndscore.py runs it over every labeled set and compares its hits with the pairs the answer guides establish as malicious:

$ python3 ndscore.py beacon
case1  caught 1 of 1, falsely flagged 0
m08    caught 1 of 2, falsely flagged 1
         missed  10.20.30.45 www.example.net
         false   10.20.30.46 www.example.net
m11    caught 0 of 0, falsely flagged 3
         false   10.20.30.45 fleet.example.com
         false   10.20.30.45 www.example.net
         false   10.20.30.46 www.example.net
total  caught 2, missed 1, falsely flagged 4

It caught the beacon in Case 1 and the cloud-metrics schedule in Module 8. It did not use the name, so a new name with the same behavior would still be caught. It missed the fronted sessions, because there were only six of them in the window and the rule's default minimum is eight.

It also flagged four pairs that are not malicious, three of them in the benign window. One is PrintFleet, an approved agent whose documentation says its timer is randomized:

$ grep -A1 PrintFleet hosts.txt
        PrintFleet 4.2       fleet.example.com/v4/status, every 60 s, randomized
                             by up to 25% to spread load

That line describes, in a vendor's words, the behavior the rule was written to find. It is the most common way a behavioral rule fails on its first day:

The reasonable mistake

Testing only on the attack

A rule is written from the case and run on the case's capture. It fires, and it ships. On the first day it fires on PrintFleet, an approved agent whose documentation says its timer is randomized by up to 25%.

Instead: Run every rule on a benign set as well as the malicious ones, and count caught, missed and falsely flagged for each before it goes anywhere.

A jittered timer with a fixed exchange is exactly what the rule looks for, and an approved agent built to spread load across a fleet produces exactly that. The rule is not wrong about the behavior. It cannot know who approved it.

The other three false alarms are visits to www.example.net that happen to be regular and send the same request each time. On a real network, health checks, dashboards that refresh and status pages would land in the same place. Module 11's lesson on behavioral queries reads ndrule.py line by line, and shows how each condition narrows the set of pairs:

The beacon rule against the benign windowEvery client and server namethe whole ssl.logRepeated enoughat least the minimum countJittered within the bandspread inside the rangeFixed exchangesent bytes barely vary

Every TLS pair in twenty minutes of approved software, narrowed by each condition of the rule. The few left are false alarms to explain or exclude.

The last stage is where the approved agents survive, because their exchanges are as fixed as the beacon's. Nothing in the rule's conditions can remove them; only knowledge of what is approved can, and that knowledge lives outside the logs, in an asset list or a vendor's guide.

4

Thresholds and Exceptions

Placed in a measured gap, with an owner

Two kinds of change make a behavioral rule quiet enough to ship: a threshold moved into a gap the measurements show, and an exception for a specific approved pair. The minimum count first:

$ python3 ndscore.py beacon --min 6
case1  caught 1 of 1, falsely flagged 0
m08    caught 2 of 2, falsely flagged 1
         false   10.20.30.46 www.example.net
m11    caught 0 of 0, falsely flagged 3
         false   10.20.30.45 fleet.example.com
         false   10.20.30.45 www.example.net
         false   10.20.30.46 www.example.net
total  caught 3, missed 0, falsely flagged 4

With a minimum of six sessions the rule now catches the fronted pair too: three of three, nothing missed. The false alarms are unchanged, four of them, because the benign pairs already had more than eight sessions each. A lower threshold bought coverage and cost nothing here, which is the case for moving it, measured rather than guessed.

The false alarms need a different fix. Module 11 tries one: a history of destinations already seen on the network, so that the rule reports only first-seen ones. It removes all four false alarms, and it loses the fronted pair again, because www.example.net is a site the network has always visited.

Two improvements that each help alone make the rule worse together, and only rerunning the whole score shows it. A threshold or a filter is a trade, and it belongs where the measurements show a gap, not where the false alarms happen to stop.

The upload rule shows the other kind of fix:

$ python3 ndscore.py upload | tail -7
m10    caught 1 of 1, falsely flagged 1
         false   10.20.30.46 cdn-sync.example.org
case5  caught 1 of 1, falsely flagged 1
         false   10.20.30.46 cdn-sync.example.org
m11    caught 0 of 0, falsely flagged 1
         false   10.20.30.46 cdn-sync.example.org
total  caught 2, missed 0, falsely flagged 3

It catches the uploads to the file-sharing service in Module 10 and Case 5. It also flags CloudSync in every set, the approved sync client on finance workstations, which sends far more than it receives because uploading changes is its job.

No threshold separates the two cleanly. An exception does: one line naming the client range, the service, the reason, an owner and an expiry date, so that it is reviewed rather than forgotten.

Exceptions are where detection programs decay. An exception keyed on a destination alone hides every client that uses it, including one that should not; an exception with no owner never gets removed. Module 11's lesson on baselines, thresholds and exceptions keys each exception on the client and destination pair, gives it an owner and an expiry, and shows ndrule.py refusing to apply one that has lapsed.

5

Measuring a Rule

Caught, missed and falsely flagged

Every result in this lesson has been a count of three things: what the rule caught of what it was built for, what it missed, and what it flagged that it should not have. The DNS rule, which looks for many long-label names under one zone from one client:

$ python3 ndscore.py dns
case2  caught 1 of 1, falsely flagged 0
m05    caught 1 of 1, falsely flagged 0
m11    caught 0 of 0, falsely flagged 0
total  caught 2, missed 0, falsely flagged 0

Both DNS channels caught, Case 2 and the Module 5 tunnel; nothing missed; nothing flagged in the benign window. That is the result a rule needs before it ships, and it is only meaningful because the benign window exists.

A rule tested only on the attack it came from has a catch rate and no false alarm rate, and the false alarm rate is what decides whether anyone will read its alerts.

The sets are small, and the module says so. Two malicious examples and twenty minutes of benign traffic cannot prove a rule will be quiet on a real network for a month. What they can do is catch the obvious failures before anyone else does, and give each rule a repeatable test that can be run again when the rule, the logs or the network changes.

That repeatability is the real value of the score. A rule that scored caught 2, missed 0, falsely flagged 0 last month and scores differently today has told someone that something changed, in the rule, in the logs it reads or in the traffic it sees, and that is worth knowing before an incident depends on it.

The score also keeps tuning honest. Every change in this lesson, the lower minimum, the history filter and the exception, was judged by rerunning the same three counts on the same sets. A change that improves one count and worsens another is visible immediately, rather than being discovered weeks later as a missed intrusion.

Module 11's lesson on measuring rule performance scores every rule this way, and adds the other cost: how much work the sensor does to run it. A rule that is accurate and too expensive to run on the sensor's real traffic is not a rule the sensor will keep.

6

The Detection Record

What it finds, what it misses, who owns it

A rule that has been tested is still not finished until someone else can maintain it. The course keeps a record for each detection, in a fixed format:

$ cat records/dr-11-02.txt
id       DR-11-02
title    Many long-label names under one zone, from one client
finding  case 2 (exercise 5.8) and m05 (the Module 5 lab): data sent out in DNS names
rule     dns --label 44 --names 10
version  7f76ac3f7877
needs    dns.log id.orig_h query
clients  10.20.30.
expect   caught 2, missed 0, falsely flagged 0
miss     labels of 44 characters or fewer: a channel that splits its data finer
miss     fewer than ten names: a channel that sends little, or rotates its zone
miss     DNS over HTTPS to an outside resolver: never reaches dns.log (argued)
owner    detection lead, SOC (backup: on-call analyst)
review   2027-02-10

Each line answers one of four questions that a person inheriting the rule will ask:

What a detection record holds

What it finds

The finding it came from, the rule and its version

What it needs

The logs and fields, and the clients it can see

What it misses

Each known gap, stated

Who owns it

An owner and a review date

The record names the finding the rule came from, the exact rule and the version of ndrule.py it was tested with, the logs and fields it needs, the clients it can see, the result it expects on the sets, the gaps it is known to have, an owner and a review date.

The miss lines are the part most often left out and the most useful: a channel that splits its data into shorter labels, one that sends fewer names, and DNS over HTTPS to an outside resolver, which never reaches the DNS log at all.

None of those is a defect in the rule. Each is a limit of what one rule can see, and writing it down tells the next analyst where to look instead: shorter labels need a different threshold, a quiet channel needs a longer window, and encrypted DNS to an outside resolver needs the TLS log rather than the DNS log.

The pack's ndrecord.py checks every line of a record that a machine can check, against the lab:

$ python3 ndrecord.py records/dr-11-02.txt | tail -5
OK    clients  10.20.30. in m05: 2 source(s)
OK    clients  10.20.30. in m11: 2 source(s)
OK    expect   caught 2, missed 0, falsely flagged 0
OK    review   due 2027-02-10
9 check(s), 0 failed

The version, the logs, the clients, the expected result and the review date are all confirmed, so the record is still true today. When the rule changes, or the logs lose a field, or the review date passes, the check fails and says which line.

Module 11's lesson on the detection record writes one for each of the module's rules and runs the check on all of them. A record that fails its check is the most useful alert a detection program produces about itself: it says that a rule everyone is relying on may no longer be doing what its record claims.

7

What Module 11 Builds On It

Detection engineering in depth

Module 11 takes the course's findings to detections in four ways, signatures, Zeek analytics, behavioral queries over logs and exceptions, then measures and records them. Its exercise is to detect the channel and not the service: build detections that find a malicious channel and leave alone the approved software that resembles it, test them on the malicious and benign sets, and record what they cannot see.

Two of the module's detections are tested in it, the beacon rule with an exceptions file and the long-label DNS rule, on the benign window and on the malicious packs built in earlier modules.

Module 11, in order

0111.1 to 11.2From finding to detection; Suricata signatures
0211.3 to 11.4Zeek-based analytics; behavioral queries over logs
0311.5 to 11.6Baselines, thresholds and exceptions; measuring rule performance
0411.7 to 11.8The detection record; detect the channel, not the service

The tools are the ones this lesson ran: Suricata on the captures, ndrule.py over the Zeek logs, ndscore.py against the labeled sets and ndrecord.py on the records, all in the Module 11 pack, with the malicious packs from earlier modules beside it. Nothing new has to be installed, and every number in the module can be reproduced with the commands each lesson prints.

The habit to carry into it is to treat a rule as a claim that has to be tested in both directions. Test it on the attack, test it on the benign set, put thresholds where the measurements show a gap, give every exception an owner and an expiry, and write down what the rule cannot see before someone discovers it the hard way.

Most of the difficulty in Module 11 is in the benign set, not the malicious one. Several approved products in it resemble one of the course's malicious channels in some way: a jittered agent, a long DNS label, a large upload, a self-signed certificate.

A rule that survives all of them has been tested against the realistic failure, which is an approved product that looks like an attack. Building such a benign set for a real network is part of the work the module teaches: a window of ordinary traffic, kept, labeled and rerun every time a rule changes.

Do

Practice

Which detection is ready to ship?
the lab detections

The prepared evidence below is every detection this lesson ran, with the sets it was tested on and its results. Decide which is ready to ship, which would break the first time the program changed, and which needs an exception before it can go anywhere. Each answer comes with its reasoning, so check it against the rows.

Read each row as a whole: a high catch count means little next to a false alarm count, and a clean result on the benign set means little if the miss column holds the attacker's obvious next move.

The question to ask of each row is the one a detection lead would ask before turning a rule on: what will this page someone about, and how often will it be wrong?

If you run any detection rules, pick one and ask the three questions: what has it caught, what has it falsely flagged, and what is it known to miss. If any of the three has no answer, that is the next piece of work. Most teams can answer the first, some the second, and few the third.

A detection is a finding made repeatable, and it is judged in both directions: caught and missed on the attack, falsely flagged on benign traffic. Names and addresses make precise, short-lived signatures; behavior makes durable rules that need thresholds and owned, expiring exceptions; and every rule ships with a record of what it needs, what it misses and who reviews it.

The next lesson puts every source together on one incident and turns it into a report.

Versions this lesson relies on: Zeek 9.0.1 (LTS); Suricata 8.0.8.