In this section

0.6 Machine Learning for Defenders

Module 0

Introduction

Machine learning is already in most security tools: the scoring behind an endpoint alert, the phishing classifier in a mail gateway, the anomaly model that flags a sign-in.

Module 6 trains a detection classifier of its own and then attacks it the way an adversary would, with evasion, poisoning and extraction, before measuring which defense costs the least accuracy. That work needs a precise vocabulary for what a model is, how it is measured, and why it can be attacked at all.

This lesson builds the smallest version of all of it. An assistant writes a classifier that scores process starts for malice, from five features, with a train and test split and an evaluation at a chosen threshold.

The lesson then reads what the model learned, sweeps the threshold to show what precision and recall trade, evades the model one feature at a time, and measures two defenses and one attack: training only on what an attacker cannot easily change, and poisoning the training labels.

Procedure: train, measure, then evadeSplit: shuffle with a seed;hold out 30% for testTrain logistic regressionon the five featuresFlag test rows at or abovethe thresholdThe threshold tradesprecision for recall.Count TP, FP, FN, TN;precision and recallTake a flagged rowSet one mutable feature tothe next value from 0 upLength, entropy, URLs:the author chooses them.Below the threshold?yesEvaded: returnthe changed rownoValues or featuresleft?yesNext valuenoNone: this row cannotevade with one featurestepdecisionnotestops or failsbranch ends

The classifier is measured on rows it never trained on, then attacked the way an author would: by changing only the features the author controls.

1

Features and Labels

What the model is given

A model never sees a process start. It sees numbers computed from one, called features, and for training, a label saying what the right answer was. Here are the first rows of the lab's data.

Visual Studio Codeprocs.jsonl - sec410-lab
procs.jsonl
lab › data › foundations › procs.jsonl
1{"id": "p-0000", "cmd_length": 87, "entropy": 4.04, "encoded_flag": 0, "office_parent": 0, "url_count": 1, "malicious": 0}2{"id": "p-0001", "cmd_length": 1170, "entropy": 5.01, "encoded_flag": 0, "office_parent": 0, "url_count": 0, "malicious": 1}3{"id": "p-0002", "cmd_length": 284, "entropy": 5.83, "encoded_flag": 0, "office_parent": 1, "url_count": 1, "malicious": 1}4{"id": "p-0003", "cmd_length": 132, "entropy": 3.65, "encoded_flag": 0, "office_parent": 0, "url_count": 0, "malicious": 0}5{"id": "p-0004", "cmd_length": 1343, "entropy": 4.95, "encoded_flag": 1, "office_parent": 1, "url_count": 1, "malicious": 1}6{"id": "p-0005", "cmd_length": 240, "entropy": 4.27, "encoded_flag": 0, "office_parent": 0, "url_count": 1, "malicious": 0}7{"id": "p-0006", "cmd_length": 851, "entropy": 5.18, "encoded_flag": 1, "office_parent": 0, "url_count": 0, "malicious": 0}8{"id": "p-0007", "cmd_length": 371, "entropy": 4.7, "encoded_flag": 0, "office_parent": 0, "url_count": 0, "malicious": 0}
mainJSONUTF-8Ln 1, Col 1
660 process starts, five features each, labeled. The first eight.

There are 660 rows, 120 of them labeled malicious. Each has five features: the command line's length in characters, its entropy, a measure of how random its characters are, whether it used an encoded command flag, whether its parent was an Office application, and how many URLs it contains.

The data is synthetic and generated by a script in the lab, with one detail drawn from real estates: some benign management agents also run long, encoded, high-entropy commands, so no single feature separates the classes cleanly.

The most important property of each feature is who controls it. The person who writes a command chooses its length, its entropy and its URLs, and can change them freely: padding a command with comments lowers its entropy, and fetching from a variable instead of a literal URL removes a URL.

Whether the parent was Word depends on how the attack was delivered, and whether the command was encoded depends on how it was built; both can change, but not freely in the middle of an attack. The contract calls the first group mutable. That distinction decides how hard the model is to evade, and it is a security decision, not a statistical one.

The labels deserve as much suspicion as the features. In a real pipeline they come from analysts' verdicts, sandbox results or threat intelligence, and each source is wrong some of the time. A benign tool marked malicious teaches the model to flag it; an attack closed as benign teaches the model to miss the next one.

The class balance matters too. With 120 malicious rows among 660, a model that flagged nothing would be right 82 percent of the time, which is why accuracy is never the measure for a detector and the lesson never reports it.

2

The Contract

Split, train, evaluate, evade

The exercise file states what the classifier must do.

Visual Studio Codeml.py - sec410-lab
ml.py
lab › foundations › ml.py
⋯ 29 lines folded3031def split(rows, test_fraction=0.3, seed=0):32    raise NotImplementedError('lesson 0.6: write split')⋯ 5 lines folded3839def evaluate(model, rows, threshold):40    raise NotImplementedError('lesson 0.6: write evaluate')⋯ 1 lines folded4243def evade(model, row, threshold, mutable=MUTABLE, steps=50):44    raise NotImplementedError('lesson 0.6: write evade')
mainPythonUTF-8Ln 1, Col 1
Four functions: the whole life of a detection model, from data to attack.

The split comes first because everything else depends on it. A model is trained on one part of the data and measured on another it has never seen, because a model measured on its own training data reports how well it memorized, not how well it detects. The split is seeded so that the same test set is used every time, which makes results comparable from run to run.

Run the check against the file as shipped.

$ python3 -m lab check f06
f06  0.6 Machine Learning for Defenders: lab/foundations/ml.py
FAIL  the functions are written
      lesson 0.6: write split
FAIL

The check needs every function before it can test the next, because each one feeds the one after: the split feeds training, training feeds evaluation, and evaluation decides which rows are worth evading. Its properties test agreement with scikit-learn's own metric functions rather than fixed numbers, so a correct implementation passes whatever small differences in order or rounding it introduces.

A random split is the simplest correct choice and not always the best. Detection models are trained on the past and used on the future, so a split by time, training on earlier weeks and testing on the latest, measures what the model will face more honestly than a shuffle does.

Duplicate or near-duplicate rows on both sides of a split inflate every metric, because the model is partly tested on examples it has already seen. Module 6's dataset is split by time for that reason.

3

Session: The Brief

Measure the weakness, do not hide it

The brief points at the contract and adds one instruction about intent.

SEC410 lab, lesson 0.62026-11-18
ml.pyPYprocs.jsonlJSONL

Write the four functions in ml.py to the contract in its docstring, with scikit-learn. Keep evade() honest to the contract: one mutable feature, values from 0 to the row's own value, a copy rather than the original. I want to measure how easily the detector is evaded, not to make it look hard to evade.

ml.pyCode · PY · 81 linesDownload
Reply to Claude…Opus 5.5
Recorded unedited in assistant-records/m00/0.6-brief.md

The last sentence of the brief matters with an assistant. Asked to write an evasion function for a detector, an assistant may reasonably try to make the detector look good: a narrow search, an early exit, a conservative interpretation.

The brief says the purpose is to measure how easily the model is evaded, which makes a thorough search the correct answer. The constraints on that search, one feature, from zero to its current value, on a copy, come from the contract, and the brief repeats them so they are not relaxed.

The contract also fixes the metrics, which removes a choice an assistant would otherwise make. Asked to evaluate a classifier without being told how, assistants commonly report accuracy, because it is the default in many examples, and on imbalanced security data accuracy flatters any model.

Naming precision, recall and the confusion counts in the contract means the draft measures what a detection engineer needs, and the check can verify it against a reference.

4

The Draft

Four short functions

The reply was the file. Here is the code below the docstring.

Visual Studio Codeml.py - sec410-lab
ml.py
lab › foundations › ml.py
26import random2728from sklearn.linear_model import LogisticRegression2930FEATURES = ['cmd_length', 'entropy', 'encoded_flag', 'office_parent', 'url_count']31MUTABLE = ['cmd_length', 'entropy', 'url_count']323334def _matrix(rows):35    return [[r[f] for f in FEATURES] for r in rows]363738def split(rows, test_fraction=0.3, seed=0):39    shuffled = list(rows)40    random.Random(seed).shuffle(shuffled)41    n = round(len(shuffled) * test_fraction)42    return shuffled[n:], shuffled[:n]434445def train(rows):46    model = LogisticRegression(max_iter=1000)47    model.fit(_matrix(rows), [r['malicious'] for r in rows])48    return model495051def evaluate(model, rows, threshold):52    probs = model.predict_proba(_matrix(rows))[:, 1]53    tp = fp = fn = tn = 054    for row, p in zip(rows, probs):55        flagged, bad = p >= threshold, row['malicious'] == 156        if flagged and bad:57            tp += 158        elif flagged:59            fp += 160        elif bad:61            fn += 162        else:63            tn += 164    precision = tp / (tp + fp) if tp + fp else 0.065    recall = tp / (tp + fn) if tp + fn else 0.066    return {'tp': tp, 'fp': fp, 'fn': fn, 'tn': tn, 'precision': precision, 'recall': recall}676869def evade(model, row, threshold, mutable=MUTABLE, steps=50):70    for feature in mutable:71        top = row[feature]72        for k in range(steps):73            value = top * k / (steps - 1)74            if isinstance(top, int):75                value = int(round(value))76            candidate = dict(row, **{feature: value})77            if candidate[feature] == row[feature]:78                continue79            if model.predict_proba(_matrix([candidate]))[0, 1] < threshold:80                return candidate81    return None
mainPythonUTF-8Ln 1, Col 1
The draft: a seeded split, logistic regression, a confusion count, and a one-feature search.

The split shuffles a copy with the given seed and takes the test set from the front. Training fits a logistic regression, a model that learns one weight per feature and turns their weighted sum into a probability between 0 and 1.

Evaluation scores every row, flags those at or above the threshold, and counts the four outcomes: true positives, malicious and flagged; false positives, benign and flagged; false negatives, malicious and missed; true negatives, benign and passed. Precision is the share of flags that were right.

Recall is the share of malicious rows that were flagged. The evasion search tries each mutable feature in turn, stepping from zero up to the row's own value, and returns the first copy that scores below the threshold.

Logistic regression is chosen here because it is readable, not because it is the strongest model available. Each weight says how much its feature moves the score, so the model's reasoning can be inspected directly, which section 5 does.

More complex models, gradient-boosted trees and neural networks, usually separate classes better and are harder to read, and evading them works on the same principle. The features are used as they are, unscaled, which is why the weight on length looks tiny: a single character moves the score very little, and a thousand characters move it a great deal.

Run the check against the draft.

$ cp records/ml-0.6.py lab/foundations/ml.py
$ python3 -m lab check f06
f06  0.6 Machine Learning for Defenders: lab/foundations/ml.py
ok    split() is repeatable, disjoint and the right size
      462 train, 198 test
ok    evaluate() counts and scores agree with scikit-learn
      tp 28 fp 9 fn 2 tn 159, precision 0.76, recall 0.93
ok    at threshold 0.3 the detector catches at least 90% of malicious test rows
      recall 0.93
ok    evade() changes one mutable feature, on a copy, and gets below threshold
      28 of 28 caught malicious rows evaded
PASS

The split put 198 rows in the test set. At a threshold of 0.3 the model flagged most malicious test rows, its counts agree exactly with scikit-learn's own metrics, and every malicious row it caught could be evaded by changing a single feature.

5

What the Model Learned

Weights, and the threshold

A logistic regression's weights are readable. This prints them.

$ cp records/ml-0.6.py lab/foundations/ml.py
$ python3 exp.py weights
cmd_length       0.003  chosen by the attacker
entropy          2.090  chosen by the attacker
encoded_flag     0.180  
office_parent    2.218  
url_count        1.747  chosen by the attacker

A positive weight raises the score as the feature grows. Two of the three strongest weights are on features the attacker chooses: entropy and URL count. The model learned what separates the classes in this data, which is right, and it learned to rely on things an attacker controls, which is the weakness section 6 measures.

Length carries almost no weight per character, though across hundreds of characters it adds up. A model does not know which features an adversary controls. Only the person who chose the features does.

The weights have a precise meaning. Logistic regression works in log-odds, so a weight is how much one unit of its feature adds to the log of the odds that a row is malicious.

Raising a command's entropy by one unit multiplies those odds by about 8; an Office parent multiplies them by about 9. That reading is why an analyst can look at a logistic regression and say what it believes, and why section 6 can predict which feature an evasion will reach for before running it.

The model outputs a probability. The detection is the threshold, the point above which a probability becomes an alert. This sweeps it.

$ cp records/ml-0.6.py lab/foundations/ml.py
$ python3 exp.py sweep
threshold  tp  fp  fn  precision  recall
      0.1  30  26   0       0.54    1.00
      0.2  29  14   1       0.67    0.97
      0.3  28   9   2       0.76    0.93
      0.5  22   3   8       0.88    0.73
      0.7  19   2  11       0.90    0.63
      0.9  13   0  17       1.00    0.43

Lower the threshold and recall rises while precision falls: more malicious rows are caught, and more benign ones are flagged with them. At 0.3, precision is 0.76 and recall 0.93, with 9 false positives. At 0.5, precision rises to 0.88 and recall falls to 0.73, missing 8 malicious rows.

At 0.9 nothing benign is flagged and recall is 0.43. There is no correct threshold in the abstract. It is chosen against the cost of a missed attack and the cost of an analyst's time on a false alarm, and it is a decision the team running the detection should make and record, not a default.

The test set's proportions also flatter the false positives. Malicious rows are about one in 7 of the test set here; in a real estate, malicious process starts are a tiny fraction of millions. What carries over from the test set is the false positive rate, the share of benign rows flagged, which at a threshold of 0.3 is 9 in 168 benign test rows.

Applied to a day of a large estate's benign process starts, a rate like that produces far more alerts than any team can read, and that arithmetic, more than any attack, is why real detectors are tuned toward fewer false positives at the cost of recall.

6

Evasion, Measured

One feature, and what is realistic

The check confirmed that every caught malicious row could be evaded. This shows how.

$ cp records/ml-0.6.py lab/foundations/ml.py
$ python3 exp.py evade
p-0531  entropy      4.89 -> 0.0
p-0513  entropy      5.74 -> 0.0
p-0035  entropy      5.38 -> 0.0
p-0169  cmd_length    673 -> 0
p-0393  cmd_length    752 -> 0
evaded by feature: {'entropy': 17, 'cmd_length': 11} of 28 caught

Each evasion changed one feature, and the changes are where the review has to be careful. The search starts at zero, so the first value it tried for entropy was 0.0 and for length was 0, and those are what it returned.

No command line has zero entropy or zero length; those evasions are artifacts of a search that ignored what an attacker can actually produce. The contract allowed it, the draft followed the contract, and the check passed. The measurement was still wrong about the threat.

A realistic search bounds each feature to values an attacker can reach. Padding a command with ordinary text lowers its entropy toward that of English, but not to zero. This repeats the evasion with entropy only, padded no lower than four floors.

$ cp records/ml-0.6.py lab/foundations/ml.py
$ python3 exp.py floor
entropy floor   caught rows that evade
          3.0    22 of 28
          3.5    20 of 28
          4.0    16 of 28
          4.5     7 of 28

With entropy padded down to 3.0, 22 of 28 caught rows evade. At 4.0, 16 do; at 4.5, 7. Those are the numbers that describe the detector's weakness, because they describe an attack someone could run: append a paragraph of comment text to the command. Module 6 builds evasion with realistic constraints from the start, and the practice asks you to bound every mutable feature the same way.

The difference between the two searches has a name in the research literature on adversarial machine learning: changing the numbers a model sees, against changing the real object those numbers describe.

An evasion found in feature space only matters if some real input, here an actual command line that still does the attacker's job, produces those features. Padding with comments is a real-world change; a command line of length zero is not. Every evasion result should say which kind it is.

An attacker rarely has the model itself, which this lesson's search assumed. They have something close to it: the detection products their target is likely to run, a test machine to try commands against, and the knowledge that encoded PowerShell from a document is what defenders look for.

Evasions found against one model often work against others trained on similar data, so the attacker's lack of access is less protection than it sounds. The defender's advantage is the opposite knowledge: which features are cheap for an attacker to change and which are expensive, and building detections on the expensive ones.

7

Two Defenses and an Attack

Immutable features, and poisoned labels

The obvious defense is to stop using what the attacker controls. This retrains on the encoded flag and the Office parent alone.

$ cp records/ml-0.6.py lab/foundations/ml.py
$ python3 exp.py immutable
encoded_flag and office_parent only: precision 0.44, recall 0.87, fp 33, fn 4

The model can no longer be evaded through entropy, length or URLs, and it pays for that: precision drops to 0.44 with 33 false positives, because the benign management agents also run encoded commands and the model can no longer tell them apart.

Recall is 0.87. This is the trade Module 6 measures carefully: a defense that removes a weakness by removing information costs accuracy on ordinary traffic, and the best defense is the one that costs least for the protection it buys.

There are middle grounds the two extremes miss, and Module 6 measures several. A feature can be made harder to change: entropy measured on the command after comments are stripped cannot be lowered by adding comments.

A feature can be added that describes the evasion itself: an unusually long comment in a command line is a signal in its own right. And the attacker-controlled features can be kept with their influence capped, so they can raise a score but cannot alone push a dangerous row below the threshold. Each costs something different, and the right one depends on the traffic the detector sees.

Evasion attacks the input. Poisoning attacks the training data. This retrains the model with some malicious training rows relabeled benign, as an attacker who could influence labeling might arrange.

$ cp records/ml-0.6.py lab/foundations/ml.py
$ python3 exp.py poison
 0 malicious training rows relabeled benign: recall 0.93, precision 0.76
10 malicious training rows relabeled benign: recall 0.87, precision 0.87
20 malicious training rows relabeled benign: recall 0.70, precision 0.84
40 malicious training rows relabeled benign: recall 0.60, precision 1.00

With no poisoned labels, recall at the threshold is 0.93. With 20 of the training set's malicious rows relabeled, it falls to 0.70, and with 40, to 0.60. Precision rises at the same time, because the model has learned to flag less, which on a dashboard looks like an improvement. A poisoned model fails quietly: it misses more, and the alerts it does raise look cleaner than before.

Poisoning needs access to the labels, and in many pipelines that access is wider than it looks. Models that retrain on analyst verdicts learn from whoever closes alerts, and models that retrain on telemetry learn from whatever the environment produced, including activity an attacker generated to look normal.

The defenses are mostly about the pipeline rather than the model: knowing where every label came from, reviewing the training data that changed between versions, and comparing a retrained model against the last one on a fixed set of known attacks before it ships.

The third attack, extraction, needs no access to the model's insides. An attacker who can submit inputs and see scores can query the model enough times to fit a copy, then search the copy for evasions offline and use them against the original. It is why exposing scores is a choice with consequences, and Module 6 measures how many queries it takes.

The defense against extraction is mostly about what the model reveals.

A detector that returns only a decision, flagged or not, leaks less per query than one that returns a score; one that limits how many queries any caller can make leaks less in total; and one whose decisions an attacker cannot observe at all, because alerts go to a security team rather than back to the user, leaks almost nothing.

Many security models sit in that last position by design. The ones that do not, such as a classifier offered as a public scanning service that returns a verdict to anyone who submits a file, are the ones where extraction deserves the most attention, because every query is an answer the attacker keeps.

8

From the Classifier to Module 6

What the review carries forward

The session record keeps what held, every figure measured here, and the one correction the measurements forced.

Visual Studio Code0.6-session.md - sec410-lab
0.6-session.md
records › 0.6-session.md
1# Session 0.6: a detection classifier, evaded23Assistant: Claude Opus 5.5 (claude-opus-5-5), the SEC410 build session, 2026-11-1845## Held6- Seeded split, test first; logistic regression on FEATURES.7- evaluate() counts agree with scikit-learn's own metrics.8- evade(): one mutable feature, on a copy. check f06: PASS.910## Measured11- Threshold 0.3: precision 0.76, recall 0.93. 0.5: 0.88, 0.73.12- Every one of 28 caught malicious rows evaded by one feature.13- The search starts at 0: entropy 0 and length 0 are not14  command lines. With entropy padded no lower than 4.0,15  16 of 28 still evade.16- Immutable features only: precision 0.44, recall 0.87.17- 20 poisoned labels: recall 0.93 -> 0.70.1819## For Module 620- Bound each mutable feature to values an attacker can reach.
mainMarkdownUTF-8Ln 10, Col 1

The correction is the lesson's main finding about method. The draft met its contract, the check passed, and the evasion rate it produced overstated the threat because the contract's search allowed values no attacker could produce.

Every security measurement on a model depends on the threat model built into it: which features the attacker controls, over what range, at what cost. Module 6 states those constraints before measuring, and so should every claim about how hard a detector is to evade.

Do

Practice

A detection classifier, evaded
the lab

Brief your assistant with the contract and the instruction to measure honestly, check the draft, then choose a threshold, bound the evasion to realistic values and record the rate.

Visual Studio Code0.6.md - sec410-lab
0.6.md
practice › 0.6.md
1# Practice 0.6: a detection classifier, evaded23## Done when4- [ ] lab/foundations/ml.py meets its docstring's contract5- [ ] python -m lab check f06 prints PASS6- [ ] the threshold sweep run; your chosen threshold and why7- [ ] evasion rerun with each mutable feature bounded to values8      a real command line can take; rate recorded9- [ ] records/0.6-<assistant>.md: brief, draft, runs, review1011## Constraints12- Every metric comes from the test split, never the training.13- The split's seed is recorded with every number.1415## Stretch16- Retrain with poisoned labels and find the fewest that drop17  recall below 0.8 at your threshold.
mainMarkdownUTF-8Ln 3, Col 1

A detection model learns weights from labeled features, scores each input, and a threshold turns the score into an alert; precision and recall trade as the threshold moves, and choosing it is a decision about costs.

A model relies on whatever separates the data, including features an attacker controls, and every caught malicious row here could be evaded by one of them. Realistic bounds turned that into a smaller, true number. Removing attacker-controlled features stopped the evasion and raised false positives from 9 to 33; poisoned labels cut recall while making the alerts look cleaner.