0.3 The Corpus Your Rules Must Not Match

A rule has two jobs and testing it needs two populations. The specimens from Section 0.2 tell you whether a rule catches what it should; they cannot tell you whether it catches things it should not, because everything in that directory is something you want matched. For that you need the opposite: a body of files you have good reason to believe are benign, large enough that a coincidence has room to happen. This section assembles one, explains why the size and the composition both matter more than people expect, and shows what a corpus tells you that no amount of reading a rule ever will.

What a corpus is for

A clean corpus answers one question: how often does this pattern occur in software nobody is worried about? That number is not available from reading the rule, and it is frequently very different from what the rule's author expects.

It answers a second question implicitly, and this one is easy to miss. It tells you whether the pattern could occur at all in the population your rule will run against, which is the same information the negation from Section 0.1 provides at scan time. A pattern absent from four hundred clean files might be genuinely rare, or it might be one that could never have appeared there, and distinguishing those is the subject of the composition section below.

The reason is that human judgment about how distinctive a string looks is close to worthless. A sequence that seems specific to your sample turns out to be a compiler artifact, a library string, or a piece of boilerplate shared by everything built with the same toolchain. There is no way to know without counting, and counting takes one command.

yr scan common.yar /usr/bin/ | wc -l
628

Six hundred and twenty-eight of the roughly one thousand files in that directory contain the pattern. Whatever the rule's author believed they were describing, they were describing most of the operating system.

Notice how little the rule's appearance told you. It is a single well-formed pattern in a four-line rule that compiles without a warning, and yr check has nothing to say about it, because the linter reports whether a pattern will make a scan slow rather than whether it will make a scan useful. The two questions are genuinely separate, and only one of them has a command that answers it automatically.

That is the whole value of a corpus in one number, and note how cheap it was. No judgment, no argument about whether the string looks distinctive, no waiting for production to tell you. One scan of a directory that was already on the machine.

Size, and why small corpora lie

The instinct is that a few dozen known-good files should be enough to catch an obviously bad pattern, and it is wrong in a specific way worth understanding.

Where the instinct comes from is reasonable enough. If a pattern is bad, the reasoning goes, it will be bad obviously, so a handful of files will expose it. That holds for the worst patterns and fails for the ones that actually cause trouble, which are not obviously bad at all: they are patterns that occur in a small but real fraction of ordinary software, and a small fraction of a small corpus rounds to zero.

A corpus tells you a rate. A rule matching one file in two thousand is a rule that will fire several times a day across a large estate, and against a corpus of thirty files it will almost certainly match nothing at all and report a clean bill of health. The rate you are trying to measure is smaller than the resolution of a small sample, so a small corpus does not give you an imprecise answer. It gives you a confident wrong one.

A few thousand files is a reasonable working minimum, and the practical point is that you almost certainly have them already. A Linux system directory holds a thousand binaries. A Windows System32 holds several thousand. Neither needs assembling; they need pointing at.

Scanning that many files is also faster than people expect, which removes the last excuse for not doing it. A simple rule over a thousand binaries completes in a couple of seconds on an ordinary machine, because the pattern search is one pass and the engine is built for exactly this. The cost of a corpus test is not the scan; it is the thirty seconds of deciding to run one.

Composition matters more than size

The trap that catches careful people is a corpus that is large and unrepresentative.

A corpus is a sample of the population your rule will actually run against, and a measurement against the wrong population is not a weaker result, it is a misleading one. Test a rule aimed at Windows malware against a thousand Linux binaries and you will get zero false positives, because the corpus structurally cannot contain the constructs your rule describes. That zero looks exactly like the zero you would get from a genuinely precise rule, and it means something entirely different.

The same problem appears in subtler forms. Distribution binaries are stripped of debugging information, so a rule keyed on a compiler artifact scores zero against them and would score close to a hundred percent against a build machine's output. A corpus of system utilities contains almost no network clients, so a rule keyed on an HTTP user-agent string sails through it. In each case the corpus was clean, the scan was correct, and the conclusion drawn was wrong.

The habit that prevents it: when a result surprises you by being better than expected, ask what the corpus was capable of punishing before you accept the number. A pattern scoring zero is either rare or structurally absent, and those are very different findings.

Surprise is the right trigger to train, and the direction matters. A disappointing number sends you back to look at the sample, which is a productive place to be. A flattering one sends you to commit the rule, which is why the flattering result is the more dangerous of the two and deserves the extra minute of checking.

Assembling one

Start with what is on the machine and add to it deliberately.

mkdir -p ~/yara/clean
cp -d /usr/bin/* ~/yara/clean/ 2>/dev/null
ls ~/yara/clean | wc -l
1069
#
find ~/yara/clean -type f | wc -l
795

That is a working corpus in one command, and the -d matters more than it looks. Without it cp follows every symlink and writes a full copy of each target, which turns 274 links into 274 additional real files. Your corpus would then hold 1,068 files rather than 795, and every figure you measure would disagree with every figure in this course.

795 is the denominator, because that is the number of files a scan examines. ls counts directory entries and 274 of them are links that yr does not follow. Take the number from the tool:

yr scan -c rule.yar ~/yara/clean/ | wc -l
795

Three things improve the corpus, in order of value.

Match the file type to your rules. If you write rules for Windows executables, a Windows corpus is not optional. Copy a System32 directory from a machine you trust, or from a clean virtual machine image, and keep it separate from the Linux one so you can scan either deliberately.

Keeping them separate matters more than it sounds. A merged corpus produces a single number that averages two populations, and an average across populations your rule treats differently is not a measurement of anything. Two directories and two scans cost one extra command and give you two interpretable numbers.

Add the software your organization actually runs. The false positives that matter are the ones on your estate, and a generic corpus of operating system files will not contain your line-of-business application, your deployment tooling or your endpoint agent. Those are exactly the things a broad rule fires on.

This is also the corpus addition that pays back fastest, because bespoke and niche software is where coincidental matches concentrate. Operating system files are written by people with enormous resources and reviewed heavily; a vendor tool with two hundred customers is not, and it carries the unusual strings, odd section names and embedded blobs that a broad rule finds interesting.

Keep the results rather than the files. A rule's false-positive count is interesting compared to what it was last week and to what the previous version of the same rule scored. Save the scan output alongside the rule and Module 6's tuning work becomes a comparison rather than a fresh measurement each time.

A corpus is also something you maintain rather than build once. Software gets installed, versions change, and a corpus assembled two years ago describes an estate that no longer exists, so a rule measured against it carries a number that was true then. Refreshing it is cheap and the discipline is the same as any other reference data: date it, and treat an undated measurement as unverified.

Record the corpus size alongside the count as well, because a bare number is uninterpretable later. "Three false positives" means something quite different against four hundred files than against forty thousand, and six months on nobody remembers which corpus was in use.

Reading a corpus result

Three outcomes and three different responses.

Zero matches. Provisionally good, and check the mechanism before believing it. Was the corpus capable of containing this pattern at all? Run the rule against a file you know matches, so you have proved the rule can fire, and Section 0.1's control applies here unchanged.

A zero is also the result most likely to be reported without qualification, which is why it deserves the most scrutiny. Nobody writes "the rule matched four hundred files" in a rule's notes and moves on; a zero invites exactly that. Record what the corpus was, how many files it held, and the date, so the zero can be interpreted later by somebody who was not there.

A handful of matches. Frequently the most useful result. Look at what matched: if it is three related files from one vendor, you have found something specific about your estate and an allowlist entry, rather than a broken rule. If it is three unrelated files, the pattern is more common than you thought.

This is where --print-strings from Section 0.1 earns its place. Knowing that three files matched is a count; knowing which pattern fired and at what offset tells you whether the rule matched for the reason you intended, and those two are frequently not the same.

Hundreds of matches. The pattern describes a platform rather than a program. No amount of gating or quantifying rescues it, and the response is to go back and choose a different pattern, which is Module 3's subject.

Resist the instinct to tune at this point, because it is the wrong response to this particular result and it is the instinctive one. Adding a file-type gate, bounding the size, or requiring a second pattern alongside it will reduce the count, and the underlying problem is untouched: the pattern still describes the platform, so the rule now describes the platform with extra steps. A pattern matching hundreds of clean files is telling you something about the pattern, and the only useful response is to pick another one.

Corpus guide: build one and measure a deliberately bad rule

Step 1, assemble the corpus. Copy a system binary directory as above and confirm the count. A thousand files is a workable starting point.

Step 2, write a rule you expect to be bad. Pick a string you believe is common, a library name or a format specifier, and write a single-pattern rule around it. Run yr check first.

Step 3, count the matches with yr scan piped to wc -l, or with -c if you want per-file counts.

Step 4, write a rule you expect to be good using the mutex identifier from your Section 0.2 family, and count that against the same corpus.

Step 5, compare the two numbers and note that both rules are four lines, both compile clean, and only the measurement distinguished them.

Section 0.4 covers where the rules you write are actually run, which changes what a good rule looks like more than anything else in this module.