0.3 The Environment You Will Build

Two engines, on purpose

Most courses have you install one tool. This one has you install two, and the reason is Section 0.2's argument: you will author on YARA-X and your rules will frequently have to run somewhere that is still on 4.x. You cannot check that from one side of the line.

The installation is not the interesting part. What matters is that from Module 1 onward you have a way to answer "does this compile on both?" in a second, because that question turns out to decide a surprising number of authoring choices.

It also changes how you read an error. When one engine rejects a rule and the other accepts it, the useful question is not which engine is right but what the disagreement reveals about the rule, and that is almost always something you wanted to know.

Installing YARA-X

The binary is yr, not yara. That trips people who have used YARA before, and it is deliberate: the two can coexist on the same machine without either shadowing the other.

Coexisting matters more than it sounds. Through Module 6 you will want both available, and a rename that forced you to choose would make the compatibility work considerably more awkward than it needs to be.

# Release binaries: github.com/VirusTotal/yara-x/releases
# Or via cargo if you have Rust:
cargo install yara-x-cli
#
yr --version
yr scan --help | head -20

Three subcommands matter for this course. yr scan runs rules against files. yr fmt reformats a rule file to a canonical layout. yr check validates rules without scanning anything, which is the one you will use most while writing.

yr scan also takes --output-format with text, json and ndjson. The last of those, one JSON object per line, is what you want when the output is going into another tool rather than onto your screen, and Module 7 uses it.

The formatter is worth adopting early rather than at the end. A rule set written by four people in four layouts is harder to review than one that has been through fmt, and running it before every commit removes an entire category of pointless diff.

There is also a config file at ${HOME}/.yara-x.toml in TOML format, currently controlling the behavior of fmt and check. Module 5 covers it. If the file does not exist you get the defaults, which are reasonable.

Installing YARA 4.x alongside it

You need this to answer the compatibility question, and the easiest route is the Python bindings rather than building the C library.

pip install yara-python    # the 4.x engine
pip install yara-x         # the YARA-X engine, same interface shape
#
python3 -c "import yara, yara_x; print('4.x:', yara.__version__); print('yara-x: ok')"

4.x: 4.5.4
yara-x: ok

That gives you both engines callable from one script, which is what the next section builds.

If you would rather have the real 4.x command-line tool as well, it builds from source or comes from most package managers, and Module 6 uses it once to show that the Python bindings and the CLI agree. Nothing in the course requires it.

The check you will run constantly

This is the single most useful thing in the environment, and it is about fifteen lines. It takes a rule and tells you what each engine thinks of it.

import sys, yara as y4, yara_x as yx
#
def both(src):
    for name, fn in (("YARA-X", lambda s: yx.compile(s)),
                     ("4.x   ", lambda s: y4.compile(source=s))):
        try:
            fn(src)
            print(f"{name}: ok")
        except Exception as e:
            print(f"{name}: {str(e).splitlines()[0]}")
#
both(open(sys.argv[1]).read())

Save it as both.py. Point it at a rule file and you get two lines back.

It is worth understanding why this is so short. The two Python bindings expose the same shape of interface even though the underlying implementations share no code: compile a source string, get an object, scan bytes with it. The APIs are not compatible in the sense that you cannot swap one for the other in a program, but they are similar enough that a wrapper like this is trivial. That is the difference the YARA-X documentation is describing when it says migration is usually simple but not automatic.

What it looks like when the engines disagree

Worth running once now, before you need it, so the output is familiar when it matters. Here is a rule with a regular expression containing an escape that is not valid.

rule Escape_Demo
{
    strings:
        $path = /C:\\Users\\[a-z]+\\Release\\loader\.pdb/
    condition:
        $path
}

That one compiles on both, because the backslashes are escaped properly. Now the version somebody actually writes by accident, with \R instead of \\R:

$ python3 both.py bad_escape.yar
YARA-X: error[E014]: invalid regular expression
4.x   : ok

YARA 4.x accepts \R and quietly treats it as a literal R, so the pattern silently becomes something you did not write. YARA-X refuses to compile it. The strict engine is catching a real bug in the rule, which is the pattern for almost every incompatibility in Module 6: 4.x was permissive in ways that hid mistakes.

The same thing happens with short base64 patterns:

$ python3 both.py short_base64.yar
YARA-X: error[E024]: invalid pattern `$a`
4.x   : ok

YARA-X requires at least three characters for a base64 pattern. In exchange it does not produce the false positives 4.x generates, where the way encoded strings are trimmed makes two different plaintexts share an encoding.

The documented example is a good one to keep in mind, because it looks impossible until you see it: after trimming, one base64 encoding of "Dhis program cannow" is identical to one encoding of " This program cannot". A rule keyed on the second will fire on files containing the first, and nothing about the rule looks wrong.

Both of those errors are reproduced in Module 6 with the constructs that cause them. The point of meeting them here is only that you recognize the shape when your own rule produces one.

The clean corpus

You need files a rule should not match, and you need a lot of them. This is the part people skip and it is the part that makes testing mean anything.

# On the lab VM, this is already a usable corpus
ls /usr/bin /usr/lib/x86_64-linux-gnu | wc -l
#
# Copy a Windows System32 sample across if you have one available,
# because most rules in this course target PE files
mkdir -p ~/yara/clean

A few thousand files is enough to be informative. What matters is that they are files you have some reason to believe are benign and that they resemble the population your rule will actually run against. A corpus of Linux binaries tells you very little about a rule targeting Office documents.

The second half of that is the part that catches people. A corpus is not a generic quality bar you clear once. It is a sample of the population you are making claims about, and if your rule targets signed vendor software then a corpus of unsigned Linux utilities cannot tell you whether it over-reaches. Module 5 covers assembling one per rule family rather than one for everything.

What the environment does not include

No malware. Every specimen in this course is built in Section 0.6 from software you already have. That is not a safety compromise; it is a teaching choice, because you can only reason cleanly about a false positive when you know with certainty what the file is.

There is a second benefit that becomes obvious in Module 2. You can rebuild a specimen with one thing changed, which is the only way to demonstrate that a pattern survives a recompile rather than asserting that it does.

No sandbox or detonation environment. This course reads files, it does not run them. If you need behavioral evidence, that is FOR203's territory and it has the environment section for it.

The distinction is worth holding onto. YARA answers "is this file the same thing as that one", and it answers it from bytes on disk. It does not answer what a file did, and no amount of rule sophistication changes that, which Section 0.5 goes into properly.

No THOR or commercial scanner. Module 7 covers deploying to them and explains what each expects, but the course does not assume you have a license for anything.

Verify before you continue

Section 0.4 covers where the rules you write here actually end up running, which changes what counts as a good one.