0.4 Installing the Tools and the Lab Pack

Introduction

This lesson installs everything Lesson 0.3 described and gets the lab running: Visual Studio Code and the Logic Apps extension, Python and Node.js, and the lab pack with the harness and its data. It ends with the harness checking itself and all three lab processes started in the right order, ready for the first run in Lesson 0.5.

Allow an hour, most of it waiting for downloads. The extension's dependencies are the largest download, and they arrive in the background after the extension installs. The steps are the same on Windows, macOS and Linux except where the lesson says otherwise. Nothing here needs an Azure account, and nothing installs anything outside your own user profile except the tools themselves.

The extension brings most of the tools with it, so it goes first:

code --version

Azure Logic Apps (Standard) > Install

~/.azurelogicapps/dependencies/

Project Runtime: ~4
Auto Start Azurite: on

Visual Studio Code itself comes from its download page and installs like any application. The extension is installed from inside it: Learn's steps open the Extensions view, search for azure logic apps standard, and select Azure Logic Apps (Standard) and Install. When it finishes, it appears in the installed list.

The extension then installs its dependencies. Learn's page for the older single-tenant steps describes what happens from version 2.81.5 onward: a dependency installer puts Azure Functions Core Tools, the .NET SDK and Node.js in a folder of their own, under .azurelogicapps in your home folder, and leaves any versions you already had alone.

That means the extension cannot break another project that uses different versions, and it also means the tools it installed are not on your PATH: they are for the extension's use.

Reload Visual Studio Code when it asks, then open the extension's settings, under File, Preferences, Settings, and the User tab's Extensions section. Learn asks you to confirm that Project Runtime is ~4. Turn on Auto Start Azurite if it is off, and when Visual Studio Code asks where Azurite should keep its data, accept the default folder Learn names, .azurelogicapps/.azurite in your home folder.

On Windows, the first time the runtime starts, Windows Defender Firewall may ask whether Visual Studio Code and func.exe may communicate. Learn's advice is to allow private networks only, which is all the lab needs: every call goes to your own machine.

If Visual Studio Code is already installed with other Azure extensions, nothing about them needs to change. The Logic Apps (Standard) extension works beside the Azure Functions and Azure Account extensions many people already have, and if it needs one of them it says so when it installs.

If you have an old version of the Logic Apps extension, update it from the same Extensions view before going further, because the dependency installer Learn describes only exists from version 2.81.5, and an older extension expects you to install the dependencies yourself.

You can see what the dependency installer did by opening the dependencies folder: one folder per tool, each with the version it installed. If a later lesson's runtime behaves differently from the course, that folder is where to read which Core Tools and .NET your extension is using.

Check that Visual Studio Code answers from a terminal with code --version, as the procedure shows. If the command is not found on macOS, Visual Studio Code's command palette has a command to install it into your PATH.

The harness and run-spec need Python, and the harness's query engine needs Node.js on your PATH, separately from the copy the extension installed for itself. Check whether you already have them:

# Windows (PowerShell)
py --version
node --version

# macOS and Linux
python3 --version
node --version

If either answers with an error, install it. Python comes from python.org or your system's package manager; on Windows, the python.org installer adds the py launcher shown above. Node.js comes from nodejs.org, where the long-term support release is the one to choose. The course was built with the versions Lesson 0.3 lists, and anything at or above them works.

On macOS and Linux the commands are python3 and node; on Windows they are py and node. The lessons use python3, so on Windows read py wherever a lesson says python3.

The harness needs nothing else. It uses only Python's standard library, so there are no packages to install and no virtual environment to create, and it calls node only to run the query engine in the pack.

If node is missing, the harness still starts and answers every call except queries, which it answers with an error saying the query engine is not available. That message, the first time a workflow queries Log Analytics in Module 4, means Node.js is not on the PATH of the terminal the harness was started from.

Two installs of Node.js on one machine, the extension's and yours, are normal and do not conflict. The extension uses its own copy from its dependencies folder for inline code in workflows, and finds it through its own settings; the harness uses whichever node your terminal finds first on the PATH. Neither looks at the other, and updating one does not change the other.

The same is true of Python. Many systems have more than one version installed, especially macOS and Linux, where the system itself may depend on an older one. The harness only needs the one your terminal finds as python3, or py on Windows, to be recent; it does not need to be the system's default for anything else.

If you install Node.js while the harness is running, stop the harness and start it again from a new terminal, so that it sees the updated PATH. The same is true of Visual Studio Code's own terminals, which take their PATH from when Visual Studio Code started.

The lab pack is one zip file, 40.9 MB, from the course's download link at /downloads/lab-packs/au-lab-pack.zip. Verify it before unzipping it, by comparing its SHA-256 hash with the one below:

sha256sum au-lab-pack.zip
7ad1a243c7f5112069e88925d7a5394f2c29fafda153801d903f8ed2fe21329a  au-lab-pack.zip

That is the command on macOS and Linux, run in the folder the pack was downloaded to. On Windows, PowerShell has its own:

Get-FileHash .\au-lab-pack.zip -Algorithm SHA256

The hash is the pack's fingerprint. If yours matches, the file is exactly the one the course was built with, byte for byte; if it does not, the download was interrupted or altered, and the right response is to download it again rather than to unzip it and hope. On Windows, Get-FileHash prints the hash in capitals, which is the same value.

The pack is built the same way every time: its files are stored uncompressed, in a fixed order, with fixed dates, so the same files always produce the same hash.

That is why a hash can be printed in a lesson at all, and why a later version of the pack, with a changed harness or new data, has a different one. If a lesson's hash and the download's differ, the pack has been updated since you read the lesson, and the course's download page has the current hash.

Download the pack from the course while signed in; the link checks that your subscription includes the course before it serves the file. The download is a single request, so a slow connection only makes it take longer, and if it is interrupted the hash will not match and you simply download it again.

Unzip it anywhere you like; the lessons assume a folder called au-lab-pack in your home folder, created by unzipping it there. Keep it separate from your Logic Apps project folder. The pack is the lab's equipment, the project is your work, and keeping them apart means you can replace the pack with a newer one without touching your playbook.

Hash checks are a habit the course uses everywhere. The template pack in Module 9 ships with its own hashes, Lesson 11.6 regenerates the files a deployment ships and fails if they differ from what is committed, and the handover in Module 12 is regenerated and compared rather than edited.

Verifying the lab pack is the first instance of a rule that runs through the whole course: a file is what its hash says it is, not what its name says.

Inside is a second check: a file called SHA256SUMS lists the hash of every file in the pack, and sha256sum -c SHA256SUMS run in the pack's folder checks them all, which is useful if you suspect a single file was changed later.

The pack holds 30 files, in these folders:

{
  "au-lab-pack, files by folder": {
    "README.md": 1,
    "SHA256SUMS": 1,
    "lab-scripts/verify/run-query.js": 1,
    "public/data/ne-corpus": 16,
    "public/js/kql-engine": 3,
    "scripts/au/harness": 8
  }
}

The harness folder holds the harness, run-spec, a token map that shows the claims in any token the harness issues, the secret scan Module 3 uses, the permission matrix tool, the self-test, and the capstone's cases.

The engine is the course's KQL query engine, the same one the browser console beside each lesson uses, and the corpus folder holds the tables the harness reads, with the watchlists the enrichment queries look up.

The folders are laid out as they are in the course's own repository, which is why the harness is three folders deep. The harness finds the engine and the data by their paths relative to itself, so moving the harness folder out of the pack breaks its queries. Move the whole pack if you need to; never part of it.

There are no workflows in the pack. The workflows are what you build, starting in Lesson 0.5, and the course's reference versions are shown in the lessons rather than shipped, because a playbook copied from a file teaches nothing. The deployment files for Module 11 are in the template pack, a separate download that Module 9 introduces.

The data files are the largest part of the pack, and they are plain JSON: one file per table, each an array of rows with the same columns the tables have in a Sentinel workspace.

You can open one in Visual Studio Code to see what a sign-in record or a process event looks like, which is a useful thing to do once before Module 4 starts querying them. Do not edit them. A changed row changes what the harness answers, and the lessons' results stop matching yours.

The self-test and run-spec do not depend on your workflows, so the pack can be checked on its own before you have built anything, which is exactly what the next section does.

The README in the pack lists the commands this lesson uses, so the pack can be used on its own once you know the course.

Before using the harness, run its self-test from the pack. It starts its own copy of the harness on a free port, calls every operation, injects every fault and checks every answer:

$ cd au-lab-pack/scripts/au/harness
$ python3 selftest.py
ok   bad secret: 401
ok   no token: 401
ok   reader cannot isolate: 403
ok   comment required: 400
...
ok   reset restores the device: Workstations
ok   reset restores the incident: True
all passed

72 checks, each a call the harness must answer with a particular status or value: a token request with a bad secret refused, an isolation by an identity that may only read refused with 403, a device reset to its starting state.

The last line is all passed. Anything else means something in your copy of the pack or your Python is not as the course expects, and the line that failed names it.

The checks run against the same data and identities your workflows will use, so a pass also confirms that the data files unzipped intact and that the engine can read them. A pack with a damaged table would fail the query checks here rather than in the middle of Module 4.

The self-test is the fastest way to tell a problem in your lab from a problem in your workflow. If it passes and a workflow fails, the workflow is the place to look; if it fails, nothing a workflow does will be reliable until it passes.

It also shows what the harness is capable of, in a page of output: tokens, permissions, machines, machine actions, users, queries, approvals and every fault. Reading it once is a quick tour of everything the playbook will call over the next ten modules.

The self-test runs with Python's isolated mode turned on for the harness it starts, so it ignores any Python packages or environment variables on your machine that might change how the harness behaves.

A pass therefore means the harness works as shipped, not that it happened to work with your particular setup. If it passes in a new terminal and something still behaves oddly in your usual one, the difference is in that terminal's environment.

Run it again whenever you update the pack, or if the harness starts behaving differently from the lessons. It takes under a minute and leaves nothing running.

From Lesson 0.5 onward, every session starts the same three processes in the same order:

Azurite Blob service listening on 127.0.0.1:10000

cd au-lab-pack/scripts/au/harness
python3 harness.py --port 8642

curl http://127.0.0.1:8642/_harness/assumptions

Functions host listening on http://localhost:7071

Azurite comes first because the runtime needs storage to start. With Auto Start Azurite on, the extension starts it when you open a Logic Apps project, and its terminal shows it listening; Learn's Azurite page gives the Blob service's default port as 10000 on 127.0.0.1.

The harness comes second, in a terminal of its own, from the pack's harness folder. It prints one line, the address it is listening on and a reminder that it is not a Microsoft service, and then waits.

Leave that terminal open: when a workflow calls it, nothing appears there, but if it stops, every call fails. The check step asks it for its assumptions list, which answers whether the harness is up without involving a workflow.

The runtime comes last, started by Visual Studio Code's Start Debugging with your project open. Lesson 0.5 creates the project. When it is up, its terminal lists the workflows it found and the local address each one's trigger listens on, which is the confirmation that all three processes are running and can reach each other.

Each process has its own terminal, and it is worth naming them, because three terminals that look alike are easy to confuse. Visual Studio Code's terminal panel can hold several and lets you rename each one: storage, harness and runtime is a good set of names. Closing a terminal stops the process in it, which is the quickest way to stop one process without touching the others.

You do not need to restart everything for every change. A change to a workflow needs only the runtime restarted, which Visual Studio Code does when you stop and start debugging. A change to the harness's options needs only the harness restarted. Azurite can stay up for days, and usually does.

From Module 3, the harness is started with one more option, --strict-tokens, which makes it check every token's audience and permissions. Module 3 explains when and why. Until then, the command above is the one to use.

Most problems in a first setup are one of a few, each with a message that names it:

{
  "what you see": {
    "refused on port 10000": "Azurite is not running: start it, then the runtime",
    "address already in use": "another program has the port: stop it or move it",
    "Query engine not available": "node is not on your PATH: install Node.js",
    "a workflow named invalid": "its workflow.json has an error: read the message",
    "401 from the harness": "from Module 3: a token missing or for another service"
  }
}

Read the messages in the order they appeared. The runtime in particular prints a burst of lines when it starts, and when something is wrong the later lines are usually consequences of the first: a storage error early on is followed by workflows that could not be loaded, which are not separate problems.

A refused connection on port 10000 is the most common: the runtime started before Azurite. Start Azurite, then stop and start the runtime. An address already in use means something else, often an earlier copy of the harness or the runtime, still holds the port; closing the old terminal usually frees it.

The query engine message is the Node.js PATH problem from Section 2, and only appears when a workflow queries Log Analytics. A workflow the runtime names as invalid has a mistake in its JSON, and the message says which workflow and usually which action.

A 401 from the harness before Module 3 means a call sent a token the harness could not read; from Module 3 it means the token is missing, expired or for another service, which is a lesson in itself.

When the message does not match any of these, run the harness's self-test first, then read the runtime's terminal from the top. The first error printed is usually the cause, and the ones after it are its consequences.

Two quieter problems do not produce a clear message. A run that starts and then sits in Running for a long time is usually waiting for something: from Module 8 that is an approval, which in the lab the harness answers when a case tells it to, so a run started by hand will wait until its approval expires.

And a workflow change that seems to have no effect usually means the runtime was not restarted after the file was saved; the runtime reads definitions when it starts.

If you get stuck, the lab can always be reset. Stop all three processes, delete Azurite's folder to clear the run history, and start them again in order. Your workflows are in your project folder and are not touched. Nothing in the lab keeps state you would miss.

Install the tools, verify the pack, run the self-test and start all three processes.

Lesson 0.5 creates your first Logic Apps project, runs a workflow that calls the harness, and reads what the run recorded.