0.1 What This Course Is For

Introduction

This course trains you to secure applications built on large language models: assistants that answer from an organization's documents, agents that work through tasks with tools, and the platforms both run on. It is an engineering course. You review an application's architecture, prove each weakness that can be exploited, build the control that closes it, and show with evidence that the control holds without breaking the application's real work.

This lesson sets out the job the course prepares you for, the principle every module rests on, the loop each module repeats, the two applications you will work on throughout, where you practice, and what you will be able to hand over at the end.

The course is written for one job: the engineer who has been handed an LLM application or an agent and asked to make it secure. In practice that person is a security practitioner, an application developer or a cloud engineer.

The title varies; the work does not. Before release, they decide whether the design is safe to ship and what must change. After release, they keep it safe as the application, the model behind it and the threats against it change.

By the end of the course you will be able to explain four things about any LLM application, with evidence for each. What can it access: which documents, which records, which systems, as which identity. What can it do: which tools it can call and what each one changes.

How are those boundaries enforced: by what code, at which point, independent of what the model decides. And what evidence shows they hold: which tests, run how many times, against which attacks, with what effect on the work the application exists to do.

The bad answers in the specimen are not lies. Each describes what the team intended, and each would pass a design review that only asks about intentions. The good answers name a mechanism, a place in the code where it runs, and a measurement. That is the difference the course trains: from describing what an application is meant to do to showing what it can do and what stops it.

Notice also that the good answer to the first question is worse news than the bad one. The index can read every tenant's documents. An honest review often makes the application look less safe before it makes it more safe, because it replaces assumptions with facts, and some of the facts are uncomfortable. A reviewer who only reports what holds is not reviewing.

Those four questions are the spine of every review you will carry out. Testers and incident responders use the course too, and their work appears where the engineer needs it: proving a weakness, evaluating a release, answering an incident. But the course always comes back to the engineer's question, which is not whether an attack works but whether the application can be trusted when it does.

Every module rests on one principle: the model will sometimes follow text it should have ignored, so the boundaries around what the application can access and do must stay enforceable when it does.

The principle is not pessimism about models. It follows from how they work. The entry on prompt injection in the OWASP Top 10 for LLM Applications 2026 notes that models make no architectural distinction between instructions and data, so there is no clean equivalent to the parameterized query that fixed SQL injection. Model providers say the same about their own systems.

Anthropic's report on prompt injection in browser use, published on 24 November 2025, calls it "far from a solved problem", and the best result it reports, about one percent of attempts succeeding against an adaptive attacker with its safeguards in place, is one provider's measurement of its own models under its own test.

The service-desk agent will run on that model. It reads customer tickets, can
send email from the support address and looks up customers as a service
identity. The vendor's figure comes from its own test set.

Not on that evidence. One percent of attempts getting through is a lot when
anyone can open a ticket and try as often as they like, and a vendor's test
set is not your application. The question is what the agent can do when an
attempt gets through: send anywhere and read every customer, as built.

One percent sounds small until you remember who gets to try. An application that reads tickets, emails or web pages lets anyone who can write those try as often as they like, in any wording, against the same model every time. A boundary that holds ninety-nine times in a hundred is a boundary that fails on a schedule the attacker controls.

There is a second reason. The model is the part of the application you control least. You do not train it, you cannot read its weights, and the provider will replace it, sometimes with a version that follows injected text more readily or less.

A boundary that depends on the model's judgment has to be re-established every time the model changes. A boundary enforced in your own code stays where you put it, and you can test it the same way after every update.

So the course teaches two kinds of control and keeps them apart. Some controls hold whatever the model does: a search that only returns what the asker may read, a send that only reaches the ticket's own customer, a tool that acts with the user's permissions rather than a service account's.

Others lower the odds: marking untrusted text, filtering inputs, choosing a more resistant model. The first kind is the boundary. The second kind is measured, honestly, as a layer on top.

Each teaching module secures one part of an application, and each one runs the same loop.

Review is reading the part of the design the module covers: what the application reads there, who writes it, what it can do and as whom. Prove is showing that a weakness can be exploited, in the browser range or the lab, with evidence another person could repeat.

Fix is building the control in code, in the lab's copy of the application, at the point where it holds whatever the model is persuaded to do. Verify is running the module's check, which measures the attacks and the application's normal work together.

The last step is where most security testing of AI applications goes wrong. A control that blocks every request stops every attack and makes the application useless, and a test that only counts blocked attacks rates it perfect.

The course's checks count both: an attack that gets through fails the check, and so does a legitimate task the control broke. You will see in lesson 0.5 how a check reports both, and in lesson 0.6 how the loop runs on a real weakness.

The dashed line in the figure is part of the loop too. A control verified today can be undone by a new tool, a wider identity or a model update that changes how often an attack is followed.

Module 11 turns each module's check into a test suite that runs every time the application changes, so that a control you verified in Module 3 is verified again, automatically, when someone adds a tool in month six.

The course does not teach on a different toy for every lesson. It teaches on two applications, which you meet in the browser range, run in the lab, investigate in the recorded logs and review in the final exam. By the end you will know them the way you know a system you own.

The knowledge assistant answers staff questions for two client organizations from one shared index. It is the course's subject for everything about data: who may see which document, how documents get into the index, and what happens to an answer once it leaves the model.

As built, it has the weaknesses many real retrieval systems have, starting with an index that can read everything and a search that never asks who is asking.

The service-desk agent works a support queue for engineers. It reads tickets that anyone can open, follows links in them, keeps notes between sessions and acts through tools, including tools served over the Model Context Protocol. It is the course's subject for everything about action: what the agent may do, as whom, with whose approval, and what stops a ticket from steering it.

Neither application is exotic. Each is assembled from the parts most teams use: a hosted model, a vector index, a few tools and a service identity. Lesson 0.2 traced one request to each. You will find the same weaknesses again in Modules 2 to 9, prove each one, and close it.

The course gives you three places to work, because the job has three kinds of work in it. The three are not interchangeable, and the course uses each for what it does best. The range is fast and graded but simulated: its model follows written rules, so a result there shows that a design is weak, not how often a real model would fall for it.

The lab is slower but real code: a fix there is a fix you could carry into your own application, and its check is the same kind of test you would put in a release pipeline. The recording is the only place you see an application over time, with normal traffic around the attack, which is what an investigation actually looks like.

The AI Range runs in the browser. It simulates both applications with a model that follows written rules, so every result can be explained and every exercise graded. You use it to prove a weakness and to compare designs quickly: plant an instruction where an attacker could write, run the application, and read the trace of what happened at each step.

The SEC502 lab runs on your own machine. It is working Python code for both applications, with deliberate weaknesses, a mock model that follows the same rules as the range, and local mock services that record every email sent and every ticket changed.

Nothing in it calls a model provider or any service outside your machine. You fix weaknesses there by changing the code, and you verify each fix with the module's check. An optional adapter runs the same checks against a real open-weight model on your machine, for the repeated trials that show how a real model varies.

The AI App Range is a recorded month of logs from the two applications, queried in the platform's console. Module 12 uses it to investigate an incident from the first suspicious request to the containment.

The twelve teaching modules follow the parts of an application, from the threat model to the incident. The order is deliberate. You cannot judge a control without knowing what it protects, so the threat model comes first. Injection comes next because it is how most of the later attacks arrive: a tenant leak, a poisoned answer, an unwanted send each begins with text the model should not have followed.

Identity comes before tools because a tool is only as dangerous as the identity it acts as, and fixing tool permissions without fixing the identity behind them moves the problem rather than closing it. Evaluation and response come last because they depend on everything before them: a regression suite is the collected checks of Modules 2 to 10, and an investigation needs the logging those modules added.

Module 1 builds the threat model that the rest of the course works from. Modules 2 to 5 secure what goes into the context and what comes out: injection, sensitive data and tenant isolation, retrieval and ingestion, and output handling.

Modules 6 to 9 secure what an agent can do: its identity, its tools and their protocols, the containment of its actions, and its memory and its dealings with other agents. Module 10 covers the supply chain of models, packages and connectors. Module 11 turns the checks into an evaluation suite that gates a release, and Module 12 investigates and contains an incident.

The OWASP Top 10 for LLM Applications 2026 and the Top 10 for Agentic Applications 2026 are used as coverage checks rather than as the organizing frame. Each module closes by mapping its controls to the entries it covers, so that by the end you can show which entries your design addresses and which it leaves open.

MITRE ATLAS and ATT&CK v20 appear where a finding needs to be named for a red team or a SOC.

The course ends with a security review of one of the two applications, the same kind of review you would deliver at work.

Each item in the pack answers one of the four questions from section 1. The threat model and the permission matrix say what the application can access and do. The fixes say how the boundaries are enforced.

The findings and the regression results are the evidence. The monitoring plan says how you would know if a boundary failed, and the release recommendation says, in a sentence a manager can act on, whether the application should ship and what risk remains if it does.

The mistake matters for the pack because it changes what goes in it. A review built around the model produces a pack of model scores: refusal rates, filter catch rates, a benchmark.

A review built around the application produces the pack above, where the model's behavior is one input to the evidence and the boundaries are the findings. The first kind of pack goes stale with every model update. The second tells the next engineer which boundary to retest when the model changes.

Every module produces part of the pack. Module 1 produces the first threat model; Modules 2 to 10 produce findings and fixes with their checks; Module 11 produces the regression suite; Module 12 the monitoring and response plan. The final review assembles them for a fresh copy of the application, one you have not seen, so the exam measures the method rather than your memory of the answers.

The release recommendation is the hardest item to write well. It is short, and it commits you. "Ship, with the cross-tenant filter and the send allow-list in place; the remaining risk is that a poisoned knowledge article can still mislead an answer, and Module 4's provenance control is scheduled before the second tenant joins" is a recommendation a manager can act on.

"Several risks were identified and should be addressed" is not. The course asks you to write the first kind from Module 1 onward, so that by the final review it is a habit.

Several neighboring subjects are left to other courses, and knowing where the edges are saves time.

The course does not teach training or fine-tuning models, or the mathematics of attacks on a model's training. Module 10 introduces poisoning and the questions to ask a model's supplier, because an engineer must recognize those risks, but it does not practice them.

It does not teach jailbreaks whose purpose is harmful content: making a model produce dangerous material is a content safety problem with its own discipline, and the attacks in this course aim at what an application lets a model do, with harmless goals stated in the open.

It does not teach writing detection rules; Module 12 designs what the application should record and investigates with it, and detection engineering has its own courses. And it does not teach AI governance, deciding whether a system may be used at all, which the governance course covers.

The third card is the one that surprises people. Cross-site scripting, SQL injection and command injection are old subjects, and a course on AI could reasonably assume them. It does not, where a model is involved, because the model changes who writes the payload.

A developer who escapes user input carefully may pass model output straight to a template, on the assumption that the application wrote it. The application did not; the model did, and the model was reading text an attacker chose.

General application security appears wherever the model changes it. When model output reaches a browser, a database or a shell, the course teaches the injection that results in full, because the model has changed how the payload arrives. The rest of web security is assumed knowledge, and Module 0's lab lesson lists the Python, HTTP, OAuth and container basics the exercises need.

The course trains the engineer who secures an LLM application or agent. Its principle is that model behavior can fail, so access and action boundaries must hold when it does, and every module runs the same loop: review the design, prove the weakness, fix it in code, and verify with a check that measures attacks and normal work together.

Two applications, a knowledge assistant and a service-desk agent, carry the whole course, practiced in the browser range, the lab and the recorded logs. The twelve modules end in a security review that hands over a threat model, findings, fixes with regression results, a permission and approval matrix, a monitoring plan and a release recommendation.

Versions this lesson relies on: OWASP Top 10 for LLM Applications 2026 (published 4 August 2026); OWASP Top 10 for Agentic Applications 2026 (announced 9 December 2025); MITRE ATLAS content v2026.10 (28 October 2026); MITRE ATT&CK Enterprise v20 (13 October 2026).