Reading width
Wide uses the full column for everything, text, diagrams, code, and exercises. Narrow keeps the standard reading width.
Text size
Scales the body text. Headings and code blocks keep their size.
In this section
0.9 Privacy Foundations
Introduction
Security work handles personal data constantly: sign-in logs name people, phishing simulations record who clicked, incident tickets describe what someone did. AI makes the exposure larger, because data is copied into prompts, retrieval indexes and training sets, and shared with vendors who run the models.
Module 9 builds a redaction layer that removes personal data before it reaches a model, and adds differential privacy to aggregate results. This lesson lays the foundations: why removing names does not anonymize data, what k-anonymity measures, and how adding calibrated noise protects a count, at what cost.
The lesson's data is a phishing simulation's results, shared with a vendor with names removed, and a public list of people who registered for a local conference. An assistant writes the functions to re-identify the results, measure their k-anonymity, coarsen them, and answer a count with noise. Then the lesson measures what each step buys.
Removing names is the start of the check, not the end: the release is generalized until its smallest group is large enough, then tested by the join an attacker would run.
Data Without Names
What was sharedHere are the first rows of what was shared.
1[2 {3 "zip": "30309",4 "birth_date": "1967-11-15",5 "sex": "F",6 "department": "sales",7 "clicked": 0,8 "reported": 09 },10 {11 "zip": "30318",12 "birth_date": "2001-10-14",13 "sex": "M",14 "department": "support",15 "clicked": 0,16 "reported": 017 },18 {19 "zip": "30309",20 "birth_date": "1976-04-08",21 "sex": "F",22 "department": "finance",23 "clicked": 0,24 "reported": 125 }26]
No row has a name, an email address or an employee number. Each has a ZIP code, a birth date and a sex, which seem harmless on their own, and the department and results the vendor needs. Columns like these are quasi-identifiers: not identifying alone, and identifying in combination, when the combination can be matched against another dataset that does carry names.
The classic measurement of how often that works is Latanya Sweeney's study of the 1990 US census, published as a Carnegie Mellon working paper in 2000.
Her project page reports that 87 percent of the population, 216 of 248 million people, was likely to be uniquely identified by five-digit ZIP code, gender and date of birth; with the place of residence instead of the ZIP code it was 53 percent, and 18 percent even at the level of a county.
She showed it concretely by linking a de-identified hospital dataset to a voter registration list on exactly those three fields. In this lesson the conference list plays the voter roll.
Security data is full of quasi-identifiers. A sign-in log has an IP address, a device name, a time zone and a time of day; an endpoint alert has a hostname that often contains a person's initials; a mail log has a sender's display name even after the address is masked.
Any of these, combined with a second dataset the recipient can get, can put a name back. The question to ask of every column before data leaves a team is not whether it is a name, but what it could be joined to.
AI raises the stakes in a specific way. Pasting log extracts into a model's prompt, indexing tickets for retrieval, or fine-tuning on incident write-ups all send data to a system, and often an organization, other than the one that collected it. That is a disclosure, with the same exposure as sharing the file with a vendor, and it needs the same measurement first.
The Contract
Measure the exposure, then reduce itThe exercise file states the four functions and what each one measures or changes.
⋯ 28 lines folded293031def reidentify(released, public, quasi=QUASI):32 raise NotImplementedError('lesson 0.9: write reidentify')33⋯ 7 lines folded414243def laplace_count(count, epsilon, rng):44 raise NotImplementedError('lesson 0.9: write laplace_count')
The first two measure. Re-identification joins the shared rows to the public list on the quasi-identifiers and keeps only unique matches, which are the ones an attacker can be confident of. k-anonymity counts the smallest group of rows sharing the same quasi-identifier values: a dataset is k-anonymous if every row is indistinguishable from at least k minus one others on those columns.
The last two reduce. Generalization coarsens the quasi-identifiers so that more rows share each combination. The noisy count answers a question about the data with random noise added, sized so that no single person's presence changes the answer by much.
Run the check against the file as shipped.
$ python3 -m lab check f09
f09 0.9 Privacy Foundations: lab/foundations/privacy.py
FAIL the functions are written
lesson 0.9: write reidentify
FAIL
Once the functions exist, the check computes the join itself and compares every name, so a re-identification that matched too loosely, or kept ambiguous matches, fails rather than looking more effective. It checks generalization on a copy, so the shared data is never altered in place, and it measures the noise statistically, over thousands of draws, because a single noisy answer proves nothing about the distribution it came from.
Session: The Brief
Measure honestly, reproduce exactlyThe brief points at the contract and the data, and adds two requirements.
Write the four functions in privacy.py to the contract in its docstring, standard library only. The point of the exercise is to measure how exposed the shared phishing results are, so reidentify() should find every unique match, and laplace_count() must draw its noise from the rng it is given and nothing else, so results can be reproduced.
An assistant asked to re-identify people may hesitate, and the hesitation is reasonable outside a lab. Here the data is synthetic, generated by a script in the lab, and the purpose is to measure how exposed a real release of this shape would be before anyone shares one; the brief says so.
The second requirement, drawing noise only from the random number generator passed in, is about reproducibility. A privacy mechanism whose output cannot be reproduced cannot be tested, and one that quietly uses a second random source makes every measurement of its error unrepeatable.
The data in this exercise is synthetic, which is the right way to build privacy tooling with an assistant. Writing the code does not need real personal data; it needs data with the same shape.
Pasting a real phishing result into an assistant to get help with re-identification code would be the very disclosure the code is meant to measure. Generate a synthetic sample with the same columns and distributions, build and test against it, and run the finished code on the real data inside the environment that is allowed to hold it.
The Draft
A join, a count, and noiseThe reply was the file. Here is the code below the docstring.
27import math28from collections import Counter, defaultdict2930QUASI = ('zip', 'birth_date', 'sex')313233def _key(row, quasi):34 return tuple(row[q] for q in quasi)353637def reidentify(released, public, quasi=QUASI):38 names = defaultdict(list)39 for person in public:40 names[_key(person, quasi)].append(person['name'])41 found = {}42 for i, row in enumerate(released):43 match = names.get(_key(row, quasi), [])44 if len(match) == 1:45 found[i] = match[0]46 return found474849def k_anonymity(rows, quasi=QUASI):50 counts = Counter(_key(r, quasi) for r in rows)51 return min(counts.values()) if counts else 0525354def generalize(row):55 year = int(row['birth_date'][:4])56 return dict(row, zip=row['zip'][:3] + '**', birth_date=str(year - year % 5))575859def laplace_count(count, epsilon, rng):60 if not epsilon > 0:61 raise ValueError('epsilon must be greater than 0')62 scale = 1.0 / epsilon63 u = rng.random() - 0.564 return count - scale * math.copysign(1.0, u) * math.log(1 - 2 * abs(u))
Re-identification indexes the public list by the three quasi-identifiers and, for each shared row, keeps the name only when exactly one person matches. k-anonymity counts rows per combination and returns the smallest count.
Generalization keeps the first three digits of the ZIP code and replaces the birth date with the first year of its five-year band, on a copy. The noisy count draws one uniform number, centers it, and converts it into a Laplace-distributed value with scale one over epsilon by the inverse of the distribution's cumulative function.
The inverse transform deserves one check the contract did not ask for. Python's generator returns values from zero up to but not including one, so the centered value can be exactly minus one half, and at that value the logarithm is taken of zero and raises an error.
The chance is about one in nine thousand trillion per draw, which is why the check never meets it, and it is still a crash waiting in a function that may be called millions of times. The review records it. Production differential privacy libraries also guard against subtler problems with floating-point noise that hand-written mechanisms do not, which is one reason to use them.
Run the check against the draft, copied into the lab's foundations folder.
$ cp records/privacy-0.9.py lab/foundations/privacy.py
$ python3 -m lab check f09
f09 0.9 Privacy Foundations: lab/foundations/privacy.py
ok every released row with exactly one public match is re-identified, correctly
149 of 150 rows named
ok ...including who clicked
33 named clickers
ok k-anonymity of the release as shared
k = 1
ok generalize() coarsens zip and birth date, on a copy
303** 1965
ok after generalizing both files, no row is unique
k = 2, 0 named
ok epsilon must be positive
ok noise at epsilon 0.1: unbiased, mean error near 1/epsilon
mean 99.93, mean error 10.03
ok noise at epsilon 1.0: unbiased, mean error near 1/epsilon
mean 99.99, mean error 1.00
ok the same rng seed gives the same noise
PASS
Every property held on the first run, including the ones that are easy to get wrong: ambiguous matches left out rather than guessed, and the shared rows left untouched by generalization.
The check confirmed every unique match by computing the join independently, measured k before and after generalizing, and drew twenty thousand noisy counts at two values of epsilon to confirm the noise is centered on the truth and the average error is one over epsilon.
Putting the Names Back
What the vendor could learnThis script does what anyone holding both files could do.
$ cp records/privacy-0.9.py lab/foundations/privacy.py
$ python3 link.py
149 of 150 released rows matched exactly one attendee
Dara Adams sales clicked 0 reported 0
Kofi Zhou support clicked 0 reported 0
Nia Hill finance clicked 0 reported 1
Kofi Park finance clicked 0 reported 0
Eli Diaz finance clicked 0 reported 0
33 of 33 people who clicked are now named
149 of the 150 shared rows matched exactly one person on the conference list, and every one of the 33 people who clicked the simulated phishing link is among them, by name.
The vendor, or anyone the vendor's copy reaches, now knows who fell for the test, by name and department, which is exactly the information the organization meant to keep inside. Nothing in the shared file was a name. The three columns together were.
The public list in the lab is a conference registration; in practice the second dataset is whatever is easiest to get. Professional networking profiles carry employers, locations and often birthdays. Breach collections carry names, addresses and dates of birth for large parts of the population.
Data brokers sell exactly this kind of join as a service. A release is only as anonymous as the best dataset someone could join it to, now or later, and later is the hard part: data shared today is joined against whatever is published next year.
The cheapest protection comes before any technique: share less. The vendor analyzing click rates needed the department and the result; it never needed a birth date at all, and a region would have served better than a ZIP code.
Minimizing what is shared removes quasi-identifiers entirely instead of blurring them, and it is the first question to ask of any dataset bound for a vendor, a prompt or a training set: which columns does the recipient's question actually need.
This is the first measurement to make before any dataset leaves the organization, whether to a vendor, into a prompt or into a training set: join it against the most likely public source on its quasi-identifiers and count the unique matches.
Do it with the same tools an attacker would use, which in this lesson took a dozen lines and less than a second to run against a few hundred rows. The result is usually worse than anyone expects, because birth dates are close to unique within a ZIP code, and a ZIP code and a sex narrow the field the rest of the way.
Generalization and k
Coarser columns, smaller groupsGeneralization trades detail for anonymity. This compares the release as shared with the same release, and the same public list, generalized.
$ cp records/privacy-0.9.py lab/foundations/privacy.py
$ python3 kshow.py
as shared: k = 1 named: 149
generalized: k = 2 named: 0
example row: {'zip': '303**', 'birth_date': '1965', 'sex': 'F'}
As shared, k is 1: some combinations of ZIP code, birth date and sex occur only once, which is what made re-identification possible. Generalized to a three-digit ZIP prefix and a five-year birth band, k is 2 and no row matches a single person. The vendor still knows the department and the region and roughly the age, which may be all they need for their analysis.
Generalization always costs something, and the cost should be judged against the question the recipient is answering. If the vendor wants to know whether click rates differ by department, coarsening ZIP codes and birth dates costs nothing.
If they want to know whether staff under thirty click more, five-year bands blur the boundary. The right generalization is the coarsest one that still answers the question, and the way to find it is to ask what the analysis needs before deciding what to share, not after.
k-anonymity has known limits, and k = 2 shows them. If both people in a group of two clicked, the group reveals that each of them clicked, without anyone needing to tell them apart; this homogeneity problem is why later definitions also require diversity in the sensitive values within each group. And k is only measured against the quasi-identifiers you chose.
A department column, or the date the person reported the email, can narrow a group again. k-anonymity is a property of a table and a list of columns, not a guarantee about the people in it.
That makes the choice of quasi-identifier columns a threat model, the same way lesson 0.6's choice of mutable features was. Measure k against the columns an attacker is likely to have, which means asking what public or purchasable data covers the same people.
A release can be k-anonymous against a voter list and unique against a professional networking site that also lists job titles, if the release includes job titles. The measurement is only as good as the attacker it imagines.
Noise With a Budget
Differential privacy, and asking againDifferential privacy takes a different approach: instead of releasing rows, release answers to questions, each with random noise calibrated so that adding or removing any one person changes the probability of any answer by only a bounded factor.
NIST SP 800-226, published in March 2025, describes it as a framework that quantifies the privacy loss to people whose data appears in a dataset, and sets out how to evaluate such guarantees in practice. The bound is set by epsilon: smaller epsilon, more noise, stronger protection.
The noise is sized to the query's sensitivity: how much one person can change the true answer. For a count it is one, since one person adds at most one to a count, which is why the contract's scale is one over epsilon.
A sum of amounts or durations has no natural bound, because one person can contribute an arbitrarily large value, so a differentially private sum first clips every contribution to a stated maximum and then adds noise sized to that maximum. Choosing the clip is a decision about accuracy, and leaving it out makes the guarantee meaningless.
This asks how many finance staff clicked, at four values of epsilon.
$ cp records/privacy-0.9.py lab/foundations/privacy.py
$ python3 dp.py epsilon
true count of finance staff who clicked: 4
epsilon five noisy answers mean error over 10,000
0.1 [7.3, -26.0, -2.0, -4.1, 10.4] 9.93
0.5 [4.7, -2.0, 2.8, 2.4, 5.3] 1.99
1.0 [4.3, 1.0, 3.4, 3.2, 4.6] 0.99
5.0 [4.1, 3.4, 3.9, 3.8, 4.1] 0.20
The true count is 4. At epsilon 0.1 the mean error is about 9.93, larger than the count itself, and some answers are negative; at epsilon 1 it is about 0.99, and at 5 about 0.20. The average error is one over epsilon, as the check confirmed.
Negative answers can simply be rounded up to zero, because processing a differentially private answer after it is released does not weaken the guarantee. Choosing epsilon is choosing how much error the people asking can live with against how much any individual's data can show through.
Differential privacy protects individuals, not groups, and that is by design. If finance staff click far more often than everyone else, a private count will still show it, because the fact is about the department and holds whether or not any one person is in the data.
That is usually what the organization wants to learn and share. It is worth saying plainly to anyone who expects noise to hide everything, because a finding about a small group can still feel like a finding about the people in it.
Now ask the same question many times, with fresh noise each time, and average the answers.
$ cp records/privacy-0.9.py lab/foundations/privacy.py
$ python3 dp.py average
1 answers at epsilon 1.0, averaged 4.33 total epsilon spent 1
10 answers at epsilon 1.0, averaged 3.26 total epsilon spent 10
100 answers at epsilon 1.0, averaged 4.02 total epsilon spent 100
1000 answers at epsilon 1.0, averaged 4.06 total epsilon spent 1000
With a thousand answers at epsilon 1 the average is 4.06, close to the true count of 4. Each answer was protected; together they were not. With ten answers the average was already within about one of the truth, and every further question narrowed it, at a total cost in epsilon that grew with every answer given.
Privacy loss adds up across queries, which is why differential privacy is managed as a budget: every answer spends some epsilon, the total for a dataset is capped, and once it is spent, no more questions are answered. A system that adds noise without tracking the budget only looks private.
In practice budgets are managed in a few ways. A dataset can be released once as a fixed set of noisy statistics, so the budget is spent at publication and nobody can query again. An interactive system can give each analyst or each project a budget and refuse queries once it is gone.
Repeated identical questions can be answered from a cache, so asking again returns the same noisy answer rather than a fresh one, which closes the averaging attack in section 7 at no cost. Each of these is a design decision about who may learn what, made before the first query.
From Counts to Module 9
What the review carries forwardThe session record keeps what held, the measurements, and the two gaps the lesson left open.
1# Session 0.9: re-identification and noise23Assistant: Claude Opus 5.5 (claude-opus-5-5), the SEC410 build session, 2026-11-1845## Held6- A join on zip, birth date and sex; only unique matches kept.7- Generalization on a copy; k counted over all groups.8- Laplace noise from the given rng only. check f09: PASS.910## Measured11- 149 of 150 shared rows named; 33 of 33 clickers named.12- Generalized both sides: k = 2, nobody unique.13- Mean error at epsilon 0.1, 1 and 5: 9.93, 0.99, 0.20.14- Asking again: 1000 answers at epsilon 1 averaged 4.0615 against a true count of 4. The budget was the protection.1617## For Module 918- Nothing tracks a budget across queries.19- k = 2 still lets a group's shared value be read off.20- laplace_count() fails if rng.random() returns exactly 0.0.
The two gaps are the starting point for Module 9. The noise has no budget, so the averaging attack in section 7 works against it; Module 9 builds the accounting that refuses the thousandth question. And generalization reached k = 2 without checking what each group reveals.
Module 9 also builds the redaction layer that sits between personal data and a model, measures what it misses, and finds the privacy budget at which aggregate results stop being useful, because that point, not a default epsilon, is where the real decision lies.
Redaction for models is harder than this lesson's tables in one important way. Here the quasi-identifiers were known columns. In a prompt they are buried in free text, a name in a sentence, a hostname in a stack trace, a birthday in an email signature, and finding them is a detection problem with its own misses and false alarms.
Module 9 measures what its redaction layer misses on planted personal data, because a redactor that catches most of it still sends the rest.
Practice
Re-identification and noise the labBrief your assistant with the contract and the two requirements, check the draft, measure the exposure before and after generalizing, then choose an epsilon for the finance count and say what error you accepted.
1# Practice 0.9: re-identification and noise23## Done when4- [ ] lab/foundations/privacy.py meets its docstring's contract5- [ ] python -m lab check f09 prints PASS6- [ ] link.py and kshow.py run; the exposure before and after7 generalizing recorded8- [ ] an epsilon chosen for the finance count, with the error9 you accept and why10- [ ] records/0.9-<assistant>.md: brief, draft, runs, review1112## Constraints13- Noise comes only from the rng passed in.14- No re-identified name leaves your lab records.1516## Stretch17- Add a budget: refuse a query once the epsilon spent on a18 dataset reaches a limit you set.
Removing names does not anonymize data: 149 of 150 shared rows were re-identified by joining three ordinary columns to a public list, including every person who clicked. Generalizing those columns raised k from 1 to 2 and left nobody unique, with known limits when a small group shares a sensitive value.
Differential privacy protects answers rather than rows with noise of scale one over epsilon, and a thousand protected answers averaged back to the truth, which is why the budget, not the noise, is the protection.