Why Fluent AI Output Defeats an Analyst's Verification Instincts

The cues you lost

Consider how you actually evaluate information from a colleague, because you do it constantly and almost none of it is conscious.

A junior analyst tells you the account was compromised at 02:14. You do not audit that claim from first principles. You read a dozen signals in parallel: how long they took to answer, whether they said "I think" or "it was", whether they volunteered how they know, whether they are someone who overstates, whether the claim is the kind of thing that is easy to get wrong. If enough of those point the wrong way, you ask a follow-up. If they do not, you accept it and move on.

That system is efficient and it is almost entirely built on signals that have nothing to do with the claim itself. It works because human confidence and human accuracy are correlated. Imperfectly, and enough to be useful.

Generated output arrives with every one of those signals stripped out.

It has no latency that varies with difficulty: the hard question and the trivial one both come back in four seconds. It has no register that shifts with confidence, because the fluency is a property of the text rather than of any assessment of the claim. It does not volunteer its reasoning unless asked, and when asked it will produce reasoning that sounds equally plausible whether or not it is the reasoning that produced the answer. It has no track record you have built up, and no reputation to protect.

Why checking feels done when it is not

There is a second effect, and it is more insidious than the first because it operates after you have decided to be careful.

When you review a generated query, you read it. It parses. The column names are real. The logic is coherent: a filter, a summarize, a sort, in a sensible order. Nothing is obviously absurd. You have now performed a review, and it felt like one, because you did in fact examine the artifact attentively for several seconds.

What you checked was plausibility. What matters is correspondence: whether this query, run against this data, answers the question you actually asked. Those are different properties and only one of them is visible on the page.

Plausibility is cheap to assess and it is what the eye does automatically. Correspondence requires you to hold your original question in mind, work out what the query would return, and compare. That is real cognitive work, it takes longer than reading, and there is nothing in the experience of reading a well-formed query that tells you that you have not done it.

The example on the orientation page is exactly this. Nothing about where ResultType == 0 looks wrong. It is a real column, a valid comparison, and filtering by result is precisely what a thoughtful analyst would do. It reads as diligence. You have to hold the original request in mind, remember that the analyst asked about failures, and notice that zero means success, before it becomes visible.

See it happen, on one query

Reading about plausibility is not the same as being caught by it. Here is a query. Give it ten seconds, decide whether you would run its result into a ticket, then read on.

SigninLogs
| where ConditionalAccessStatus == "notApplied"
| where IPAddress !startswith "10.0."
| summarize AffectedUsers = count() by UserPrincipalName, AppDisplayName

Most people accept it. It is clean, the column names are real, the intent is legible, and the variable is honestly named. Run it and two rows come back, both service accounts using Azure AD PowerShell from outside the estate. That is a genuine finding and the query found it.

Now read it again with your original question in mind. AffectedUsers = count() counts EVENTS and labels them users. With two rows and small numbers nothing looks wrong. Ask the same query about a wider filter and the label starts reporting a number that is not the thing it claims to be.

That is plausibility checking failing in real time. You read it, it looked right, and it was well-formed and mislabelled at once. Nothing about reading it more carefully would have helped, because the error is in the relationship between the label and the function, not in either one.

The cost asymmetry

It is worth being explicit about the economics, because they explain why this problem persists in teams that know about it.

Verification is paid every time. The error is paid rarely.

Checking a generated query costs perhaps ninety seconds: read your original question again, work out what the query returns, look for the mismatch. Do that thirty times a shift and it is forty-five minutes. Most of those ninety-second checks find nothing wrong, because most generated output is fine.

The failure costs nothing at all, most of the time. A wrong query on an alert that was genuinely benign produces a wrong answer that leads to the correct action. You never find out. The cost only lands on the small number of occasions where the alert mattered, and on those occasions it lands as a missed intrusion.

This is also why "be careful with AI output" is useless advice. Everyone agrees with it and the incentive structure defeats it within a week. What survives is a check specific enough to be quick, which is the subject of the rest of this course.

The three states a reviewer can be in

It helps to be able to name where you are, because two of these feel identical from the inside.

Unchecked. You read the artifact, it looked fine, you ran it. You have performed no verification and you know it. This is a legitimate state for low-stakes work and it should be a deliberate choice rather than a default.

Plausibility-checked. You read the artifact attentively, looked for anything malformed, found nothing, and formed a positive impression. This feels like verification and is not. It is the state most reviews stop in, and it is the state the six failure modes are specifically designed to survive, because every one of them produces a plausible artifact.

Correspondence-checked. You named what would have to be true, and confirmed it. This is verification, it takes longer, and it produces a defensible answer.

The reason to name these is that the first and third are honest and the second is not. An analyst who says "I did not check it" has given you accurate information. An analyst who says "I reviewed it" when they were in the second state has told you something that sounds like the third, and they believe it themselves.

Why more careful reading does not fix it

The obvious response to all of this is to read the output more carefully, and it does not work. Two reasons, both structural.

Careful reading finds a different class of error. Attention catches things that are wrong on the page: a mismatched bracket, a column that does not exist, a filter that contradicts another filter. Every one of the six failure modes is internally consistent, so there is nothing on the page for attention to catch. The error is in the relationship between the artifact and something not in front of you, which is either the schema, the data, or your original question.

Attention does not survive volume. Whatever your review standard is at the first alert of a shift, it is lower at the twenty-fifth, and the twenty-fifth is where the intrusion is. A check that depends on sustained care degrades exactly when it is most needed, which is the same reason runbooks exist for incident response rather than trusting experienced responders to remember.

What survives volume is a short, specific, finishable check. Not "read it carefully" but "confirm the filter matches the intent, confirm the window covers the event, confirm an empty result is really empty". Three questions with answers, which is why the next sub is a list of six rather than an exhortation to be thorough.

The cost that arrives late

Nothing in a shift will ever teach you this habit, which is why it has to be installed deliberately.

What replaces the missing cue

You cannot restore the signals you lost. There is no way to make generated text hedge honestly, and prompting it to express uncertainty produces text about uncertainty rather than a reliable measure of it.

What you can do is stop relying on a signal at all, and instead ask a question whose answer is checkable.

The reason this works is that generated errors are not random. They cluster into a small number of shapes, because they arise from the same mechanism every time: the model produced the likely continuation rather than the true one. Likely-but-untrue has a limited vocabulary in a query language. It looks like the common column rather than the correct one, the common time field, the common join, the shape of an answer to a nearby question.

That is why the next sub can name six failure modes and claim the list is close to exhaustive for practical purposes. Once you know the shapes, checking stops being "review this carefully", which is unbounded and therefore never finished, and becomes six specific questions you can answer in ninety seconds.

The one question

Underneath all six is a single question, and if you take one thing from this module it should be this one:

What would have to be true for this answer to be correct?

Not "does this look right", which is plausibility. Not "did it run", which is syntax. The question forces you to name a claim about the world and then check it.

Applied to the orientation example, it produces: for this to answer my question, ResultType == 0 would have to select failures. That is now a checkable statement, and thirty seconds with the schema settles it.

Applied to a detection rule: for this to be a useful rule, the behavior it matches would have to be rare in my estate. Checkable, by running it against a week of data.

Applied to a summary: for this summary to be complete, nothing outside it would have to change the finding. Checkable, by looking at what it left out.

The question is the same every time. What changes is what you check, and that is what the six failure modes give you.