Staying critical of output that sounds right

Fluent writing suppresses scrutiny. Build habits that make checking automatic, because vigilance on its own does not survive week three.

The dangerous output is not the obviously wrong one. It is the fluent, well-structured, confident one that is subtly wrong — because fluency is exactly what your brain uses as a proxy for competence.

This gets worse with time, not better. In week one everyone checks everything. By week four the tool has been right often enough that checking feels like box-ticking, and that is precisely when the first real error goes out.

You cannot fix this by asking people to be careful. Vigilance is a finite resource and it decays. You fix it with habits that do not depend on anyone feeling suspicious that day.

Check the claim, not the prose

When you review, do not read it top to bottom — you will assess the writing. Instead pull out the specific claims and check those:

every number: where did it come from, and could you reproduce it?
every name, date and reference: does it exist?
every "always", "never", "all customers": is it actually true, or a plausible generalisation?
every causal claim: is the causation stated, or inferred by the model?

Four claims checked properly beats a page read attentively.

Ask the question you do not know the answer to

The strongest habit, and the cheapest: include one question in every session where you already know the answer and the tool should get it wrong — an obsolete price, a policy exception, a customer who left.

If it confidently answers, you have just measured something real about the session you are in. Teams that do this catch degradation weeks before teams that do not.

Make it show its work

Ask for the source alongside the claim, and for what it did not find. "I could not find anything about X" is more informative than a paragraph that quietly routes around the gap.

If the output cannot be traced back to something you can open, treat it as a hypothesis rather than an answer. That is not paranoia — it is the same standard you would apply to a new colleague's first month.

The three questions before anything leaves the building

Short enough that people actually use it:

Which part of this did a person verify?
What would it look like if this were wrong?
Who is accountable if it is?

If the answer to the first is "the formatting", it has not been reviewed.

Watch for these specific traps

Trap	What it looks like
Confident specificity	An exact figure or date with no source. Precision reads as authority.
Smoothed-over gaps	A complete-sounding summary of documents where one was missing
Agreeing with you	You framed the question with an assumption; the answer confirms it
The second draft	You asked it to revise, and it changed the facts while improving the prose
Familiar territory	The domain you know best is where you skim hardest

The last one is counter-intuitive and catches senior people most.

Protect the person who says "this is wrong"

The organisational failure mode is worse than the technical one: someone notices the output is wrong, and it is easier to let it go than to be the person slowing things down.

If nobody has pushed back on the tool in a month, that is not a sign of quality — it is a sign the pushback is not being welcomed. Ask directly, and thank the person who answers. What determines whether you find out about problems early is almost entirely whether saying so was rewarded last time.

Keep a log of what it got wrong

One shared page. Date, what it said, what was true.

It takes a minute, and it does three jobs: it makes the failure modes concrete for new team members, it feeds your test set, and it stops the collective memory resetting to "it's been fine" every few weeks.