Nono B

Why can it draft a contract but not count the letters in a word? Same reason.

The model does not read letters. Your text is chopped into tokens and converted into identifiers before it ever reaches the model. So spelling, counting and strict string matching are hard for a system that has never seen a character.

Which means exact data parsing, reference number validation, anything where the characters themselves matter, is a quiet failure waiting to happen. No error message. Just wrong, sometimes.

You do not have to break into a model to compromise it. You can just feed it.

Data poisoning means manipulating what a system learns from rather than attacking the system itself. It matters most where training data is scraped, crowdsourced, or supplied by a vendor nobody audited, which is very nearly all of it.

And it is hard to catch because a poisoned model does not look broken. It behaves normally on everything except the specific case somebody wanted it to get wrong.

So the provenance of your training data is not a documentation exercise for the file. It is a security control, and it should sit with the people who treat it like one.

If your AI impact assessment does not answer these six questions, it is not an assessment. It is a form.

One, does this system make or materially influence a decision about a person. Two, where did the training data come from and do we have the right to use it. Three, can we explain a single adverse output to the person affected. Four, what does drift look like here and who watches for it. Five, how does someone contest an outcome, and does that route actually exist or is it a mailbox. Six, what triggers deactivation and who holds that decision.

You do not need a new process for this. Put them in the DPIA you already run.

Comment TEMPLATE and I will send it to you.

A message worth sharing.

Take a moment to watch, reflect, and share your perspective.

What are your thoughts on this?

A better prompt will not fix hallucination. It was never a prompting problem.

The model is not lying to you. It has no mechanism for telling the truth in the first place.

It predicts a plausible continuation, and fluency and accuracy are completely independent properties in that process. Something can read as authoritative and be entirely invented, because the thing that makes it read well is not the thing that makes it correct.

So the fix is architectural. Ground it in retrieved sources. Require citations you can click. Put a human at the decision point.

A better prompt reduces the frequency. It does not change the mechanism.

Four functions in the NIST AI framework. Three of them are steps. One is not, and almost everybody gets it the wrong way round.

Map, Measure and Manage apply to one system, at one point in its life. You do them and they are done.

Govern is not a stage you complete and tick off. It runs throughout. It sets who is accountable and it decides what your culture rewards when a system starts going wrong. It is the thing the other three happen inside.

Watch the full 48 seconds, then go and check when your governance group last met about something already running.

Most people wait until they have a framework. You can start Today.

Take your highest risk AI system. One meeting, five questions, one page. Intended use. Who gets hurt when it fails. What third party parts are inside it. Go, or no go.

That page is the first dated entry in your evidence base, and dated records are the only thing an auditor actually asks for.

Which system would you pick first? Tell me in the comments.