AI Compliance False Positives: Regex First, Model Second (Simply Explained)

The Scanner That Embarrassed Itself

A financial advisory firm managing over $500 million came to me with a problem that sounds simple until you try to solve it.

Their website has to follow a strict rulebook. The wrong claim, the wrong job title next to someone's name, a missing legal notice. None of those are typos. They are legal problems that can cost real money.

So I built them a scanner. Think of it as a digital proofreader that reads every page and flags anything that breaks the rules.

The first version was simple. I handed each page to an AI that reads and writes like a human and asked it to find violations. It looked great in a demo.

Then it ran for real and made a fool of itself.

It flagged the phrase "best interest" as a problem. Trouble is, "best interest" is a phrase the law actually requires them to use. The AI saw the word "best," assumed it was a bragging marketing claim, and waved the flag.

Then it claimed a required legal notice was missing. It was not missing. The text had just gotten shuffled out of order when the page was processed, like a paragraph cut up and taped back together in the wrong sequence. Every word was there. The AI couldn't tell.

Two flags on the first run. Both wrong.

The owner asked the right question: if it gets the obvious stuff wrong, why would I trust it on the hard stuff?

That is the whole problem with AI compliance, and it's a fair doubt for any business owner to have.

Why the AI Kept Getting It Wrong

Here is the thing I've learned building these systems. An AI is a great judge but a terrible detective.

Ask it to look at something and give an opinion, and it shines. Ask it to spot exact words and phrases reliably, and it gets creative in exactly the place you need it to be boring.

That's why it "corrected" the law. It had seen thousands of bragging marketing claims with the word "best" in them. So when it saw "best interest," it assumed the worst. It didn't know that specific phrase was required, because nothing told it.

That's the dangerous part. It wasn't just missing problems. It was flagging perfectly legal, required language as a violation.

And every false alarm chips away at trust. A scanner that cries wolf twice on day one is a scanner the owner stops using by week two. Then you're back to a junior employee reading every page by hand.

A tool nobody trusts gets switched off. That's the real cost.

The Fix: Let the Robot Do the Counting First

The fix wasn't a smarter prompt. It was a different setup entirely.

Instead of asking the AI to do everything, I split the work into two stages, like a kitchen with a prep cook and a head chef.

The prep cook is plain, simple software that does one thing perfectly: find exact matches. The rulebook has a fixed list of banned words and phrases. Either they're on the page or they're not. No opinion needed, no guessing. The software just catalogs everything suspicious and says "look here."

This is the part the AI kept botching, and it turns out it has a fixed answer. So I gave it to software that doesn't get creative.

The prep cook also checks for required legal notices by looking for the right words anywhere on the page, not in any particular order. That killed the "missing notice" false alarm. Shuffle the words all you want, the check still finds them.

The cleverest part is what I call a closeness check. A wrong job title is only a problem when it sits right next to a specific person's name. The correct title might appear elsewhere on the page and look fine. So the software checks whether the wrong title is sitting next to that person's name specifically. Simple proofreading misses this completely.

Then the head chef steps in. The AI only looks at the items the prep cook flagged, and it makes the judgment call. Is "best interest" the bragging kind or the required kind? That's context, and context is what the AI is actually good at.

Driving the False Alarms to Zero

I'll be honest. This new setup got us close, but not all the way on the first real run.

Two false alarms slipped through. So I gave the AI two clear instructions.

First: a legal notice with its words shuffled is still a present notice. Judge by whether the words exist, not by their order.

Second, my favorite example. One rule says the firm can't pretend to be a broker-dealer. But they're legally required to print the line "is NOT a broker-dealer." A naive scanner trips over the banned phrase and flags the very notice that keeps them compliant.

So I taught the AI to read the "NOT." If the phrase is negated, it's the required notice, not a violation. That one little word flips the whole meaning, and the AI is the only piece that can catch it.

After those two fixes, false alarms hit zero on their entire website.

Now the honest limitation. Zero on this website, with this rulebook. New rules need new patterns. A strange page I haven't seen could surface something new. I'm not selling you a scanner that's perfect forever. That doesn't exist.

What I am selling is this: when something does go wrong, I know exactly where to fix it. Did the prep cook flag the wrong thing, or did the head chef judge it wrong? One is a quick software tweak, the other is a quick instruction tweak. Either way, I know where to put my hands.

A do-everything AI black box gives you nowhere to put the fix. You poke at it, rerun, and pray.

This Isn't Just About Compliance

This same approach works anywhere you mix firm rules with judgment calls.

Reviewing contracts. Moderating comments. Checking that your marketing stays on-brand. The pattern is always the same. Let plain software handle anything with a fixed answer. Save the AI for the genuine judgment calls software can't make.

Most people do the opposite. They reach for the flashy AI first because it demos beautifully and sounds like the future. So the AI does everything, including the boring detective work it's worst at.

The reliable systems flip that. Cheap, simple software does the cataloging. The AI judges last. And a human approves every flag before a single word changes on the live website.

Three layers, each doing the job it's actually good at. For regulated work, that human check isn't optional. One wrong claim is a real legal problem, and no scanner gets to publish unsupervised.

Trust doesn't come from a clever prompt. It comes from a setup where every decision can be inspected and every failure has an address.

Want to explore what AI could do for your business?

Book a free 30-minute strategy call. No pitch deck, no sales team, just a real conversation about your operations and where AI actually fits.

Book a Discovery Call