Validate Software Against Historical Data: Replay It

The problem: a new engine that has to be right on the first real run

I built a payroll tax engine in-house for a company I run. Before it could file a single real tax, it had to match reality, to the penny.

Think about what that means. There is no "we'll catch it next sprint" when the output is someone's paycheck. There is no soft launch when the number you compute gets remitted to the IRS and the state. You get one shot to be right, and being wrong costs real money and real trust.

So the question I had to answer is the same question every CEO buying custom software or AI-built tools should be asking: how do I know it's correct before I rely on it? Not "does it look right in a demo." Not "did a smart person build it." How do I actually know.

Most people answer that question by testing against what they think should happen. They write test cases based on their understanding of the rules, run them, watch them pass, and ship. That is testing your assumptions against your assumptions. It proves nothing about reality.

The honest way to validate software against historical data is to flip it. Don't test against what you think should happen. Test against what actually happened. Real money was withheld from real people over real pay periods. Those numbers exist. They are sitting in your records right now.

The whole job becomes matching them, or explaining every penny you miss.

That is the difference between a system you can bet the business on and a system that looks like it works. This piece is how I closed that gap on a payroll engine, and why the same discipline applies to any software that moves money.

Why unit tests weren't enough

I wrote unit tests. Plenty of them. They all passed.

Here's the problem. Unit tests prove the code does what I told it to do. They do not prove that what I told it to do matches the IRS, matches the state of California, and matches 18 months of real paychecks that already went out the door.

My hand-written test cases reflect my assumptions about the tax rules. And my assumptions are exactly the thing under suspicion. If I misread a withholding table, my unit test will faithfully confirm that I implemented my misreading correctly. Green checkmarks, wrong number.

This is the trap with any new calculation engine. The code can be internally consistent and externally wrong. Worse, when you build fast with AI, you generate a lot of plausible-looking logic very quickly. As I've written before, AI is great at finding problems and terrible at being trusted. It will write a payroll formula that reads beautifully and is off by a rounding convention you didn't know existed.

So I needed an oracle. An oracle, in testing terms, is the source of truth you compare your output against. The question was what to use as the oracle.

I couldn't use my own expectations, because those were under suspicion. I couldn't use a competitor's engine, because then I'd just be cloning their bugs.

The only honest oracle is history. Real money was withheld. Real taxes were filed. Real net pay landed in real bank accounts. Somebody, somewhere, already produced the correct answer for thousands of paychecks.

My job was not to invent the right answer. It was to match the right answer that already existed, or explain, to the cent, every place I came up short.

What a replay oracle actually is

A replay oracle is simple to state: take every historical input, run it through the new engine, and compare every output field against what actually happened in production. If the new engine produces the same numbers as reality, it's correct. Where it doesn't, you have a list of exactly what to investigate.

That's replay testing. It's regression testing finance against reality instead of against a frozen snapshot of your own code.

The shape of the harness

The harness pulls every historical paycheck, feeds the inputs through the new engine, and writes a diff report. One row per check. One column per output field. Green where it matches, flagged where it doesn't.

Replay oracle harness flow

The point is not to run it once. It's to run it every time the engine changes, so a fix to one tax line can't silently break another. That's the "regression" part. Reality is the baseline, and the baseline never drifts.

Diff every field, not just the total

Here's the mistake that feels like rigor but isn't: checking that the total matches.

Total match vs field-by-field match (offsetting errors)

A matching total is not enough. If the engine is $5 over on federal income tax and $5 under on a deduction, the net pay matches perfectly. Two bugs, zero diff at the bottom line. You'd ship it and never know.

So I diffed 13 fields per paycheck. Gross pay. Each individual tax line. Each deduction. Net. Every number that goes into producing the final check got its own comparison.

This is the part most people skip because it's tedious. It's also the part that catches offsetting errors, which are the most dangerous kind, because they hide inside a clean-looking total. The total tells you the system might be right. The fields tell you whether it actually is.

The missing input nobody exports: back-solving the elections

Now the honest hard part.

Back-solving missing W-4 election inputs

To re-run a historical paycheck, you need the inputs that produced it. Most of them I had: pay period, gross wages, hours, pay rate. But one input was never stored in any export I could pull: each employee's W-4 elections. Filing status, allowances, adjustments. The settings that determine how much federal tax gets withheld.

I had the historical withholding amount. I did not have the setting that produced it. Without that setting, I couldn't re-run the check, which meant I couldn't validate the one tax line that depended on it.

The solution was to back-solve.

Back-solving inputs works like this: when you know the output and you know the formula, you can often recover the missing input. I had the known withholding for each check. I had the known tax tables. So I solved the equation backward. Given this withholding and these tables, what filing status and allowances must this employee have elected?

For most employees, the answer fell out cleanly. One election produced the historical withholding, and no other did.

But I'm not going to pretend it was clean for everyone. Back-solved inputs can be ambiguous. Sometimes two different elections produce near-identical withholding, especially at certain wage levels where the tables flatten out. When that happened, I couldn't be sure which election was the real one.

So I treated those as a flag, not a fact. If a back-solved input was ambiguous, that check got marked for manual review rather than counted as a confirmed match. You don't get to launder a guess into a validation. An uncertain input means an uncertain result, and the only honest thing to do is say so.

Reading the diffs: where they cluster tells you what they are

This is the part I wish someone had told me before I started. The pattern of the diffs is the diagnosis. Where the differences cluster tells you what they are, before you read a single line of code.

Reading diff clusters: methodology vs bug

When they cluster at one penny: it's methodology

If diffs are exactly one cent and they show up across many different checks and many different people, that's almost never a bug. That's a rounding methodology difference.

The classic one is per-check rounding versus cumulative rounding. Do you round each paycheck independently, or do you track the cumulative figure and reconcile? Both are legitimate. The IRS and the state accept both. They produce penny-level differences that scatter randomly across a population.

When I saw a cloud of one-cent diffs spread evenly, I knew I wasn't looking for a bug. I was looking at a defensible convention difference. Important to understand, not urgent to "fix."

When they cluster on one person: it's a bug

The opposite pattern is the dangerous one. When diffs cluster on one person, or one specific check, that's a real bug or a real edge case you just found.

Here are the actual results. State disability insurance came back at 100% match. Every check, every person, exact. Federal income tax matched on 97.6% of checks.

I did not declare victory at 97.6%. That remaining 2.4% is exactly where the truth lives.

Every miss over ten cents got a name. One turned out to be a terminated employee's final check, a one-off with a different calculation path I hadn't accounted for. Another cluster was the per-check-versus-cumulative rounding convention I mentioned above. Each explainable. Each accounted for.

That's the discipline. You don't stop at "close enough." You account for the 2.4% until every remaining diff has a name and the name is acceptable. A diff with no explanation is a bug you haven't found yet, not a diff you can ignore.

The rule: a passing total is a trap, every diff needs a name

This generalizes far beyond payroll, and if you're buying software, this is the part to internalize.

The trust gate for validating money-moving software

It applies to any financial or calculation engine. Pricing. Billing. Commissions. Subscription proration. Accounting close. Anything where the software computes a number that someone relies on.

The dangerous state is software that looks like it works. A clean total is the most seductive lie in software, because it passes the eyeball test. The CEO looks at the bottom line, it matches, everyone moves on. Meanwhile two errors are quietly canceling each other out, waiting for the one edge case where they don't.

The trust gate is not "the numbers are close." Close is worthless. The trust gate is: I can explain, to the cent, every place the new engine disagrees with reality, and every explanation is acceptable.

That's a hard standard. It's also the only one that actually protects you. This is the same logic behind the kill-switches I build into every system. You define the trust gate before you go live, not after something breaks in production.

The deliverable that comes out of this isn't a passing test suite buried in a repo. It's a diff report a non-technical owner or an auditor can read line by line. Field by field. Check by check. With the explanation for every variance written in plain language.

If you can't produce that report, you don't actually know your engine is right. You just haven't been caught yet.

How to know your custom or AI-built system is correct

If you're buying or building custom software that moves money, here's the question that matters.

It's not "did a smart person build it." Smart people ship offsetting errors every day. The question is: was it replayed against reality, and was every diff accounted for?

That's the line between a system you can file real taxes with and a demo that happens to compute numbers.

I'll be straight about what AI did here and what it didn't. AI helped me build the engine fast. AI helped me build the replay harness fast. The typing got compressed dramatically. But the judgment of which diffs were acceptable, which back-solved inputs were trustworthy, and when a one-cent cloud was methodology versus a bug, that was all human. As I've said before, AI replaced the typing, not the thinking.

That distinction is the whole job. Anyone can generate a payroll engine now. Proving it's correct before you bet payroll on it is the half almost everyone skips, because it's slower, less glamorous, and requires admitting you might be wrong.

That validation discipline is what I bring as Chief AI Officer. Building the engine is the easy half. Proving it's right before the business depends on it is the half that separates real systems from impressive demos.

If you've got software moving money and you're not certain it's right, that's the conversation worth having.

Ready to bring AI leadership into your company?

I work with a small number of companies at a time. If you're serious about AI, apply to work together and I'll review your application personally.

Apply to Work Together