How I Cut AI API Costs With a Library-First Pattern

The bill that made me stop generating on every request

A while back I built a consumer app that gives users fresh daily content. Activities, suggestions, a little hit of "here's something new for you today." It worked beautifully in the demo. Every time a user opened the app, the most-used screen called a frontier model and generated something fresh on the spot.

Then I ran the numbers on what happens when people actually use it.

Picture a few thousand daily active users, each opening the app two or three times a day. That's not a big consumer success story. That's a modest little app. And every single one of those opens was firing off a frontier-model call. The math scared me. My model bill didn't track value, it tracked taps. The more people loved the app, the more it cost me to run, with no extra revenue to cover it.

Three problems stacked up at once.

Cost grew linearly with usage. Every open equaled an API call, so success made the bill worse, not better.

Latency lived on the hottest screen in the app. A frontier-model round-trip takes a couple of seconds, so users stared at a spinner on the exact screen they hit most.

And the generation endpoint was an abuse magnet. Free users could open the app over and over and farm expensive generations all day for free.

So here's the doubt every CEO has when I suggest putting AI in front of customers: won't running AI on every request cost a fortune and feel slow?

Short answer: yes. If you do it the obvious way.

The good news is that the obvious way is almost never the right way. And the fix to reduce AI API costs here had nothing to do with a cheaper model. It had to do with when I generated the content at all.

Why live generation is the wrong default

Calling the model at runtime feels like the natural move. It's the first thing that works in a prototype. But it's the lazy default, not the right one, and it fails for three specific reasons.

Three problems with live generation

Cost scales with traffic, not value

When you call a model per request, your largest variable line item tracks raw usage. That's backwards. You want costs that track revenue or value, not the number of times somebody opened a screen. A per-request model call means every bit of growth, every viral spike, every bored user refreshing, adds directly to your bill. Finance hates a cost line that goes up every time the product succeeds.

Latency lives on your hottest screen

The slowest part of almost any AI flow is the model round-trip. Two seconds, sometimes more. When you put live generation on a high-traffic screen, you've parked your worst latency on the path people hit most. A spinner on the home screen is a tax every user pays, every visit. That's a brutal place to spend your latency budget.

Live endpoints are abuse magnets

Any generation endpoint that isn't tightly throttled will get farmed. And frontier-model calls are the single most expensive thing somebody can farm on your dime. If a free user can trigger a generation by tapping a button, somebody will tap that button ten thousand times. You've handed out a credit card with no limit.

Here's the insight that changes everything. Most "AI content" your users see does not actually need to be unique to that user at that exact moment. It just needs to feel intelligent. The gap between "genuinely unique right now" and "feels smart and personal" is enormous. That gap is where all the savings live.

The library-first pattern: do the heavy lifting once, offline

So I stopped generating on every request. Instead, I built what I call a library-first pattern. You do the expensive thinking once, offline, and you do cheap retrieval at runtime.

Live generation vs library-first architecture

Generate the corpus with the expensive model, offline

I built a content library table and populated it with a one-time pass (now periodic) using the most capable model I had. No latency pressure. No per-user cost. Just one job, running offline, authoring a large and diverse corpus of daily-activity content across every age band, category, and context the app supports.

This is where you spend your expensive model. Once. With time to spare, the offline pass can actually produce better content than a latency-constrained runtime call ever could, because it isn't racing a spinner.

At runtime, select with code, not AI

At runtime, two functions do all the work, and neither one calls a model.

One finds candidate content matching the user's context: their age band, the category they're in, what they've already seen. The other selects from those candidates and orders them. Both are pure database queries and deterministic logic. Zero AI calls.

This is the same principle I wrote about in let the model judge and let code do the work. The model does the thinking once. Code does the math forever after. The frontier model authored the corpus; the app just retrieves and orders it.

Same perceived intelligence. No spinner. Near-zero marginal cost per request, because a database read costs a rounding error compared to a model call.

It's the same idea I applied in search the index, not the library: do the expensive AI work once, build a queryable structure, then answer every future request cheaply. Different problem, identical move.

Keeping it personal: a thin, optional adaptation layer

The obvious objection: doesn't pre-built content feel generic? If everyone's pulling from the same library, where's the personalization?

Cost layering: expensive once vs cheap polish per request

Fair. So I added a thin, optional adaptation layer on top.

For users who want a more tailored experience, a small, cheap model runs a single light pass on the selected content's description. It tweaks the tone, adjusts the level of detail, nudges the language to fit that specific user. The content was already chosen by code; the cheap model just gives it a personal touch.

This is the only runtime AI call in the whole flow. And it's optional, fast, and runs on the cheapest model tier available.

Look at the cost contrast. The heavy generation, the part that actually requires intelligence, happens once on the expensive model. The runtime touch, when it happens at all, costs a fraction of a cent on the cheap model. You're not paying frontier prices on every open. You're paying frontier prices once, then pennies for optional polish.

This pairs well with a technique I covered in iterate on AI output for almost nothing. The same instinct applies: get the expensive reasoning done cheaply or once, then iterate and adapt on top of it without re-paying for the hard part.

The point is that "precomputed base" and "real-time intelligence" are not opposites. You can layer genuine personalization on top of an authored library and still keep your cost structure sane.

Reserving the live model for paid, throttled use

Now the abuse hole. The library and the cheap adaptation layer handle the vast majority of usage. But some users genuinely want fresh, on-demand generation. So I kept it, and I gated it.

Server-side weekly limits

Every user gets a weekly generation limit, enforced server-side. Not in the client. Client-side limits are trivially bypassed by anyone who opens the network tab, so they're security theater. The cap lives on the server, where the user can't touch it.

Returning a clean 429 over the cap

When a user exceeds their limit, the server returns a clean 429 with a clear message explaining they've hit their cap and when it resets. No crash, no silent failure, no surprise. Just a polite "you've used your generations for the week."

Free users get the library plus the occasional cheap adaptation. The genuinely live, on-demand frontier generation is reserved for paid, throttled use.

This does two things at once. It caps the worst-case cost any single user can inflict, so no one can farm me into bankruptcy. And it turns the expensive capability into a paid feature instead of a free liability.

Here's the business point. My model spend now tracks paying users and the authored corpus, not raw traffic. The bill stopped being a function of how many people opened the app. It became predictable. And predictable is the thing finance actually wants, far more than "cheap." A cost you can forecast is a cost you can plan around.

What it actually saved, and what it didn't fix

Let me be honest about the full picture, because every pattern has tradeoffs.

Wins and tradeoffs of the library-first pattern

The wins were real. Marginal cost per app-open dropped to near zero, because it's now a database read. The spinner disappeared from the hottest screen, because there's no model round-trip in the default path. And abuse stopped being a cost vector entirely, since the only live generation is paid and throttled.

Perceived quality held, and arguably improved. The offline pass could use a stronger model and take its time, so the authored content was often better than what a panicked, latency-constrained runtime call would have produced.

Now the tradeoffs.

The corpus needs periodic regeneration to stay fresh. If you author it once and never touch it again, it goes stale. You need a schedule.

You lose true real-time uniqueness for the base content. Two users in the same context might see the same item. For this app, that was completely fine. For some products, it wouldn't be.

And there's upfront investment. You have to build the generation pipeline and the selection logic. That's real work you don't do with naive live generation.

For output that genuinely must be one-of-a-kind and fresh per user, live generation is still the right call. Just gate it and throttle it.

The rule of thumb I use: precompute anything that doesn't need to be unique to this exact user at this exact moment. That covers far more of your "AI content" than you'd guess.

The question to ask before you wire up another AI endpoint

Here's the diagnostic question I'd hand any CEO or operator. For every place you're calling AI at runtime, ask: does this output truly need to be generated fresh for this user right now? Or could the model have authored it once, offline, while the app just selects from what's already there?

Decision diagnostic: precompute or generate live

Most teams default to live generation because it's the first thing that works in a demo. Then they get blindsided by the bill at scale, when the thing that made the prototype feel magical turns into a cost line that climbs with every new user.

The library-first pattern is one of several cost levers I check first whenever I find a model bill that scales with traffic instead of revenue. It's not always the answer, but it's astonishingly often the answer, and it's almost never the thing a team tried first.

If your AI spend is creeping up with usage and you're not sure which calls actually need to be live versus which could be precomputed, that's exactly the kind of thing I untangle. I'll go through your AI cost structure call by call and tell you where the money is leaking. You can bring me in to look at your AI cost structure if that's where you're stuck.

Want to explore what AI could do for your business?

Book a free 30-minute strategy call. No pitch deck, no sales team, just a real conversation about your operations and where AI actually fits.

Book a Discovery Call