Somewhere around the third time I watched a token usage dashboard spike overnight for no obvious reason, I stopped treating prompt design as an afterthought.
It’s easy to build an LLM-powered feature that works fine in a demo and then watch the bill quietly triple once real traffic and real conversations show up. Tokens are the actual unit of cost and latency in these systems, and most teams don’t think about them until the invoice forces the conversation.
A few of these lessons came from agent workloads specifically, the kind that hold multi-turn context, call tools, and retrieve documents, since those are where token usage compounds fastest. Here’s what’s actually worked, and a couple of things that sounded good on paper but didn’t hold up.
Most of the waste doesn’t happen at the model call itself; it happens in everything leading up to it. That’s worth keeping in mind as you go through these, since it’s tempting to focus optimization effort on the one step that feels most “AI,” when the bigger opportunities are usually upstream.
Start With the Prompt Itself
Verbose, over-engineered prompts are the easiest win nobody takes. A five-paragraph system prompt with elaborate role descriptions and redundant instructions burns tokens on every single call, not just once.
Cutting a bloated persona description down to a single clear sentence sounds trivial, but multiply that savings across thousands of calls a day and it adds up fast.
Structured output helps here too. Asking a model to return structured output instead of open-ended prose can reduce token usage, since prose tends to pad itself with transitional language that a tightly defined schema simply doesn’t need.
The catch is that structured output only pays off if your downstream code actually parses it cleanly. If you’re still eyeballing responses for formatting quirks, you haven’t actually captured the savings yet.
Context Management Is Where the Real Discipline Lives
This is the part that separates a demo from a production system. Multi-turn conversations don’t need the entire history replayed every time.
A sliding window — keeping the last several turns verbatim and summarizing everything older — keeps context bounded without losing the thread of the conversation.
I’ve seen teams skip this because summarization felt like it might lose important detail. In practice, a decent summarization pass loses far less than people expect, and the alternative, unbounded context growth, is worse for both cost and latency.
Selective context injection matters just as much. It’s tempting to hand a model everything that might be relevant, but quality beats quantity here. A retrieval step that returns ten loosely related chunks instead of three tightly relevant ones doesn’t just cost more tokens; it usually makes the response worse too, since the model has to work harder to figure out what actually matters.
Retrieval Deserves More Attention Than It Gets
If you’re running RAG, chunk size is one of those settings people pick once early on and never revisit. Too large and you’re paying for irrelevant context. Too small and you lose coherence, forcing more retrieval rounds to compensate.
Re-ranking retrieved results before they hit the model, rather than trusting raw similarity scores, is a cheap step that measurably improves what actually gets used.
Caching is the other one that’s easy to underrate. Prompt-cache awareness — structuring prompts so repeated prefixes can actually be reused — can meaningfully cut both cost and latency, but only if your prompt structure is consistent enough for the cache to recognize the overlap.
Change your system prompt formatting every deploy and you can lose this benefit without realizing it.
Not Every Query Needs Your Biggest Model
Model routing is probably the single most underused strategy on this list. Not every request needs your most expensive, most capable model.
Simple classification, basic lookups and straightforward rephrasing — these can go to a smaller, cheaper model without any noticeable quality drop, while genuinely complex reasoning gets escalated to something bigger.
Getting this routing logic right takes some upfront work; deciding what “simple” actually means for your use case isn’t always obvious, but the cost difference between routing everything to a frontier model versus routing intelligently is not small.
The Boring Stuff Still Matters
Output length control is unglamorous advice, but it works. Explicitly asking for concise responses, and enforcing a reasonable max token limit, prevents the model from padding answers the way it sometimes defaults to.
And general response caching, storing and reusing answers to genuinely repeated queries, is one of those optimizations that feels almost too simple to mention, right up until you check how often the same question actually gets asked.
What I’d Tell Someone Starting This Today
Don’t try to implement all of this at once. Start with the two or three that map to where your actual token spend is concentrated, usually context management and model routing account for the biggest chunk, then work outward from there.
Measure before and after each change specifically, since it’s easy to assume something helped when a dozen other variables shifted at the same time.
None of this is exotic. It’s mostly discipline: don’t send more than you need, don’t use a bigger model than the task requires, and don’t recompute what you could have cached.
The teams that get this right aren’t doing anything clever; they’re just consistently doing the boring things that add up.
