LLM API Costs for AI Agents: Where the Tokens Actually Go

Published by Rushikesh Dhule, LessBill · November 3, 2026 · Updated November 16, 2026
Token prices keep falling, yet AI bills keep rising. If that feels like a contradiction, look at how your agents actually spend money. Most of it doesn't go to the first answer. It goes to the retries, the self-checks, the giant system prompt sent again on every call, and the premium model doing work a small one could handle.
This is now a mainstream problem. McKinsey's mid-2026 AI FinOps research found that 93% of surveyed enterprises had gone over their AI budgets, and that roughly 60% of agentic AI spend goes to "response refinement": the loops where an agent checks and corrects itself before it produces something usable. About one in five organizations in McKinsey's wider AI survey said they had already held back AI use because of running costs. RBC's CIO survey in June told a similar story: almost half of respondents had blown past their original AI spending plans.
The good news: most of this waste is fixable with plumbing, not a new model. Here are six places to look.
1. You re-send the same giant prompt on every call
Agents carry a lot of baggage: system instructions, tool definitions, policy docs, few-shot examples. That block is often thousands of tokens and identical on every request. Without caching, you pay full price for it every single time.
Fix: Turn on prompt caching and put the stable parts (system prompt, tools, reference docs) at the start of the prompt, with the changing parts at the end. On Anthropic's API, cache reads are billed at about a tenth of the normal input price or less, depending on the model. OpenAI caches long repeated prefixes automatically. Check your cache hit rate weekly; if it's near zero, your prompt order is probably breaking the cache.
2. Refinement loops with no ceiling
An agent that critiques its own work is useful. An agent that critiques its own work eight times on a simple task is a money pit. Since refinement is the biggest slice of agent spend, this is where caps pay off fastest.
Fix:
- Set a hard limit on retries and self-review passes per task (2-3 is plenty for most work).
- Stop early when a cheap check passes: tests green, schema valid, answer matches format.
- Log how many loops each task type uses. A few task types usually cause most of the extra calls.
3. The flagship model is doing intern work
Classifying a ticket, extracting fields from an email, or summarizing a log line doesn't need your most expensive model. Yet many teams wire one model into everything because it was the default in the prototype.
Fix: Add a simple router. Send routine, well-defined steps to a small, cheap model and escalate to the big one only when the task is hard or the small model's answer fails a check. Even routing half your calls to a smaller tier can cut the bill a lot, since smaller models often cost several times less per token.
4. Paying real-time prices for work that can wait
Nightly evaluations, bulk document tagging, embedding backfills, report generation: none of this needs an answer in two seconds.
Fix: Move it to batch processing. Both OpenAI and Anthropic offer batch APIs at 50% off standard token prices, with results returned within 24 hours (often much sooner). On Anthropic, batch and caching discounts can even be stacked.
5. Context windows that only grow
Long-running agents tend to append every tool result and message to the conversation. By step 30, each call is dragging along a huge history, and you pay for all of it as input tokens.
Fix: Trim tool outputs to what the next step needs, summarize old turns instead of keeping them word for word, and retrieve documents on demand rather than stuffing everything in up front. Track average input tokens per call over time; a steady climb is a red flag.
6. Nobody can see cost per feature
Most teams get one API invoice per provider and no idea which feature, customer, or agent burned the money. That makes every other fix guesswork.
Fix: Tag every request with feature, team, and environment (most providers support metadata or separate API keys per project). Then set daily spend alerts per key, so a runaway loop shows up the same day, not at month end.
Worked example: what caching does to one agent task
An agent has a 10,000-token system prompt and tool list, and makes 20 model calls to finish a task. Without caching, that prefix is billed as fresh input every time: 20 x 10,000 = 200,000 input tokens per task before the agent has read anything new.
With prompt caching on Anthropic's API, a 5-minute cache write costs 1.25x the base input price and a cache read costs 0.1x on most models. So the first call costs the equivalent of 12,500 tokens and the next 19 cost 1,000 each: 12,500 + 19,000 = 31,500 token-equivalents. That is about 84% less for the repeated part of the prompt.
The saving depends on two things: the repeated part must sit at the start of the prompt and stay identical, and the calls must arrive within the cache lifetime. An agent that pauses for ten minutes between steps pays the write price again.
For work nobody is waiting on (evaluations, backfills, nightly summaries), the same page lists the Batch API at 50% off both input and output tokens. A job that costs $400 a month in real time costs $200 in batch.
When neither helps: short, one-off prompts with nothing repeated. There the lever is model choice, covered in section 3 above.
Related guides: Idle GPUs: 5 Ways to Stop Paying for AI Compute You Don't Use and Cloud Cost Alerts That Catch Spikes in Hours, Not Weeks.
The bottom line
Cheaper tokens won't save you if your agents use ten times more of them. The biggest wins come from boring engineering: cache the repeated prompt, cap the loops, route easy work to small models, batch what can wait, keep context lean, and measure cost per feature. Do these six things and your LLM bill starts tracking the value you get, not the loops your agents run.
How LessBill can help
LessBill.org brings your AWS, GCP, Azure and AI API spend into one view, flags the features and workloads driving cost spikes, and points you to concrete savings before the invoice arrives. If your AI bill is growing faster than your usage, it's a good place to start.
Sources and further reading
Also drawn on: McKinsey Quarterly (via MarketScale), Business Insider (RBC Capital Markets CIO survey), Anthropic Claude Platform pricing docs, OpenAI Batch API docs.
Comments
Post a Comment