LLM inference has become a line item on most engineering budgets, and unlike compute infrastructure, the per-unit cost is opaque and easy to underestimate. A single popular feature can generate millions of tokens per day. Teams discover their bill too late. This explainer covers four proven levers for reducing LLM inference cost without cutting features or training new models: prompt caching, model routing, quantization, and batching. These are not theoretical optimizations. They are in production at teams managing billions of tokens monthly and delivering 30 to 60 percent cost reductions. The goal is to give engineering leads and FinOps owners the language and decision framework to deploy them.
Why this matters now
In 2026, LLM inference cost has decoupled from Moore's Law. Chip density improves, but token pricing remains stubbornly flat or rises with model quality improvements. A team operating a chat application, internal search, or content generation pipeline will spend more on inference than on serving the equivalent amount of traditional API traffic. The difference: LLM tokens scale linearly with user input size and output length, not just request count. A single customer question can consume 10,000 tokens. Ten thousand customers per day means 100 million tokens. At $0.001 per 1,000 input tokens (a baseline rate for mid-tier models), that is $100 per day, or roughly $3,000 per month before any optimization.
The math gets worse under growth. Doubling users doubles token volume. Enabling new features or increasing context window size multiplies it. Unlike database queries or API calls, token cost has no natural ceiling without deliberate intervention. This is why LLM cost optimization is no longer optional for teams with inference workloads larger than a few million tokens per month. The return on engineering time is measurable: a 40 percent cost reduction on a $10,000 monthly bill is $4,000 saved with a single quarter's work. More importantly, cost controls remove the friction from adopting new models or features.
Prompt caching: Free compute for repeated context
Prompt caching is the single highest-leverage optimization for most workloads. When an LLM processes a prompt, it computes a cache key for the input tokens. If the same tokens appear again within a retention window (typically 5 to 24 hours), the model skips recomputation and charges 90 percent less for cached tokens. This sounds simple because it is. The complexity is in architecture.
Caching works best when your prompts contain stable, expensive context that repeats across requests. Examples include: a large document that multiple users analyze; a system prompt or instruction set that drives 100 percent of queries; a company knowledge base prepended to every search result. In each case, the first request pays full price. Subsequent requests with the same cached prefix pay 10 percent of the token cost. A document 50,000 tokens long, queried 100 times, costs 50,000 + (99 * 5,000) = 545,000 tokens without caching, and 50,000 + (99 * 500) = 99,500 tokens with caching. The savings: 82 percent on that workload.
Implementation requires discipline. Cache keys are computed from exact token sequences. Adding a whitespace character or rephrasing a sentence invalidates the cache. Teams should hardcode stable prompts, version them in code, and avoid runtime concatenation of variable text. Tools like Anthropic's prompt caching or OpenAI's cache control headers make this straightforward, but you must structure prompts intentionally. The pattern is: fixed preamble (cached) followed by user input (uncached). Never interleave them.
Cache hit rates of 50 to 70 percent are realistic for systems with document analysis, knowledge base augmentation, or multi-turn conversations where the system prompt repeats. Teams should instrument caching from day one, logging cache hits and misses by feature. Monitor the cost per cache hit separately from the cost per miss. Unexpected drops in hit rate often signal a bug (a timestamp or UUID accidentally embedded in the prompt) or a change in user behavior. Set alerts on cache hit ratio degradation.
Model routing: Right-sizing inference for each task
Not every query needs a frontier model. A customer service chatbot answering frequently asked questions does not need GPT-4 class performance. A classification task deciding between three product categories does not need the same model as a customer writing a long-form report. Model routing is the discipline of sending each inference request to the smallest model that will produce acceptable output.
The savings are substantial. A small model (e.g., Llama 2 7B or Mistral 7B quantized) costs 10 to 20 percent of a frontier model. If 60 percent of your queries can run on a small model, you save roughly 40 percent on total token cost. The catch: you must be willing to measure quality per request type and accept occasional accuracy degradation.
Implementing routing requires a decision tree. Start by running your actual workload through multiple models: a small model, a mid-tier model, and a frontier model. Log the outputs and measure accuracy, latency, and cost. For classification tasks, measure precision and recall. For generation tasks, measure factual accuracy or human-rated relevance. Once you have baseline data, route based on query attributes: complexity, token count, or a quick confidence check from the small model.
A concrete example: a support ticket system can first classify incoming tickets with a small, fast model into categories like "billing", "technical", or "account". If confidence is above 95 percent, send a templated response. If below 95 percent, escalate to a larger model or a human. A tax accounting tool might use token count as a signal: if the uploaded document is under 5,000 tokens, route to a mid-tier model. If above 20,000 tokens, route to a frontier model with better long-context reasoning. These decisions are context-dependent, but they all follow the pattern: measure once, route consistently, monitor quality metrics in production.
One critical rule: never route silently. Log every routing decision and the model used. When a small model produces a bad answer, it should be flagged in monitoring, not buried. Some teams implement a fallback mechanism: if a small model's response is rated low by users, automatically reprocess with a larger model and absorb the cost as a customer recovery expense. This turns model routing into a quality mechanism, not just a cost control.
Quantization: Trading precision for speed and cost
Quantization is the process of reducing the precision of a model's weights and activations. A standard model uses 16-bit or 32-bit floating point numbers. A quantized model uses 8-bit or 4-bit integers. The effect on cost and speed is immediate: a 7B parameter model quantized to int4 can run on cheaper hardware, faster, and with lower memory bandwidth. In hosted APIs, quantized models may also receive pricing discounts.
The trade-off is accuracy. A quantization-aware trained (QAT) model minimizes loss, but even the best quantized models degrade somewhat. The degradation depends on the task. Classification and retrieval tasks are robust. Generation tasks can show measurable differences. A quantized model might hallucinate slightly more or produce output with lower coherence on long sequences. The only way to know is to benchmark.
Practically, teams should test quantization on a holdout test set that mirrors production. Run 500 to 1,000 representative queries through both the base and quantized model. Measure the difference in output quality using task-specific metrics. For a summarization task, compare ROUGE scores or human ratings. For a classification task, compare accuracy. If the quantized model is within your quality threshold (often 95 to 99 percent of base performance), deploy it. If not, keep the base model or use quantization only for low-stakes queries (e.g., internal tools).
Quantized models are now widely available. Hugging Face, GGML, and vLLM all offer pre-quantized checkpoints. If you are self-hosting, quantization is straightforward. If you rely on hosted APIs like OpenAI or Anthropic, quantized variants may not be available, and you cannot quantize the model yourself. In that case, quantization is off the table unless you switch providers or self-host. This is a constraint worth understanding early.
Batching: Shifting urgency to reduce per-token cost
Batching groups multiple inference requests into a single batch, amortizing overhead and increasing GPU utilization. Hosted APIs often price batch requests at 50 percent of real-time rates, as batch processing allows the provider to schedule inference during off-peak hours and pack requests more densely on hardware. For teams self-hosting, batching reduces per-request latency and cost per token.
The trade-off is latency. A real-time API returns results in milliseconds. A batch job may process requests hours or days later. This is acceptable for reporting, bulk analysis, async notification generation, and any workflow where the user does not wait. It is not acceptable for customer-facing chat, real-time search, or any interactive feature.
Practical batching strategies: set up a daily or hourly batch job for asynchronous workloads. Examples include generating personalized emails for a user cohort, analyzing uploaded documents in bulk, or reranking search results overnight. Use real-time APIs only for features where latency matters. The cost savings are measurable. If 30 percent of your token volume is batchable, moving that 30 percent to a batch API reduces overall inference cost by roughly 15 percent (30 percent of volume at 50 percent of per-token cost).
Implementation is straightforward: collect requests in a queue, set a batch threshold (e.g., 1,000 requests or 1 hour), submit the batch, and process results asynchronously. Use batch IDs to track results back to users. Most LLM providers (OpenAI, Anthropic, Cohere) offer batch APIs with clear pricing and SLAs. The engineering lift is moderate: a job queue, a scheduler, and a results processor.
Monitoring and governance: The unsexy necessity
Optimization without monitoring is guesswork. Teams should instrument LLM inference cost from day one, treating it like database query cost or API latency. Every inference should log: the model name, input token count, output token count, timestamp, user or account ID, and feature name. Push these logs to a data warehouse (BigQuery, Snowflake) or BI tool (Datadog, Grafana, Redshift).
Establish dashboards that answer: What is our total monthly inference cost? Which features drive the most cost? What is the cost per user? What is our token efficiency trend? Which models are we using and at what volume? Segment by model, feature, and time window. Set alerts for: total monthly cost exceeding a threshold, cost per user spiking, or a feature's cost growing faster than expected.
Governance is equally important. Assign ownership of inference cost to engineering teams or feature owners. Each team should know their monthly budget and actual spend. When a team ships a new feature, require a cost estimate (based on token projections) and track actual cost post-launch. If a team's feature costs 10x more than estimated, that is a signal to optimize or deprioritize.
In practice, this means: a FinOps dashboard visible to all engineers, monthly cost reviews with leadership, and cost-aware code review. Engineers should ask, "How many tokens will this generate?" the same way they ask about latency or memory. Culture shifts when cost is visible and owned.
When optimization hits limits
These four levers are powerful, but they have constraints. Prompt caching requires stable, repeating context; a system where each query is entirely novel will see near-zero cache hits. Model routing requires quality tolerance; if your domain demands frontier model performance on every task, routing gains nothing. Quantization assumes your workload is robust to precision loss; high-stakes financial or medical advice may not be. Batching requires latency tolerance; real-time customer-facing features cannot use it.
Some teams will implement all four and plateau at 50 to 60 percent cost reduction. Others may not be able to implement any. The right approach is to audit your workload honestly. Are your prompts repetitive? Are your tasks varied in complexity? Can you tolerate longer latencies on some requests? Can you accept small quality degradation? Answer these questions, then pick the levers that apply.
There is also a limit to how far engineering can push. After caching, routing, quantization, and batching are maxed out, further cost reduction requires either architectural change (fewer tokens per task), model switching (moving to a cheaper provider), or volume reduction (narrower use cases). These are business decisions, not engineering optimizations. Know when you have hit that wall.
Begin with monitoring and baseline cost. Run one optimization pilot (usually caching or routing) and measure the impact before layering in others. Communicate cost savings to stakeholders not as a cost-cutting exercise, but as a mechanism that unblocks new features and reduces the friction of scaling. Teams that treat LLM cost as a first-class metric will find optimization natural and continuous.
