How much does it cost to run an LLM per user?

The formula, from one call to a monthly bill

Build the estimate from the smallest unit up. Every inference and every API call has a price, and growth multiplies all of them at once, which is why cost is a design constraint rather than a finance report you read after launch. Prices here are per million tokens, the unit most providers quote.

Every term multiplies the others, so halving any one of them halves the bill. That is why the design levers further down usually matter more than the price per token.

1. Cost per call = (input tokens × input price + output tokens × output price) ÷ 1,000,000. If your provider discounts a cached prompt prefix, price the cached and uncached input tokens separately.
2. Escalation rate: the share of user actions that reach the model at all. When rules, lookups or smaller models answer first, this is well below 1.
3. Calls per escalated action: a classification step plus an answer is two calls, and an agent loop can be many more.
4. Retry factor: 1 plus the share of calls that are retried or sent to a fallback provider. Assume a retried call costs as much as a successful one, since a stream that fails halfway can still be billed.
5. Cache factor: 1 minus the share of escalated actions that a response cache answers without calling the model.
6. Cost per user per month = actions per user per month × escalation rate × calls per escalated action × retry factor × cache factor × cost per call. Multiply by monthly active users, then add embeddings, vector search, logging and the servers around the model.

Why input tokens are larger than you expect

Estimates usually count the user's question and the model's answer and forget what the backend sends with them. Measure instead: log the token counts the provider returns with every response, tagged by feature and user, and build the model from those. The ratio of tokens to words differs between tokenisers, languages and content types, so a guess calibrated on English prose will be wrong for code, tables or other languages.

These are the usual sources of the gap:

• The system prompt and template instructions, resent in full on every call.
• Retrieved context: the documents, rules or records added to ground the answer.
• Conversation history. Each turn resends the earlier turns, so a conversation costs more than the sum of its messages.
• Tool and function definitions, which count as input on every call that includes them.
• Few-shot examples kept in the prompt to steer the format.
• Retries and fallback calls, which send the whole prompt again.
• On the output side, reasoning tokens. Some models bill hidden reasoning as output, so a short visible answer can cost far more than its length suggests.

A worked example: the same feature, two designs

The running example in From Model to Product is a phone app that tells households how to dispose of an item: an on-device classifier recognises it, a rule database decides the bin, and a language model answers follow-up questions. The book then asks what happens if the language model handles every request instead. Here is that comparison in numbers.

Every figure here is illustrative, and the prices belong to no provider: for example, a model that charges $1.00 per million input tokens and $4.00 per million output tokens. Each active user takes 40 actions a month, and 10,000 users are active.

Design A sends every action to the model with a generous prompt of 3,600 input tokens (an 800-token system prompt, 1,500 tokens of retrieved context, 1,200 of history and a 100-token question) and gets 250 output tokens back. One call per action, 3% of calls retried, no cache. That is $0.0046 per call, about $0.19 per user per month and about $1,895 a month in total.

Design B follows the book's decision order. Rules and the classifier settle 85% of actions, so 15% reach the model. The prompt carries only the retrieved facts and the last two turns, 1,200 input tokens, and the response format asks for one short instruction, 150 output tokens. A response cache answers 20% of the remaining questions, and retries stay at 3%. That is $0.0018 per call, under one cent per user per month and about $89 a month in total.

The price per token never changed. The bill fell about twentyfold because the system asked the model fewer and smaller questions. Replace the inputs with your own measurements; the structure is the part that transfers.

def cost_per_user_month(actions, escalation, calls, retry_rate, cache_hit,
                        tokens_in, tokens_out, price_in, price_out):
    """Model cost per user per month. Prices are per million tokens."""
    per_call = (tokens_in * price_in + tokens_out * price_out) / 1_000_000
    return (actions * escalation * calls * (1 + retry_rate)
            * (1 - cache_hit) * per_call)


PRICES = dict(price_in=1.00, price_out=4.00)  # illustrative, not real rates
USERS = 10_000

design_a = cost_per_user_month(actions=40, escalation=1.00, calls=1,
                               retry_rate=0.03, cache_hit=0.00,
                               tokens_in=3_600, tokens_out=250, **PRICES)
design_b = cost_per_user_month(actions=40, escalation=0.15, calls=1,
                               retry_rate=0.03, cache_hit=0.20,
                               tokens_in=1_200, tokens_out=150, **PRICES)

for name, per_user in [("A, model on every action", design_a),
                       ("B, rules first, model last", design_b)]:
    print(f"{name}: ${per_user:.4f} per user, ${per_user * USERS:,.0f} a month")

The number that moves: how often the model is called

Price lists change now and then. The escalation rate can change in a week, with no deploy and no error in the logs. Chapter 14 of the book walks through this scenario: weeks after launch the bill rises sharply while the infrastructure looks healthy. The dashboard shows external LLM calls up threefold and the classifier's unknown rate up with them. Users have started photographing compostable packaging that was not in the training data, the classifier returns unknown, and every unknown goes to the language model.

The fix is to retrain the classifier, not to touch the language model. For cost estimates, the lesson is to treat the escalation rate as a live metric rather than a constant, and to chart it next to spend. When inputs drift, the bill is often the first visible symptom; data drift vs concept drift explains how to tell what changed.

Levers that cut LLM cost, in the order to pull them

Each lever shrinks one term of the formula. The order is deliberate: avoid work first, then make the remaining work smaller.

1. Do not call the model. Put rules, lookups and smaller models in front of it, escalate only what they cannot answer, and end in a safe response. This shrinks the escalation rate, usually the largest multiplier. Rules, machine learning or an LLM covers choosing the level for each decision.
2. Send less. Retrieve only the facts the current question needs, keep the template fixed and trim history to the turns that matter. Long prompts are slower and make wrong answers more likely, as well as costing more.
3. Ask for less. A response format rule that asks for a short answer with one clear instruction cuts output tokens, which most providers price above input tokens.
4. Route by difficulty. Send simple questions to a smaller, cheaper model and keep the larger one for hard ones. The client never needs to know which model answered.
5. Cache. Cache rule lookups and answers to common questions, with an expiry time so stale answers age out. If your provider offers prompt caching, put the fixed template first and the variable parts last so the repeated prefix can be reused.
6. Batch what can wait. Summaries, tagging and evaluations do not need a real-time call; group them, and use a provider's batch endpoint where one exists, which trades a delay for a lower price.
7. Cap and degrade. Limit model calls per request and per user, delay low-priority work, and switch the fallback path off under overload instead of letting spend run.
8. Move work off the server. A small model on the user's device removes a server call for every prediction it can make; see edge vs cloud inference. If you host models yourself, use a GPU only when a CPU cannot meet the latency requirement.

Per-user budgets, limits and alerts

An estimate is only useful if production is measured against it. For every model call, log the user or tenant, the feature, the model, input and output tokens, whether it was a retry and whether the cache answered. Cost per user then stops being a projection and becomes a query.

Chart cost signals beside latency and errors: requests per minute, external API calls, tokens used per hour and average cost per request. Alert when token usage exceeds the expected level, when external calls pass a daily threshold, or when average cost per call passes a set limit. Require the condition to hold for a sustained window, such as ten minutes, so one burst does not page anyone.

For hard limits, give each user or tenant a daily token or call budget, enforced in the backend before a request reaches the provider. Decide in advance what happens at the limit: a smaller model, a queued request or a polite refusal.

Questions