How do you make LLM API calls reliable in production?

Start with when not to call the model

The most reliable LLM call is the one you did not need to make. Language models are flexible, but they are slower, costlier and less predictable than a rule or a database lookup, so the book puts them near the end of a fixed decision order. The system always tries the simplest method first and escalates only when it must, and every request the early steps answer is one that cannot time out at a provider. Rules, machine learning or an LLM covers choosing the level for each decision.

1. Rules: if a rule already gives the answer, use it.
2. Classifier: if no rule applies, use the trained model's prediction when it is confident.
3. Fallback model: if the classifier is uncertain, try a stronger model.
4. Language model: when the user asks a question, or needs an explanation rather than a label.
5. Safe response: if the language model cannot answer, return a fixed, honest message.

Backend decides, model explains

When the model is called, it should not be the source of truth. In the book's worked example the classifier and the rule database decide which bin an item belongs in, and the language model only explains that decision and answers follow-up questions. The backend fills a prompt template with the facts it has already decided, and the prompt says plainly that those facts must not be changed.

This is grounding: the model is given the facts it must use instead of guessing them. It keeps answers consistent when the wording varies, and it makes guardrails simple, because the backend knows the correct value before the model replies and can check the reply against it.

Guardrails run on both sides of the call. Before it, validate and limit the user's input, retrieve the facts and fill the template. After it, reject empty or malformed output, check that every value the backend decided appears unchanged, enforce length, and return the safe response if a check fails. Deterministic checks like these add almost no latency; a second model acting as a judge adds a whole extra call, so keep that for outputs where code cannot decide.

Build the prompt template in five stages

A raw prompt fails in production because the model does not know the user's region, what the classifier saw or which rules apply. The book grows one template in five stages, each added to close a failure the previous version allowed.

Stricter rules do not eliminate hallucination, but they make behaviour far more predictable. Version the template with the code, because templates keep growing as new failure cases appear; From Model to Product prints the full Python template at each stage.

1. Context. State the application's role and fill placeholders for the recognised object, the decided category, notes from the rules database and the user's question.
2. Locked decisions. Say that the category was decided by the backend and must not be changed, and that the task is only to explain.
3. Parameters. Replace hard-coded assumptions, such as one country's rules, with placeholders like region, so the template extends without a rewrite.
4. Anti-hallucination rules. Use only the information provided, do not invent new rules, do not guess missing details.
5. Response format. Keep the answer short, give one clear instruction and add no extra discussion, so the client can display it and output tokens stay low.

Retrieval without a vector database

When the rules cannot answer a question, do not stuff every document into the prompt; long prompts are slower and make wrong answers more likely. Retrieve, then generate: the backend receives the question, searches its own data for the relevant facts, puts only those facts in the prompt, and the model writes the answer.

This does not need a vector database. In the book's example the knowledge is small and structured (rules, region settings, short notes on special cases such as batteries or electronics), and ordinary database queries find it. Embeddings, index maintenance and another store would add moving parts, each one more thing that can fail, with no clear benefit. Add vector search when the volume of free text genuinely needs semantic search; pgvector or a vector database covers that choice.

Timeouts, rate limits and what to retry

Every call needs a timeout, or one slow provider ties up a worker, then a queue, then the whole backend: the cascading failure that reliability patterns exist to stop. Measure the provider's latency for your prompt sizes, set the timeout just above the slowest normal response, and check that all attempts together still fit the user's latency budget.

Streaming calls need two timers instead of one total: a limit on the time to the first token, and an idle limit between chunks. A long answer that is still streaming steadily should not be cut off by a total timeout chosen for short ones.

Providers usually limit both requests per minute and tokens per minute. Track both on your side, estimate a request's tokens before sending it, and queue or shed low-priority work before the provider has to refuse it. Then retry only what a retry can fix:

Retries with backoff and jitter, then a fallback provider

Retry transient failures, but never immediately and never in step with everyone else. Exponential backoff doubles the longest permitted wait after each attempt, so a struggling service gets room to recover; jitter picks a random wait up to that limit, so a thousand clients that failed together do not retry together. When one provider's retries run out, the backend tries the next provider, or a smaller model, and when every option has failed it returns the safe message. The client never knows which path answered.

The sample is vendor-neutral. Each call function wraps one provider's SDK or HTTP call, sets its own request timeout, and raises TransientError for timeouts, 429 and 5xx responses, or PermanentError for other 4xx responses. Any other exception is a bug and is left to surface.

import random
import time


class TransientError(Exception):
    """A timeout, 429 or 5xx response: worth retrying after a wait."""

    def __init__(self, message="", retry_after=None):
        super().__init__(message)
        self.retry_after = retry_after  # seconds, from a Retry-After header


class PermanentError(Exception):
    """Any other 4xx response: the same request will fail again."""


SAFE_MESSAGE = "Sorry, I can't answer that right now. The rest of the app still works."


def call_with_retry(call, prompt, attempts=4, base=1.0, cap=8.0, budget=20.0):
    """Call one provider, retrying transient failures with full jitter."""
    start = time.monotonic()
    for attempt in range(attempts):
        try:
            return call(prompt)
        except TransientError as err:
            wait = err.retry_after
            if wait is None:
                wait = random.uniform(0, min(cap, base * 2 ** attempt))
            last_try = attempt == attempts - 1
            if last_try or time.monotonic() - start + wait > budget:
                raise
            time.sleep(wait)


def answer(prompt, providers):
    """Try each provider in order and return the safe message if all fail."""
    for call in providers:
        try:
            return call_with_retry(call, prompt)
        except (TransientError, PermanentError):
            continue
    return SAFE_MESSAGE


if __name__ == "__main__":
    def primary(prompt):
        raise TransientError("503 Service Unavailable")

    def backup(prompt):
        return f"Backup provider answered: {prompt}"

    print(answer("Can a glass jar go in dry recycling?", [primary, backup]))

Circuit breakers, graceful degradation and swappable providers

Retries absorb blips but make a real outage worse, because every request now costs several calls to a service that is already down. A circuit breaker stops that. Closed, calls flow normally. Open, after repeated failures in a short window, calls skip the provider and go straight to the fallback. Half-open, after a cooldown, one probe call tests the provider: success closes the breaker, failure opens it again. Keep one breaker per provider and model, check it before the retry loop, and record every outcome in it.

When no provider is available, degrade rather than fail. If the language model API is offline, switch the chat feature off for a while and keep the camera, the classifier and the rules working. Decide each feature's degraded behaviour in advance, and make the safe message honest about what the user can still do.

Finally, make the provider a configuration choice. Keep the ordered provider list, model names, timeouts and token limits in configuration behind one call interface, so adding a backup, routing simple questions to a smaller model or switching providers needs no client change. Chart the fallback rate: a primary that quietly fails over more often is an incident the error logs will not show. Shadow, canary and A/B deployment covers trialling a new model or provider on live traffic.

Questions