AI system architecture: a reference design for production AI products
A production AI system is usually a client, an API layer, product logic that owns every decision, one or more inference components (rules, classical models, vision models, language models), a data layer, batch jobs for offline work, and monitoring around all of it. Start with the simplest version of that shape that survives your real load, change and failure, often one backend and one database, and split it only when a component needs different scaling, hardware or release cycles.
Start from constraints, not components
There is no single best architecture for AI products. The right one depends on how fast the product must respond, how much traffic it expects, how expensive inference is, how often each part changes and which failures are acceptable. Architecture is a sequence of choices under those constraints: monolith or services, request-response or events, online or offline inference, CPU or GPU, and what happens when each part fails. Before drawing any boxes, answer eight questions:
- What is the input?
- Where can computation run: on the device, on your servers or through an external API?
- How fast must the response be?
- How many users will there be, and how bursty is the traffic?
- How often will the data and rules change?
- How expensive is each request?
- What must never fail?
- What can fail safely?
Let the answers rule designs out
The answers eliminate designs faster than any pattern catalogue. If the input is a photo and the answer must feel instant, the model probably runs on the device. If every request costs money, the architecture needs a way to avoid calling the model at all. If something must never fail, it needs a fallback that does not depend on the component most likely to fail. Choosing the right level of intelligence for each decision comes before any of this; rules, machine learning or an LLM covers that choice.
The reference architecture, layer by layer
Many production AI products converge on the same shape. Treat it as a reference rather than a template: a small system collapses several layers into one process, and a large one splits each layer into services.
| Layer | Job | AI-specific concerns |
|---|---|---|
| Client | Captures input and shows results | May run a small model on the device; never holds API keys or decision rules |
| Gateway or API | Authentication, validation, rate limits, routing | Strict request and response schemas, versioned endpoints, backpressure when inference is saturated |
| Product logic | Turns predictions into decisions and runs the workflow | Confidence thresholds, rules, fallback routing, prompt assembly, caching |
| Inference | Runs the models: rules engines, classical ML, vision, language models | CPU or GPU placement, model versions, timeouts, provider fallback |
| Data and features | Records, features, rules, sessions, documents for retrieval | Features computed the same way in training and serving; storage split by purpose as the system grows |
| Batch and orchestration | Retraining, feature recomputation, scheduled scoring, offline evaluation | Jobs with dependencies that run outside any user request |
| Monitoring | Logs, metrics, traces and alerts around every layer | Confidence, unknown and fallback rates, drift signals and cost, not only uptime |
Where caching, guardrails and evaluation sit
Decisions belong in product logic, not in a model: models recognise, rank or explain, and explicit logic decides what the product does. Caching sits in product logic too, in front of whatever is expensive, whether that is rule lookups, feature retrieval or language model answers to common questions. Guardrails sit on both sides of a language model: the backend grounds and constrains the prompt it builds, then validates the output before anything reaches the user. Evaluation runs twice, offline in the batch layer before a release and online through monitoring after it.
Combining rules, classical ML, vision and LLMs
Most real products need more than one kind of intelligence, and the dependable way to combine them is a decision hierarchy: try the simplest method first and escalate only when it cannot answer. Check the rules; use the classifier when no rule decides; call a larger fallback model when the classifier is uncertain; use the language model for open questions; and return a safe message when the language model cannot answer. Complexity rises only where the problem demands it, and each step can be measured on its own.
Language model components change the architecture more than a single API call suggests. Provider failures need a design of their own, set out in running LLM APIs reliably in production.
| Concern | Classical ML or vision model | LLM component |
|---|---|---|
| Where it runs | In process, in your own inference service or on the device | Usually an external API, sometimes self-hosted on GPUs |
| Latency | Milliseconds | Often seconds, which pushes long tasks towards queues or streaming |
| Cost | Mostly fixed compute | Per token, rising with usage and prompt size |
| What you version | Model weights, labels, preprocessing | Model and provider, prompt templates, retrieval data, output rules |
| Typical failures | Wrong or low-confidence predictions | Timeouts, rate limits, outages, confident answers with no grounding |
| Architectural answer | Confidence thresholds and an explicit unknown class | Grounded prompts built by the backend, output validation, retries, provider fallback, caching |
Worked example: HomeSort AI, version one
From Model to Product builds one system end to end to show these layers working together: HomeSort AI, a browser-based assistant that tells a household how to dispose of an item from a photo. It combines edge computer vision, a rule engine, a document database, a fallback model and a language model, which makes it a compact reference for mixed systems. Version one runs in nine steps:
- The browser captures an image.
- A small vision model on the device predicts a label and a confidence, such as a plastic bottle at 0.82. The client applies no rules and holds no API keys.
- The backend receives the result through a thin endpoint that only validates it and hands off.
- If confidence is below the threshold, the backend, never the client, calls a slower but more accurate fallback model in the cloud.
- The backend looks up the disposal rule for that label and region in the database.
- It stores the result against a session, so follow-up questions have context without a login.
- When the user asks a question, the backend fetches the session and the rule and builds a grounded prompt.
- The language model is called through the backend, which adds the keys and handles provider fallback.
- Only the text the client needs goes back to the browser.
What if the assumptions change?
Four early decisions keep version one cheap to change: rules live in the database rather than in code, the backend controls every external call, sessions are separate from user accounts, and the language model is optional, so it explains decisions but never makes them. Version one also assumes few users at a time, a small rule set, a single backend and a device that can run the model. Real projects break those assumptions, and the architect's job is to know what each break costs. The first question is always where the slowest step is, because it sets the speed of the whole system.
| If | The architecture becomes |
|---|---|
| The device cannot run the model | Images are uploaded and the model runs on the server: larger requests, server CPU or GPU, more latency and cost |
| The model outgrows the device | Hybrid: a small model on the device, a large one in the cloud, and the backend decides which to use; see edge or cloud inference |
| Thousands of users arrive at once | Several backend instances behind a load balancer, a queue in front of slow models, database connection pools |
| A model call takes seconds | The backend accepts the request, a worker does the work later, and the client collects the result |
| The language model bill grows | Cache common answers, route simple questions to a smaller model, call the model only when needed, limit requests per user |
| The database slows down | Separate stores for rules, logs, analytics and cache, because the requirements changed, not because the first choice was wrong |
| The input becomes long documents or video | Server-side models, much slower requests, background jobs, stored intermediate results and a progress indicator |
Simple for a startup, structured at scale
A small team should usually start with one backend, one database and one model, often as a monolith that serves the API, the product logic and the model in one deployable unit. It has fewer network hops and contracts, it is easier to debug, and it lets the team learn where the real boundaries are. A system that works and can change beats one that is perfectly designed and never finished.
Split a component out when its boundary becomes real: it must scale faster than the rest, needs different hardware such as a GPU, changes on a different rhythm, or must be isolated so it cannot take the product down. Keep services as stateless as practical, with state in databases, caches and queues, so any healthy instance can serve any request. As scale grows, what hurts changes, and so do the priorities:
- Small: speed of development and clarity, with a monolith, a simple database, modest monitoring and one deployment path.
- Medium: uneven load, where caching, replication, background workers, better observability and clearer service boundaries start to earn their cost.
- Large: resilience and cost control, with regional placement, fault isolation, graceful degradation, autoscaling and plans for slow dependencies and spikes on expensive inference.
If this chapter leaves you with one durable idea, let it be this: good AI architecture is rarely about making the system look impressive. It is about making the system behave honestly under load, under change, and under failure.
Questions
When does an AI system need a queue instead of synchronous calls?
Keep a request synchronous when it finishes quickly and a user is waiting for it. Add a queue when inference is slow or heavy, traffic arrives in bursts, GPU capacity is limited, or the user does not need the answer within a few hundred milliseconds, as with document analysis, media processing or long generations. The API accepts the job and returns an ID, and a worker does the work, which protects expensive capacity and smooths spikes.
Should an AI product start as a monolith or as microservices?
Usually as a monolith. One deployable unit is easier to build, debug and run while the product and its boundaries are still changing. Microservices bring independent scaling and team autonomy but also network latency, retries, timeouts and harder debugging, so extract a service when a boundary is real, most often when the inference path needs GPUs or a faster release cycle than the rest of the product.
Should the browser or mobile app call the model API directly?
Not for external or paid models. Calling a language model or cloud model from the client exposes API keys, scatters decisions across clients and leaves nowhere to add caching, rate limits or provider fallback. Route those calls through the backend. A small model running on the device is different: it can stay on the client, as long as its output goes to the backend, which makes the decision.
Where should decisions live when a product uses a language model?
In explicit backend logic. Let rules and classifiers decide, pass the decided values into the prompt, and ask the language model only to explain them or answer questions about them. That keeps decisions consistent, testable and auditable, limits what a hallucination can change, and lets you swap models or providers without changing what the product does.
How does an LLM application architecture differ from classical ML serving?
The outer layers stay the same: an API, product logic, a data layer and monitoring. What changes is the inference path, which usually becomes a call to an external model that takes seconds, costs money per token and can answer confidently without grounding, so the backend gains prompt assembly, retrieval, output validation, caching, retries and provider fallback, and monitoring gains token and cost tracking. Retrieval can often start with ordinary database queries and move to vector search only when the text collection demands it.
Is there one best architecture for AI products?
No. The architecture comes from the requirements, not from the model: latency, traffic, cost per request, how often parts change and what must never fail decide it. The same waste-sorting product would look very different if it had to run offline, analyse video or serve a very large number of users. Every architecture is temporary, so the measure of a good one is how cheaply it can change.