How to deploy a machine learning model to production

Choose the serving pattern first

Most models that reach users are served as a real-time API: a request arrives, the service runs the model, and the answer goes back while the client waits. That is the right default when requests are short, inference is fast and the result is needed now. It is not the only pattern, and picking the wrong one is harder to undo than any line of code.

Where the model runs is an architecture decision, not a deployment detail; AI system architecture: a reference design shows where the serving layer sits in the whole product. This guide takes the common case, a real-time API, and its packaging applies to every pattern.

Six questions every inference API must answer

A good inference API is boring in the best sense: clients know exactly what to send, what comes back and how failures behave, and they never have to reverse-engineer a response. Answer six questions before writing the endpoint, because the contract becomes the stable point the rest of the system is built around. The service in the next section answers them like this.

A minimal FastAPI inference service

The code below is a complete service for a scikit-learn classifier saved with joblib, trained on three features in a fixed order. Swap the features for your own; the patterns stay the same.

Four details carry most of the weight. The model loads once, in the lifespan handler, rather than on every request; lifespan replaces FastAPI's deprecated startup events, and if loading fails the service refuses to start, so a broken image never takes traffic. The endpoint is a plain def, not async def, so FastAPI runs it in a thread pool and CPU-bound inference cannot block the event loop. The request schema forbids extra fields, so a misspelt field earns a 422 instead of a silent default. And the 500 path logs the exception but returns a generic message, because raw exception text leaks internals to callers.

The model path, model version and decision threshold come from environment variables. That is the separation of code, model and configuration in miniature: the threshold is a business setting, and changing it should not need a new build.

Load only model files you produced yourself. Pickle and joblib files can execute code when they are loaded; for artefacts that cross a trust boundary, export to a format such as ONNX instead.

import logging
import os
from contextlib import asynccontextmanager

import joblib
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel, ConfigDict, Field

# Configuration lives outside the code, so a release can change it without a rebuild.
MODEL_PATH = os.getenv("MODEL_PATH", "models/churn.joblib")
MODEL_VERSION = os.getenv("MODEL_VERSION", "dev")
THRESHOLD = float(os.getenv("DECISION_THRESHOLD", "0.5"))

log = logging.getLogger("inference")
ml = {}


@asynccontextmanager
async def lifespan(app: FastAPI):
    # Load once, before the first request. If this fails, the service never starts.
    ml["model"] = joblib.load(MODEL_PATH)
    yield
    ml.clear()


app = FastAPI(title="Churn risk API", version="1.0.0", lifespan=lifespan)


class CustomerFeatures(BaseModel):
    model_config = ConfigDict(extra="forbid")  # unknown fields are rejected, not ignored

    tenure_months: int = Field(ge=0, le=600)
    monthly_spend: float = Field(ge=0, le=100_000)
    support_tickets_90d: int = Field(ge=0, le=500)


class Prediction(BaseModel):
    churn_probability: float = Field(ge=0, le=1)
    will_churn: bool
    model_version: str


@app.get("/health")
def health() -> dict:
    return {"status": "ok", "model_loaded": "model" in ml}


@app.get("/version")
def version() -> dict:
    return {"api_version": app.version, "model_version": MODEL_VERSION}


@app.post("/v1/predict")
def predict(features: CustomerFeatures) -> Prediction:
    # A plain def runs in FastAPI's thread pool, so CPU-bound inference
    # does not block the event loop.
    model = ml.get("model")
    if model is None:
        raise HTTPException(status_code=503, detail="Model not loaded")

    # Same feature order as training.
    row = [[features.tenure_months, features.monthly_spend, features.support_tickets_90d]]
    try:
        probability = float(model.predict_proba(row)[0][1])
    except Exception:
        log.exception("Inference failed")  # full detail in the logs
        raise HTTPException(status_code=500, detail="Inference failed")  # none for the client

    return Prediction(
        churn_probability=probability,
        will_churn=probability >= THRESHOLD,
        model_version=MODEL_VERSION,
    )

Package it with Docker

A container ships the service together with its Python version, libraries and model, so it behaves the same on a laptop, in CI and in production. AI services are unusually sensitive to their environment: a model saved under one version of a library is not guaranteed to load, or to behave the same, under another. For example, scikit-learn warns when a model is loaded under a different version from the one that saved it.

The Dockerfile starts from a slim base image, installs dependencies before copying the code so that layer is cached between builds, and then adds the code and the model. Two additions matter in production: the process runs as an unprivileged user, and the image carries a health check. The fastapi run command comes with fastapi[standard] and listens on 0.0.0.0:8000 by default. Kubernetes ignores Docker's own health check and uses liveness and readiness probes instead, so point those at /health as well.

Pin exact versions in requirements.txt (fastapi[standard], scikit-learn and joblib here), using the versions your tests passed with, captured with pip freeze from that environment. Keep the serving image smaller than the training environment: no notebooks, training frameworks or experiment tracking. Add a .dockerignore for .git, virtual environments and data, and keep secrets such as API keys out of the image entirely; pass them in at run time.

Tag each image immutably, with a semantic version or the commit hash, rather than relying on latest, because a tag you can name is a tag you can roll back to. Run one server process per container and let the platform add replicas.

# Build: docker build -t churn-api:1.4.0 .
# Run:   docker run -p 8000:8000 churn-api:1.4.0
FROM python:3.13-slim

# No .pyc files, logs straight to stdout, no pip cache left in the image
ENV PYTHONDONTWRITEBYTECODE=1 \
    PYTHONUNBUFFERED=1 \
    PIP_NO_CACHE_DIR=1 \
    PIP_DISABLE_PIP_VERSION_CHECK=1

WORKDIR /app

# Dependencies first: this layer is reused until requirements.txt changes
COPY requirements.txt .
RUN pip install -r requirements.txt

# Then the code, and the model artefact it was tested with
COPY main.py .
COPY models/ models/

# The version travels with the image; set both from your pipeline
ENV MODEL_PATH=models/churn.joblib \
    MODEL_VERSION=1.4.0

# Never run as root
RUN useradd appuser
USER appuser

EXPOSE 8000
HEALTHCHECK CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')"

# fastapi run serves on 0.0.0.0:8000 in production mode
CMD ["fastapi", "run", "main.py"]

Ship the model file: bundle, download or mount

Model files are the part of an AI service that general deployment advice forgets. There are three ways to get one into a running container, and the right choice depends on the model's size and how often it changes.

CI/CD that versions more than code

In ordinary software the code defines the behaviour. In an AI system the behaviour also depends on the model weights, the label mapping, the tokenizer, the prompt templates, the rules and the configuration, and changing any one of them changes the system even when no code changed. That holds whether the component is a classifier, a rule engine or a language model feature, so the pipeline must version and release all of them together. A typical pipeline runs four stages, and a failure at any stage stops the release:

1. Test: unit tests, plus checks that the model loads, inference returns valid output, configuration can be read and dependencies connect.
2. Build: the same docker build command every time, from a clean checkout.
3. Push: an immutably tagged image to a registry, so servers pull exactly the build that passed.
4. Deploy: a release strategy that limits the damage a bad version can do, with the previous version kept ready.

Know which model is running, and prove it works

A model registry records each model's version, training parameters, metrics and deployment status. MLflow or a cloud registry will do, and for a small system versioned object storage is enough. What matters is that you can always say which model is serving traffic, rebuild it from its inputs, and trace how it got there.

The smoke test below is the cheapest insurance in the pipeline. It stops an image that cannot load its model, answer a valid request or reject a bad one from ever being built. Open TestClient in a with statement, as here; without it the lifespan never runs and the model never loads.

from fastapi.testclient import TestClient

from main import app

GOOD = {"tenure_months": 18, "monthly_spend": 42.5, "support_tickets_90d": 3}


def test_model_loads_and_predicts():
    with TestClient(app) as client:  # "with" runs the lifespan, so the model loads
        assert client.get("/health").json()["model_loaded"] is True
        response = client.post("/v1/predict", json=GOOD)
        assert response.status_code == 200
        body = response.json()
        assert 0 <= body["churn_probability"] <= 1
        assert body["model_version"] == client.get("/version").json()["model_version"]


def test_bad_input_is_rejected_before_inference():
    with TestClient(app) as client:
        assert client.post("/v1/predict", json={**GOOD, "tenure_months": -1}).status_code == 422
        assert client.post("/v1/predict", json={**GOOD, "surprise": 1}).status_code == 422

Release gradually and keep a way back

Replacing the running version in one step is the riskiest release there is. Blue-green, canary and shadow releases let a new model meet real traffic gradually; how to roll out a new ML model safely compares them. Whichever you use, rollback is only possible when four things hold:

• Old images stay in the registry under their own tags.
• Old model versions are stored with their label mappings and preprocessing.
• Database changes remain compatible with the previous version.
• Configuration, prompts and rules are versioned rather than edited in place.

Questions