How to roll out a new ML model safely: shadow, canary, blue-green or A/B

Four ways to release a new model

Swapping one model for another in a single step puts every user on an unproven version at once, and a model can fail in ways no test caught: different behaviour on real inputs, slower inference, a container that will not start. Controlled release strategies limit how many users a problem can reach and how quickly you can undo it.

Which one to use: a decision table

The strategies are steps, not rivals. A risky model change often goes shadow, then canary, then full rollout, with an A/B test only when the open question is business value. Choose by the question you need answered.

Shadow deployment: mirror traffic without touching users

In a shadow deployment the new model sees the same requests as the live one, but only the live model's answer reaches the user. It is the safest way to learn how a model behaves on real inputs, which never quite match the test set.

You can mirror at two levels. At the infrastructure level, a proxy or service mesh copies requests to a separate shadow deployment and discards its responses; NGINX's mirror directive and Istio's traffic mirroring both work this way. At the application level, the service scores the candidate itself after answering, as in the code below. The application approach is simpler and logs both outputs against one request ID; the infrastructure approach keeps the shadow's CPU and memory off the live instances, which matters for heavy models.

Shadow traffic must be read-only. If the candidate writes to a database, sends messages or calls a paid API, stub those effects or point them at a sandbox, or the shadow stops being invisible. Compare the two versions on agreement rate, score and class distributions, the share of unknown or low-confidence outputs, latency and cost per request, and when ground truth arrives, join it to the logs by request ID and compare accuracy directly.

import json
import logging
import os
import random
import time
import uuid
from contextlib import asynccontextmanager

import joblib
from fastapi import BackgroundTasks, FastAPI
from pydantic import BaseModel, ConfigDict, Field

LIVE = os.getenv("LIVE_VERSION", "1.4.0")
CANDIDATE = os.getenv("CANDIDATE_VERSION", "1.5.0")
SHADOW_RATE = float(os.getenv("SHADOW_RATE", "1.0"))  # share of requests to mirror

logging.basicConfig(level=logging.INFO)
log = logging.getLogger("shadow")
models = {}


@asynccontextmanager
async def lifespan(app: FastAPI):
    models[LIVE] = joblib.load(f"models/churn-{LIVE}.joblib")
    models[CANDIDATE] = joblib.load(f"models/churn-{CANDIDATE}.joblib")
    yield
    models.clear()


app = FastAPI(lifespan=lifespan)


class CustomerFeatures(BaseModel):
    model_config = ConfigDict(extra="forbid")

    tenure_months: int = Field(ge=0, le=600)
    monthly_spend: float = Field(ge=0, le=100_000)
    support_tickets_90d: int = Field(ge=0, le=500)


def score(version: str, row: list) -> tuple[float, float]:
    start = time.perf_counter()
    probability = float(models[version].predict_proba(row)[0][1])
    return probability, (time.perf_counter() - start) * 1000


def compare_in_shadow(request_id: str, row: list, live_p: float, live_ms: float) -> None:
    # Runs after the response has gone, so the user never waits for it.
    try:
        shadow_p, shadow_ms = score(CANDIDATE, row)
    except Exception:
        log.exception("shadow scoring failed")  # logged, never shown to the user
        return
    log.info(json.dumps({
        "request_id": request_id,
        "live": {"version": LIVE, "p": live_p, "ms": round(live_ms, 2)},
        "shadow": {"version": CANDIDATE, "p": shadow_p, "ms": round(shadow_ms, 2)},
    }))


@app.post("/v1/predict")
def predict(features: CustomerFeatures, background: BackgroundTasks) -> dict:
    request_id = str(uuid.uuid4())
    row = [[features.tenure_months, features.monthly_spend, features.support_tickets_90d]]
    live_p, live_ms = score(LIVE, row)
    if random.random() < SHADOW_RATE:
        background.add_task(compare_in_shadow, request_id, row, live_p, live_ms)
    return {"request_id": request_id, "churn_probability": live_p, "model_version": LIVE}

Canary releases: how much traffic, for how long, and what decides

A canary sends a small share of real traffic to the new version, widens it while the metrics hold, and sends everything back if they do not. The chapter's example sends 10 per cent of traffic to the new version and 90 per cent to the old; a risky change can start smaller and widen in steps, for example 1, 5, 25, 50 and then 100 per cent.

Route by a stable key such as a user or account ID rather than at random per request, so each user sees one consistent version and their outcomes can be attributed to it. Stay at each step until the canary has handled enough requests to compare with the live version, and run the whole canary across at least one full cycle of your traffic, such as a weekday and a weekend, because the mix of inputs changes with time.

Write the promotion and rollback rules before you start, and compare the canary with the live version over the same window rather than with last month's figures. The function below encodes one rule set. Its 5 per cent error rate and 20 per cent unknown rate ceilings follow the example alert thresholds in Chapter 14; the other limits are placeholders to set from your own baseline.

import hashlib

# Example limits. Set yours from the live model's normal range.
LIMITS = {
    "error_rate": 0.05,       # never above 5% errors
    "error_margin": 0.01,     # nor more than one point worse than live
    "unknown_rate": 0.20,     # never above 20% "unknown" answers
    "p95_ratio": 1.2,         # p95 latency at most 20% above live
    "confidence_drop": 0.05,  # mean confidence at most 0.05 below live
    "cost_ratio": 1.1,        # cost per request at most 10% above live
}


def in_canary(user_id: str, percent: float) -> bool:
    """Stable routing: the same user always lands on the same version."""
    bucket = int(hashlib.sha256(user_id.encode()).hexdigest(), 16) % 10_000
    return bucket < percent * 100


def canary_verdict(live: dict, canary: dict, min_requests: int) -> str:
    """Compare both versions over the same window. Returns wait, rollback or widen."""
    if canary["requests"] < min_requests:
        return "wait"
    breaches = [
        canary["error_rate"] > LIMITS["error_rate"],
        canary["error_rate"] > live["error_rate"] + LIMITS["error_margin"],
        canary["unknown_rate"] > LIMITS["unknown_rate"],
        canary["p95_ms"] > live["p95_ms"] * LIMITS["p95_ratio"],
        canary["mean_confidence"] < live["mean_confidence"] - LIMITS["confidence_drop"],
        canary["cost_per_request"] > live["cost_per_request"] * LIMITS["cost_ratio"],
    ]
    return "rollback" if any(breaches) else "widen"

What to watch during a rollout

A model can be alive, fast and wrong at the same time, so a healthy server proves little. Watch every layer of the system, and compare the new version with the old rather than only with fixed numbers.

Alert on trends, not single spikes

Alert on sustained changes rather than single readings, for example a rate that stays above its limit for ten consecutive minutes, so noise does not reverse a good rollout and a real regression is not missed. If confidence or the class mix keeps moving after the rollout has settled, the cause may be the world rather than the release; data drift and concept drift explains how to tell them apart. For language model features, give cost per request its own gate, because a new prompt or model can multiply tokens without a single error; LLM cost per user shows how to estimate it.

A/B testing: when the question is whether it is better

Shadow and canary releases answer whether a model is safe. An A/B test answers whether it is better: users are randomly assigned to the old or the new model, and you compare a business outcome such as conversion, retention or tickets resolved. Offline accuracy rarely settles that alone, particularly for ranking, recommendation and pricing models, where the model changes what users do next.

Treat it as an experiment, not a release. Before you start, choose one primary metric and the smallest effect worth acting on, work out the sample size needed to detect it, and randomise by user rather than by request. Run to that size instead of stopping the first time the result looks significant, because repeated peeking inflates false positives. Add guardrail metrics such as latency, error rate and complaints that can stop the test early if the new model does harm, and only A/B test a model that has already passed shadow and canary checks.

Rollback: make the way back boring

Every strategy above assumes you can return to the previous version quickly. That needs old images kept in the registry under immutable tags, old model versions stored with their label mappings and preprocessing, database changes that the previous version can still run against, and configuration, prompts and rules that are versioned rather than edited in place.

Then automate the decision. Progressive delivery controllers such as Argo Rollouts and Flagger shift canary traffic step by step on Kubernetes and abort when metrics fail. If the model version is a configuration value rather than something baked into code, rollback is a one-line change that does not wait for a build. The packaging that makes this possible is covered in how to deploy a machine learning model to production.

Questions