Data drift vs concept drift vs model drift: how to detect each

Data drift, concept drift and model drift defined

Drift is any change in the world that makes the training data a worse description of production. Terminology varies between vendors and glossaries, so this guide uses the definitions most machine learning literature uses, with x as the input and y as the correct output.

The distinction matters because the responses differ. Harmful data drift is usually fixed by retraining on newer data. Concept drift means the right answers have changed, which needs new labels, or only an update to the rules around the model if that is where the change lives. Model drift is the symptom you notice first; the others are its causes.

Rule out pipeline bugs before calling it drift

Check the data pipeline first, because many failures that look like drift are bugs. The model is unchanged, but a feature pipeline has changed, a source table now means something different, an event stream is dropping records, a timestamp is handled in the wrong timezone, or an upstream service has quietly changed a field's format.

Training-serving skew is the classic case: a feature is computed one way in training and another in production, such as a ‘last 30 days’ window anchored to the wrong clock or missing values filled differently online. It looks like drift, but retraining will not fix it. The cure is shared feature definitions and point-in-time correct training data.

So monitor the data path as seriously as the model: null rates, category distributions, fields that stop arriving and feature freshness. These checks are cheap, and they catch the problems that are quickest to fix.

What to monitor for drift

Watch signals at the input, the model, the logic around it and the outcome, because each catches a different change. Simple signals are enough to start. None proves the model is wrong; each shows that behaviour has changed.

How to detect drift when labels arrive late or never

Monitor proxies continuously and measure real accuracy whenever ground truth arrives. Input distributions, confidence, unknown rate and fallback rate need no labels, so they give early warning, but they cannot prove the model is still right.

That is the main blind spot: concept drift can leave the inputs looking exactly as before. If fraudsters change tactics, the transactions can look normal while the right answer for them has changed. Only outcomes reveal it, so build the label path on purpose.

• Join predictions to delayed outcomes, such as confirmed fraud, a resolved ticket or a returned order, and compute accuracy per week or per cohort.
• Label a small random sample of live traffic by hand on a schedule, so you always have a recent estimate.
• Treat user behaviour as a label: overrides, corrections, escalations and repeat questions are cheap signs of wrong answers.
• Log the model version, input type, prediction, confidence and whether fallback ran with every request, so any later label can be matched to the model that made the call. Keep personal data out of those logs where you can.

Statistical tests and alert thresholds

Compare a recent window of production data with a fixed reference, usually the training data or a known-good period of production, one feature at a time. Each common test suits a kind of feature.

Treat every threshold as a starting point to tune, not a standard. Alert only when someone needs to act, and use sustained conditions rather than single readings: for example, alert only if the unknown rate stays above its threshold for ten consecutive minutes. Signals that fluctuate without needing action belong on a dashboard, grouped into system health, model behaviour, external services and user behaviour panels so the faulty layer is obvious at a glance.

• Population stability index (PSI): bins a numeric feature and sums, across bins, the difference in each bin's share weighted by the log of their ratio. A rule of thumb from credit scoring reads below 0.1 as stable, 0.1 to 0.25 as a moderate shift worth investigating and above 0.25 as significant.
• Kolmogorov-Smirnov test: compares two samples of a continuous feature. On large samples it flags differences too small to matter, so alert on the statistic or a distance rather than the p-value alone.
• Chi-squared test: compares category frequencies, such as predicted classes or a categorical input.
• Jensen-Shannon distance: a bounded, symmetric measure of how far apart two distributions are, useful for whole histograms and prediction mixes.

import numpy as np


def psi(baseline, current, bins=10, eps=1e-6):
    """Population stability index of one numeric feature against a baseline."""
    baseline = np.asarray(baseline, dtype=float)
    current = np.asarray(current, dtype=float)
    # Bin edges from the baseline's quantiles, open at both ends.
    edges = np.unique(np.quantile(baseline, np.linspace(0, 1, bins + 1)))
    edges[0], edges[-1] = -np.inf, np.inf
    expected = np.histogram(baseline, edges)[0] / len(baseline)
    actual = np.histogram(current, edges)[0] / len(current)
    expected = np.clip(expected, eps, None)
    actual = np.clip(actual, eps, None)
    return float(np.sum((actual - expected) * np.log(actual / expected)))


rng = np.random.default_rng(42)
train = rng.normal(0.0, 1.0, 10_000)
print(round(psi(train, rng.normal(0.0, 1.0, 5_000)), 3))  # same distribution: close to 0
print(round(psi(train, rng.normal(0.6, 1.2, 5_000)), 3))  # shifted and wider: above 0.25

How to respond: retrain, repair or update the rules

Identify the type of drift before acting, because the responses differ and some do not touch the model at all.

An illustrative case: weeks after launch, a waste-sorting assistant's bill rose sharply with healthy servers and no errors. The dashboard showed calls to the external language model up threefold and the unknown rate climbing. Users had started photographing compostable packaging that was not in the training data, so the vision model returned unknown and the system asked the LLM instead. The model was working; the dataset was out of date. Adding images of the new packaging and retraining fixed accuracy and cost together. Cost is a drift signal, so know your expected LLM cost per user and alert when it moves.

How often to retrain, and what should trigger it

Retrain as often as your world changes, and let monitoring tell you when that is. The loop of collect, train, evaluate, deploy and monitor might run daily, weekly or only when performance drops; some applications stay stable for long periods.

Hybrid systems need retraining less often. Rules can absorb a change in policy, and a fallback model or human review can cover new cases until the next retrain.

• Scheduled: where data changes steadily, retrain on a fixed cadence from a recent window.
• Triggered: retrain when accuracy on recently labelled data, or a sustained drift alert, crosses its threshold.
• Gated: evaluate every candidate against the current model on recent data before promotion, then release it through a shadow or canary deployment.
• Controlled: version models and the data that trained them, record every deployment, keep a fallback, and require human approval for high-risk systems.

Drift in LLM and RAG applications

LLM applications drift too, but the signals differ because outputs are not deterministic and there is often no label to check. Watch how often the model is called, response time, the share of empty or invalid responses, how often fallback prompts run, and response length and token counts; a jump in tokens often means unexpected prompts or unusual input.

Three sources are common. Users start asking about topics your prompts and documents never covered, which is data drift. The knowledge behind a RAG system goes stale, so a question that looks the same now has a different correct answer, which is concept drift. And a hosted model's behaviour can change when the provider updates it, so pin model versions where the provider allows and rerun your evaluation set before switching. The reliability patterns for LLM APIs cover the fallbacks.

Questions