How to monitor a vibe-coded app in production

Running is not the same as working

An uptime monitor answers one question: does the server respond? For an app that calls a language model, reads prompts or settings from a database and leans on third-party services, that is the easy question. Chapter 14 of the book, on observability, starts from the harder one: is the system still behaving the way it should?

Ordinary web software tends to fail loudly. It throws an error, stops responding or crashes, and somebody notices. AI features tend to fail quietly. The model still answers, but less usefully. The endpoint still returns 200, but slower than last week. An error handler returns a polite fallback message, so nothing reaches the error log. CPU and memory look normal while users stop trusting the feature.

Behaviour can also change when no code has changed. Users send inputs you never tried. A provider moves the model alias you call to a newer version. Someone edits a prompt, a price or a rule stored in the database. None of these raises an exception, which is why monitoring has to measure behaviour, not just catch errors. Monitoring is rarely part of the prompts that build an app, so plan to add it yourself; taking a vibe-coded app to production puts it in order with the rest.

Think in layers: what to watch at each one

The chapter's model is to treat the system as a stack of layers, each producing its own signals. An app can look healthy at one layer and be broken at another, so watch all of them.

Cover the loud failures first

Before any dashboard, catch the app that is down. These four are cheap to add:

1. An uptime check from outside your host that loads a real page and calls a health endpoint every minute or so, and messages your phone when either fails. A monitor that runs on the same platform as the app can go down with it.
2. A health endpoint that reports whether the process is alive and whether its dependencies answer: the database responds and required configuration is present. Keep it cheap. Do not call the paid model on every probe, because the probes cost money and a provider outage would mark the whole app as down. If your platform restarts instances that fail a check, keep that check to the process itself; a restart will not fix a database outage.
3. Error tracking on the server and in the browser, so an exception arrives once, grouped, with its stack trace, release and route.
4. Spending alerts at the model provider and the host, wherever they offer them, set just above what you expect to spend.

Log the right information, once per request

Logs show events, metrics show trends and monitoring shows behaviour over time. A structured log is the cheapest place to start, because most tools can count and chart its fields. The book's advice is to record what you will need to explain a result later: the model version, the input type, the result and whether fallback logic ran, plus the response time and outcome of every external call.

For an app built with AI tools, that means one JSON line per request and one per AI call, sharing a request ID so you can follow a single user action through every step. Send them somewhere searchable that keeps them as long as you might need to look back. Record:

• Request ID, route, release or commit, and a pseudonymous user or account ID.
• Duration and status for the request, and for every database and third-party call inside it.
• For AI calls: feature, provider, model and version, prompt version, input and output tokens, latency, outcome (ok, empty, invalid or error), retries and whether a fallback answered.
• The decision path: which cache, rule or model produced the answer.
• Nothing secret. Never log API keys, access tokens, passwords, session identifiers or connection strings, and keep personal data, raw prompts and uploaded files out of routine logs; the OWASP Logging Cheat Sheet lists what to exclude.

import { randomUUID } from "node:crypto";

type ModelReply = { text: string; inputTokens: number; outputTokens: number };

// Wrap every model call: time it, record the outcome and log one JSON line.
async function loggedModelCall(
  requestId: string,
  feature: string,
  model: string,
  call: () => Promise<ModelReply>,
  fallback: string,
): Promise<string> {
  const started = performance.now();
  const log: Record<string, unknown> = { event: "model_call", requestId, feature, model };
  let answer = fallback;
  try {
    const reply = await call();
    log.inputTokens = reply.inputTokens;
    log.outputTokens = reply.outputTokens;
    log.outcome = reply.text.trim() ? "ok" : "empty";
    if (log.outcome === "ok") answer = reply.text;
  } catch (err) {
    log.outcome = "error";
    log.error = err instanceof Error ? err.name : "unknown";
  }
  log.fallbackUsed = log.outcome !== "ok";
  log.ms = Math.round(performance.now() - started);
  console.log(JSON.stringify(log));
  return answer;
}

// A substitute for your provider's SDK call. This one returns an empty answer.
const fakeCall = async (): Promise<ModelReply> => ({ text: "", inputTokens: 812, outputTokens: 0 });

loggedModelCall(randomUUID(), "summary", "example-model-v1", fakeCall, "Summaries are unavailable right now.")
  .then((answer) => console.log(answer));

Watch the AI feature, its fallbacks and its bill

Follow one user action through your app: your code, perhaps a cache or a database lookup, a model call, then checks on the output. Your data should answer four questions about that path: how often each step runs, how often the app falls back, how often a lookup finds nothing, and how long each step takes.

Time each step separately, so a rise in total latency points straight at your database, your code or the model provider, whose speed can change without warning. That needs every external call to go through one place in your backend, with fallback logic explicit. If model calls are scattered across components, gathering them into one module is the first monitoring task.

Count fallbacks as a rate. A fallback keeps the app usable and also hides the failure, so the share of requests answered by a fallback, a cached default or an apology is one of the earliest signs that something upstream changed. For language model calls, also chart call volume, response time, empty or invalid responses and tokens per call; a jump in tokens often means unexpected prompts or unusual input.

Treat cost as a signal, not only a finance report. If the bill rises while traffic is flat, behaviour has changed: more fallbacks, longer prompts, a retry loop or a feature used in a way you did not expect. LLM cost per user covers per-user budgets and cost alerts. If the inputs themselves are shifting, data drift vs concept drift explains how to tell what changed.

One dashboard, four panels

A good dashboard lets you judge the app's health within seconds. The book groups an AI system's metrics into four panels so the faulty part is obvious; here they are for a typical app with an AI feature. Google's chapter on monitoring distributed systems names latency, traffic, errors and saturation as the four golden signals, and they belong in the first panel.

Alerts worth waking up for

Dashboards are for looking; alerts are for acting. The chapter sorts alerts into three kinds, and an app with an AI feature needs all of them. Infrastructure alerts: the app is down, response time stays above your limit, memory nears its ceiling, database connections fail. Behaviour alerts: the fallback or empty-response rate doubles, or the mix of outputs shifts suddenly. Cost and usage alerts: tokens or model calls pass the expected daily level, or cost per request passes a set limit. Start with simple thresholds, such as the book's example of an API error rate above five percent, and tune them on your own traffic.

Alert fatigue is the real risk for a solo builder or a small team: when alerts fire constantly, people stop reading them and miss the one that matters. Two rules from the chapter prevent most of it. Alert only when someone needs to act, and require a condition to hold for a sustained window, such as ten minutes, rather than one reading. Signals that wobble without needing action belong on the dashboard. Write the first step into each alert, such as which panel to open, so the thinking is not left for the worst moment.

The monitoring checklist

Chapter 14 gives a monitoring checklist. Here it is adapted for an app built with AI tools, roughly in the order to add it. Ticking every item does not mean nothing will fail; it means you hear about it before your users have to tell you.

• An outside uptime check and error tracking are in place, with alerts to a phone.
• System health metrics are collected: CPU, memory and latency.
• Request rates and error rates are tracked per route.
• Every AI call is logged with model, version, tokens, latency, outcome and fallback.
• Fallback and empty-response rates are measured as a share of requests.
• External API calls and token usage are tracked, and cost per request is on the dashboard.
• Signs of changing inputs are recorded: output mix, confidence or validation failures over time.
• Alerts are set for the thresholds that need action, with sustained windows.
• Logs are centralised, searchable and free of secrets.

Questions