Should your computer vision model run on the edge or in the cloud?

Why the first pass usually belongs on the device

In the book's worked product, a phone camera app that tells people how to dispose of a household item, the first classification runs in the browser on the user's phone. That choice follows from the requirements, not from novelty. Sending every image to a server makes every prediction depend on network latency and backend availability, makes cloud cost grow with each new user, and sends photos of people's homes to a server they may not want to trust.

Running locally answers all of that at once: no round trip for the basic prediction, less backend load because only uncertain cases need more processing, lower cost because the expensive model runs less often, and better privacy because most images never leave the device.

It also brings new limits. Phones have far less CPU and memory than servers, browsers are a restricted runtime, the gap between a new flagship and an old budget handset is large, and battery drain matters if the model runs often. Those limits decide the model, the label set and the routing, which is what the rest of this guide covers.

The decision table

Read your product against each row. If most rows point to the edge and a small model meets your accuracy bar, start on the device and keep a cloud fallback. On-device inference does not make cost disappear; it moves it into engineering time, testing across devices and the user's battery. What it changes is the shape: cost grows more slowly as users arrive, because the server no longer pays for every image.

Let the data and the device choose the model

Production vision systems rarely fail because the architecture was wrong. They fail because the training data did not look like the photos real users take in kitchens, garages and gardens. So design from the data outward: the dataset determines the label set, the label set determines how complex the model must be, and that complexity decides whether edge inference is possible at all.

Two consequences follow. Keep the label set as broad as the data supports: the worked example uses eleven broad classes, including unknown, rather than distinctions such as bottle versus container that the available datasets cannot support reliably. And start with the simplest task that solves version one. Users photograph one item at a time, so image classification is enough; detection and segmentation need heavier models and labels that are much harder to obtain.

Then look for the smallest model with acceptable accuracy on real images, not the most accurate model available. The worked example uses MobileNetV2 with transfer learning from ImageNet weights: its depthwise separable convolutions cut the parameters and operations in each layer, so it loads quickly and runs in a browser through JavaScript or WebAssembly. Choose the deployment target first, because training a large network and compressing it later often gives poor results.

Shrinking and shipping the model

Export the trained network to a runtime that needs neither Python nor the training framework, and shrink it during export. Quantisation, which stores weights as 8-bit integers instead of 32-bit floats, makes them a quarter of the size and usually speeds up inference for a small loss of accuracy. Measure that loss per class on your validation set after conversion, not only overall. Pruning, which removes weights that contribute little, and distillation, which trains a small model to imitate a large one, are the next tools when quantisation is not enough.

Test on the slowest device you intend to support, and measure battery as well as speed. In the author's own production work, optimising on-device inference for a mobile web app cut battery drain on low-end devices by about half, and that constraint decided the inference architecture.

The common runtimes, by target:

• Browser: ONNX Runtime Web or TensorFlow.js, on WebAssembly, WebGL or WebGPU depending on the device.
• Android: LiteRT (formerly TensorFlow Lite) or ONNX Runtime, using the GPU or NPU where a delegate supports it.
• iOS: Core ML, which can run models on the Neural Engine.
• NVIDIA Jetson modules: TensorRT, for GPU-class models at the edge.
• Raspberry Pi-class boards: an add-on accelerator, such as a Hailo or Coral module, when the CPU alone is too slow.

Do not force a guess

A classifier returns a probability for every label, and the highest one is not a promise. Two mechanisms let the model admit doubt. First, an explicit unknown class gives objects outside the training set, and blurred or badly lit photos, somewhere to go. Second, a confidence threshold: if the top probability falls below it, the result is treated as unknown and escalated. The book starts at 0.6; tune yours on held-out data by weighing wrong answers against the cost of sending more images to the cloud.

Before trusting either, read a confusion matrix built on held-out images, which shows which classes the model mixes up. If two materials are often confused, collect more images of both; if unknown objects are forced into real classes, revisit the label set or the threshold. When you build it, evaluate on an unshuffled set so that predictions and true labels stay in the same order.

The hybrid design: edge first, cloud when unsure

The edge and the cloud are not rivals. A hybrid design runs the small model on every image and sends only uncertain ones to a larger fallback vision model on the server, which can be more accurate because it does not have to fit on a phone. Only a small fraction of images reach the expensive model, which keeps cloud cost down as usage grows; the server side can be an ordinary API, as in deploying a model with FastAPI and Docker.

Keep the routing rules few and predictable, because clever heuristics are hard to debug. Send the label and confidence to the backend, not the image, and keep the decision about what to do with a label in backend rules. Recognising an object is not the same as deciding what to do with it, and separating the two lets one model serve a new region or policy by changing rules instead of retraining. The routing runs in six steps:

1. Capture the image.
2. Run the edge classifier.
3. If confidence is high, accept the result.
4. If confidence is low, send the image to the fallback model.
5. Use the fallback result if it is available.
6. If the system is still uncertain, ask the user for clarification.

Updating and monitoring models on devices

Shipping a model to devices puts its deployment inside your release process. In a web app the model file ships with the app, so give it a versioned file name and the next deploy updates every user. In a native app, download versioned model files at runtime instead of waiting for an app store release, and roll new versions out in stages. Record the model version with every prediction so you can compare versions and roll back.

Because images stay on the device, monitoring relies on what the device sends back: label, confidence and model version. That is enough to watch the confidence distribution and the unknown rate; the book's example alert fires when unknown predictions exceed 20%. A rising unknown rate often means the world has changed, such as new products the training data never saw, and it raises cloud cost too, because more images escalate. Data drift vs concept drift explains how to tell what changed, and LLM cost per user shows why escalation rate is the number to watch on the bill.

Questions