Rules vs machine learning vs LLMs: which does your problem need?
Use the simplest approach that could meet your most important requirement, and move up only when evidence shows it falls short. Rules and search fit clear, stable logic; classical machine learning fits structured data with labelled examples; an LLM fits open-ended language that rules and narrow classifiers cannot handle. In production the strongest design is often a hybrid that sends each request to the cheapest level that can handle it.
Start from the shape of the problem, not the model
The real decision is not which model to use but what level of intelligence the problem needs, and the answer can be none. Before choosing a technique, check the non-AI options: rules for a narrow policy decision, better search when people need to find things rather than generate them, human review, an existing tool, or a process change when the real gap is routing, ownership or missing documentation. Ask not whether AI could solve the problem, but whether it is the most sensible way under your constraints.
If it does need a model, three things narrow the options quickly.
- Task shape: what must the system produce? Labels, a ranking, a number, generated text, retrieved items, an anomaly flag or a transformation each rule out large parts of the design space.
- Data shape: is the input tabular, free text, images, audio, a time series or a graph of users and items, and how many trustworthy labels exist?
- Operating conditions: how fast must it answer, what can each prediction cost, how expensive is a wrong answer, must decisions be explained, and can the team run the infrastructure?
The complexity ladder: start low, climb with evidence
Order the options from cheapest to run to most capable, start at the bottom, and move up only when a simpler rung fails a requirement that matters. Each step up adds expressive power, and also operational cost, debugging difficulty, infrastructure and maintenance.
| Rung | Use it when | What it costs you | Example |
|---|---|---|---|
| Rules, search and heuristics | The logic is clear, the domain is narrow and stable, outputs must be deterministic, latency and cost must be minimal | Manual upkeep; coverage ends at the cases someone wrote down | Orders over a threshold go for approval; an exact FAQ match returns the documented answer |
| Classical ML | Structured, tabular data, a medium-sized labelled set, tight latency, decisions that must be explained | Feature engineering, feature pipelines and a retraining loop | Gradient boosting to predict churn |
| Ensembles and specialised models | The domain has mature methods: forecasting, recommendation, sequential decisions | Domain expertise and more moving parts | A feed ranked by collaborative filtering, content similarity, popularity and freshness |
| Deep learning | Unstructured input where learning the representation is the hard part, with enough data to train or fine-tune | Heavier training, GPU serving or aggressive optimisation, harder debugging and monitoring | Detecting abnormalities in medical images |
| LLMs and hybrids | Language-heavy, ambiguous or context-dependent tasks, or very few labels | Slower, often far more expensive per request, variable outputs, confident errors, governance and vendor dependency | Support requests phrased in many different ways |
When rules beat machine learning
Rules win when the decision logic can be written down and the domain does not move much. They are fast, cheap, deterministic and easy to audit, which matters for thresholds, policy checks, compliance gates and exact-match lookups.
Search deserves the same respect. Many teams build generation systems for problems that are really retrieval problems: if the job is helping someone find the right document, product or answer, better retrieval often creates more value than generated text. Heuristics such as recency, popularity, freshness and confidence thresholds sit between rules and learned models.
Rules also make the best baseline. Build them first even if you expect to replace them, because they show how much value a model must add before its extra cost is justified. Move up when the cases stop fitting explicit logic: when inputs vary in ways nobody can enumerate, or patterns change faster than anyone can maintain the rules.
When machine learning is worth it, and how much data you need
Classical machine learning is the default for structured business data. Customers, transactions and machines arrive as rows and columns, and models such as logistic regression, random forests and gradient boosting are fast to train, cheap to serve on a CPU and easier to explain than neural networks. Their weakness is that they are only as good as their features, and they struggle with raw images, audio or free text.
No universal number of labels makes machine learning worthwhile. What matters is whether you have enough representative labelled examples to beat your rules baseline and to prove that you have. Let the data you actually have pick the approach.
- Plenty of representative labels: supervised learning, with confident evaluation and fair comparison between approaches.
- Some labels: transfer learning, fine-tuning, weak supervision, retrieval or few-shot prompting.
- No labels: rules, search, anomaly detection, unsupervised methods or a zero-shot LLM workflow, while the live system starts collecting feedback you can label.
- Messy or unrepresentative data: fix the data first. That usually adds more than a better algorithm, and complex models amplify weak data while looking sophisticated.
When an LLM beats a trained classifier, and when it is overkill
An LLM earns its place when the input is open-ended language: users phrase the same need many ways, leave out details, refer to earlier conversations or ask several things at once, and no fixed label set captures it. It also helps when labels are scarce, because a prompt and a handful of examples can work long before you could train a classifier.
It is overkill when the task is narrow, deterministic, latency-sensitive or already covered by rules, search or a small model. LLMs are slower, often far more expensive per request, can give different outputs for the same input and can state wrong facts with confidence. Spam filtering, routing a few known intents and scoring transactions are classifier jobs.
Choosing an LLM also chooses an architecture: prompt versioning, retrieval, caching, guardrails, cost monitoring and dependence on a vendor. Work out the LLM cost per user before you commit, and plan for timeouts and provider outages with the reliability patterns for LLM APIs.
How rules, ML and LLMs compare on cost, latency and auditability
Accuracy, latency and cost rarely improve together. Better models tend to be slower and costlier, and fast systems need lighter models, caching or more compute. Decide which side of that triangle matters most before comparing options.
- Accuracy-critical (medical analysis, high-value fraud, safety and compliance checks): complex models, ensembles, staged human review and conservative thresholds can be justified, and auditability matters as much as accuracy.
- Latency-critical (search, autocomplete, payment-time fraud scoring, edge devices): lighter models, pre-computation, caching and tiered routing. A fast gradient boosting model can beat a stronger but slower network, and running vision on the device is often the answer; see edge vs cloud for computer vision.
- Cost-critical (moderation at scale, ad targeting, large batch jobs): classical ML, compression, aggressive caching, and good rules for the common cases with small models for the rest.
| Factor | Rules | Classical ML | LLM |
|---|---|---|---|
| Cost per prediction | Negligible | Low; usually runs on a CPU | Highest; billed per token or paid in GPU time |
| Latency | Fastest | Fast | Slowest; seconds for long outputs |
| Same input, same output | Yes | Yes, for a fixed model version | Not guaranteed |
| Auditability | The logic can be read directly | Explainable with effort, especially simpler models | Hard to explain why it answered as it did |
| Data needed to start | Domain knowledge | Labelled examples | Few or none |
| Typical change | Edit a rule | Retrain and re-evaluate | Change the prompt or model and re-evaluate |
Combine them: hybrid routing
Most production systems should not pick one approach. Route each request to the cheapest level that can handle it: rules for the obvious cases, retrieval for known answers, a lightweight model for high-volume classification, an LLM only where the task needs it, and a person for uncertain or risky cases.
A support assistant shows the pattern. Exact known questions are answered from the knowledge base, common intents are routed by a classifier, open-ended requests go to the LLM and edge cases escalate to a person. The result is usually faster, cheaper and safer than sending everything to the most expensive component.
The same idea works as a cascade: a fast first-pass model handles most requests, and only low-confidence or high-value cases reach a slower model. Hybrids also let you change one layer at a time, so treat the first model choice as a strong decision rather than a permanent one.
Four worked scenarios
The right answer comes from the problem's structure, not from model prestige. Four recurring cases show the method; Chapter 3 of From Model to Product works through them in full.
| Scenario | Shape of the problem | Usually the right approach |
|---|---|---|
| Spam detection | Classification at high volume; latency and cost matter; explanations help | Classical ML with well-chosen text features. An LLM is likely overkill. |
| Recommendations | Ranking rather than classification, with freshness and a fallback for thin data | A hybrid ranking pipeline: collaborative filtering, content signals, a popularity fallback and freshness controls |
| Medical image analysis | Unstructured images, subtle patterns, a high cost of error | Deep learning, with strong validation and human review |
| Support automation | A mixed workload of repeated questions, common intents and open-ended requests | A hybrid: retrieval and routing for common cases, an LLM for the rest, escalation for edge cases |
This pattern works because different parts of a workflow often have different needs. The main mistake is assuming the whole system must operate at the complexity level of its hardest case.
Questions
Is machine learning better than a rule-based system?
Not in general. Rules are faster, cheaper, deterministic and easier to audit, and they win when the logic is clear and the domain is stable. Machine learning wins when the patterns are too varied or too subtle to write down and you have representative labelled data. Many production systems use both, with rules handling the obvious cases.
Can an LLM replace a trained classifier?
It can, especially when labels are scarce or the categories are fuzzy, and prompting an LLM is a quick way to get a first working version. For high-volume, narrow classification a trained classifier is usually faster, cheaper and more predictable. A common path is to start with the LLM, collect and review real examples, and train a smaller model once volume justifies it.
What is a good baseline before training a model?
The simplest thing that could work: hand-written rules, keyword or exact-match lookup, a popularity ranking, or a logistic regression on a few obvious features. A baseline shows how much value a more complex model must add before its cost is justified. If the baseline is close to good enough, you may not need the model.
Is deep learning necessary for tabular data?
Usually not. Tree-based methods such as gradient boosting remain strong defaults for structured business data, and they are cheaper to train, serve and explain. Deep learning earns its cost when the hard part is learning a representation from raw images, audio, text or complex sequences.
How do I know if I need machine learning at all?
Check the non-AI options first: rules, better search, human review, an existing tool or a process change. You need machine learning when the decision depends on patterns nobody can write down and you have, or can collect, data that represents them. Skipping this question is one of the ways AI projects stall; see why AI projects fail to reach production.
Can I switch from an LLM to a cheaper model later?
Yes, if you design for it. Keep the model behind a stable interface, log inputs and outputs, and keep a rules layer in front, so you can swap a component when volume or evidence changes. Hybrid systems make this easier because each layer can be replaced without rebuilding the rest.