What does a CTO do at an AI startup?
Short answer
A CTO at an AI startup does the usual CTO job, owning architecture, the engineering team and technical risk, plus decisions most software CTOs never face: whether to build, buy or host models, what data the company may use, how quality is evaluated, what each user costs in model calls, and how the product fails safely. The role is judged by whether the AI works dependably for customers at a cost the business can carry.
By Dr. Dulan Dias, co-founder and CTO of Tellme AI
Key facts
- NIST released its AI Risk Management Framework (AI RMF 1.0) on 26 January 2023 for voluntary use; it organises AI risk work into four functions: Govern, Map, Measure and Manage.
- The OWASP Top 10 for LLM Applications 2025 ranks prompt injection as the first risk (LLM01) and lists unbounded consumption (LLM10), which covers denial of service and attacks that run up pay-per-use costs.
- As of November 2026, Anthropic commits to at least 60 days' notice before retiring a publicly released Claude model, and requests to a retired model fail.
- As of November 2026, OpenAI's deprecation policy gives at least six months' notice before retiring a generally available model, and may give much shorter notice, such as two weeks, for preview models.
- Stanford's AI Index 2025 reported that the inference cost of a system performing at the level of GPT-3.5 fell more than 280-fold between November 2022 and October 2024.
- The ICO lists artificial intelligence, machine learning and deep learning as examples of innovative technology, one of the factors that can make a data protection impact assessment (DPIA) necessary under UK GDPR.
- The UK's AI Cyber Security Code of Practice, published on 31 January 2025, sets baseline security principles for AI systems and is the basis for the ETSI standard TS 104 223.
- Dr. Dulan Dias became Chief Technology Officer of Tellme AI in December 2025, after building its production MVP end to end as founding engineer within 4 to 6 months.
Definitions
- Build, buy or host
- The three ways an AI startup gets a model: train or fine-tune its own, call a provider's model through an API, or run an open-weight model on infrastructure it controls.
- Evaluation set
- A fixed collection of real inputs with agreed good outputs or scoring rules, rerun whenever a model, prompt, retrieval step or dataset changes, so quality is measured rather than assumed.
- Model cost per user
- The model and inference spend attributable to one active user in a month: calls per action, tokens per call and price, adjusted for retries and cache hits.
- Model deprecation
- A provider's notice that a model will be retired on a set date, after which requests to it fail and the product must already have moved to a replacement.
How is the CTO role different at an AI startup?
The foundations are the same as at any software company: architecture, hiring, security, delivery and technical risk. The difference is a core component that is probabilistic, priced per call, supplied by a fast-moving vendor and able to get worse without a code change.
Chapter 14 of From Model to Product, Observability, Monitoring & Model Health, makes the point that AI systems fail quietly: servers stay healthy while confidence drops, unknown predictions rise and external model calls multiply. An AI CTO has to build the instruments that catch this, because uptime dashboards will not.
| Concern | Typical SaaS CTO | AI startup CTO |
|---|---|---|
| Core behaviour | Deterministic code that tests can pin down | Probabilistic outputs that need evaluation on representative data |
| What done means | Tests pass and the feature ships | Quality clears agreed thresholds on an evaluation set, and keeps clearing them |
| Cost | Grows with servers and storage | Grows with model calls per user action, tokens and retries |
| Dependencies | Cloud services and libraries | Model providers that change and retire models on their own schedule |
| Data | Store and protect it | Also prove the right to train on it, evaluate with it and keep it |
| Failure | Mostly loud: errors and outages | Often quiet: plausible answers that are wrong |
Build, buy or host: choosing models and managing vendor risk
Most AI startups should start by buying: call a strong model through an API, learn what users need and measure. Host an open-weight model when data control, cost at steady volume, latency or offline use demands it. Train or fine-tune when prompting and retrieval cannot reach the quality you need and you have the data to do better. The trade-offs are in open-weight vs proprietary LLMs and RAG vs fine-tuning.
Whatever the choice, the CTO's job is to keep it reversible, because models get retired. As of November 2026, Anthropic promises at least 60 days' notice before retiring a publicly released model, and OpenAI at least six months for a generally available one. Route model calls through one internal interface, keep provider-specific features out of the core unless they earn their place, and keep an evaluation set that can qualify a replacement in days.
Chapter 5 of the book, Model Architecture Patterns, treats the API versus local model question as an architectural decision rather than a vendor choice, because deployment constraints decide which designs are feasible at all.
Data strategy: what to settle before the first customer
Data decisions are hard to reverse, so settle these early and write them down.
- Rights. For every dataset, record where it came from, the terms that allow its use, and which uses those terms cover: training, evaluation or only serving.
- Customer data. Decide whether customer inputs may improve the product, say so in the contract, and make it possible to honour a customer who says no.
- Personal data. The ICO lists AI and machine learning among the innovative technologies that can make a data protection impact assessment necessary, so expect to write one. The legal picture is in UK AI regulation for startups.
- Feedback. Capture corrections, overrides and escalations in a form that can become evaluation cases and, later, training data.
- Retention and location. Decide what is logged, for how long and in which region, before a customer's security questionnaire decides it for you.
Evaluation: how the CTO knows the AI works
An AI CTO defines quality before the team builds, not after. That means an evaluation set drawn from real inputs, scoring rules the domain experts agree with, and a regression run on every change to a model, prompt, retrieval step or dataset. In production, add the signals users send without being asked: corrections, retries, abandoned sessions and escalations to a person.
Treat a provider's model upgrade like a code change and run the evaluation set first. The methods, from golden sets to LLM-as-judge and its caveats, are in how to evaluate an LLM application.
Model cost per user and unit economics
The AI CTO owns the number that decides whether the business model works: model cost per user per month. It multiplies calls per user action, tokens per call and price, adjusted for retries and caching, and it moves whenever usage patterns change. The formula and a worked example are in the LLM cost per user guide.
Prices have fallen fast: Stanford's AI Index 2025 found that the inference cost of a system performing at GPT-3.5 level fell more than 280-fold between November 2022 and October 2024. Design so falling prices help but never depend on them. Cap calls per request and per user, cache, and route easy requests to smaller models. OWASP lists unbounded consumption among the top ten risks for LLM applications, which makes runaway spend a security problem as well as a finance one.
Safety, security and governance
At a startup the CTO owns AI safety in practice, whoever owns it on paper. Start with the threats specific to language models: the OWASP Top 10 for LLM Applications 2025 puts prompt injection first, followed by sensitive information disclosure. Then make the work systematic. NIST's AI Risk Management Framework organises it into Govern, Map, Measure and Manage, and the UK's AI Cyber Security Code of Practice, published in January 2025, sets baseline security principles.
Match the controls to the risk: a product that suggests is not governed like one that decides on money, health or access to services. For every model in production, know which version is running, what data trained it, when it was deployed and how to roll it back.
Hiring the first AI engineers
The first AI hires decide how fast the company learns, and the CTO usually makes them while still writing code. Hire engineers who can ship an AI feature end to end before hiring researchers, and bring in machine learning specialists once there is data and a measured quality gap that prompting and retrieval cannot close. The order and the roles are in how to build an AI team.
At an early AI startup the CTO is often also the first builder. Dr. Dulan Dias built Tellme AI's production MVP end to end as founding engineer within 4 to 6 months, covering the Next.js frontend, the FastAPI and MongoDB backend, the admin portal, CI/CD and the AWS infrastructure, before becoming its Chief Technology Officer in December 2025.
Frequently asked questions
Does an AI startup CTO need a PhD in machine learning?
An AI startup CTO does not need a PhD. The job is mostly building dependable systems around models: architecture, data, evaluation, cost and security. A research background helps when the product's advantage is a novel model, such as medical imaging or forecasting on proprietary data. For products built on foundation models, production engineering experience and sound judgement about trade-offs usually matter more than research credentials.
Should an AI startup train its own model?
Most AI startups should not train a model from scratch. Start with a provider's API or an open-weight model, add retrieval for your own knowledge, and fine-tune only when evaluation shows a gap that prompting and retrieval cannot close. Training your own model makes sense when proprietary data is the advantage and no general model performs the task well, as in some vision, forecasting and scoring products.
What should an AI startup CTO report to the board?
An AI startup CTO should report a few numbers that tie the technology to the business: quality on the evaluation set over time, model cost per active user against target, incidents and their causes, dependence on each model provider, and hiring against plan. Add any change in data rights or regulation that affects the roadmap. Boards need trends and decisions, not architecture diagrams.
What is the biggest technical risk at an AI startup?
The biggest technical risk at most AI startups is a product that demos well but fails quietly on real inputs, so customers lose trust before the team notices. Close behind are unit economics that collapse as usage grows and a single model provider whose changes the company cannot absorb. The same habits reduce all three: an evaluation set, per-user cost tracking and a reversible model layer.
How should a startup prepare for a model provider retiring its model?
Prepare before the announcement. Keep model calls behind one internal interface, record which model version serves each feature, and keep an evaluation set that can score a replacement quickly. Watch each provider's deprecation page: as of November 2026, Anthropic commits to at least 60 days' notice for publicly released models and OpenAI to at least six months for generally available ones.
Who owns AI safety at a small startup?
At a small startup the CTO owns AI safety in practice, with the CEO accountable for the risks the company chooses to accept. Name one person responsible for the model inventory, evaluation results, incident log and security testing, even if that person is the CTO. As the company grows, bring in specialist security and compliance help, but keep a single named owner for each model in production.
How much should an AI startup CTO still code?
An AI startup CTO should code enough to keep their judgement sharp and no more than the team's needs allow. Before the first hires, that is most of the week. Once a team forms, the CTO's leverage moves to architecture, evaluation standards, hiring and unblocking others, and their own code should stay off the critical path so it never holds up a release.
Sources
About the author
Dr. Dulan Dias is co-founder and Chief Technology Officer of Tellme AI in Belfast; founder of EchonLabs, grown from 2017 to more than 60 people before its acquisition in 2026; holder of a PhD in AI and data science from Queen’s University Belfast; and the author of From Model to Product.