Short answer
Inference cost is what a business pays each time a model produces an output, billed by most providers per token of input and output, where a token is roughly three-quarters of a word. The figure that matters is not the price per million tokens but the cost per completed task, multiplied by expected volume.
Also called: Token cost, Cost per request, AI running cost
Per-task cost is where estimates go wrong, because one user question rarely means one model call. A retrieval system pays for the passages it sends as context. An agent pays for every step in its loop, and for the conversation history it resends each turn. A retry after failed validation doubles a step. Systems that looked inexpensive in a demo become surprising in production mainly through these multipliers rather than through the headline rate.
The levers are mostly architectural. Route easy cases to a smaller model and reserve the expensive one for hard ones. Cache answers to questions that repeat, which in customer support is most of them. Trim context so the model is not re-reading a document it does not need. Handle deterministic steps in code. Published prices for a given level of capability have also fallen repeatedly, so an idea that is marginal on today’s arithmetic is worth recording rather than abandoning.
Common questions
How much does it cost to run an AI feature?
It depends on model choice, how much context each request carries and how many requests you expect, so the only honest estimate comes from pricing a realistic sample. Take fifty typical requests, measure the tokens actually consumed end to end, and multiply by projected volume. Doing that before the build usually changes a design decision or two.
Where this comes up in our work
Related terms
Large Language Model (LLM)
A large language model is a neural network trained on large volumes of text to predict likely continuations, which lets it write, summarise, classify and translate without being programmed for each task.
AI Agent
An AI agent is a system that uses a language model to decide which actions to take, then takes them through connected tools: reading a database, sending an email, updating a booking record.
Retrieval-Augmented Generation (RAG)
Retrieval-augmented generation is a technique that searches a private collection of documents for passages relevant to a question, then passes those passages to a language model as context for its answer.
Workflow Automation
Workflow automation is software that runs a defined sequence of business steps without a person moving data between systems: a submitted form creates a record, notifies the right team, files the attachment and schedules a follow-up.
Reading definitions because you are scoping a project? Skip ahead and just ask.