GenAI FinOps: How Token Pricing Really Works

For FinOps Practitioners, the advertised per-token price for GenAI is misleading; the real costs are in the details. The first Paper in this series introduced the “fuzzy math” of token pricing as a key difference between cloud and GenAI FinOps. This Paper explains the costs behind those seemingly simple price lists.

Judging GenAI cost by the price per million tokens (MTok) is like judging a car’s cost by the price of gas, ignoring the engine, the driving style, and maintenance. Cloud practitioners will recognize the pattern from storage IOPS pricing: simple on the surface, complex once peak loads and block sizes enter the picture. The total cost of ownership (TCO) of a GenAI application is driven by factors that rarely appear on a vendor’s pricing page.

This Paper covers how tokens are priced, the hidden cost of context window creep, the main optimization techniques, why the cheapest model is often not the most economical, and why the use case, not the token, is the unit FinOps should measure.

Learn better via podcast? Stay up to date on token economics with The Tokenomics Brief.

Not All Tokens Are Created Equal

GenAI models charge for processing tokens, which are small chunks of text. A token’s price is not a flat fee. It varies with what the token is and what it is doing.

• Input and output: Input tokens are the data you send to the model; output tokens are what the model generates. Generating a response takes far more computation than reading a prompt, so output tokens cost more, often three to five times more than input tokens. This matters most for use cases that produce long responses, such as summarization or drafting.
• Reasoning tokens: Models that “think” before responding generate internal reasoning tokens. Many providers bill these as output tokens even when the reasoning is not returned, so the same question can cost far more on a reasoning model, and the cost varies with how much reasoning effort is allowed.
• Modality: Text is the cheapest form of input. Interpreting an image can cost twice as much as the equivalent text, and some flagship models charge more than eight times as much for audio input as for text. Providers rarely bill vision as a separate unit. Instead, they convert an image into tokens using a provider-specific method that changes often, and the cost appears alongside text tokens.
• Model tier: A provider’s most capable model has the highest per-token price, and its smaller, faster models cost much less. Price generally tracks reasoning capability.

Prices and ratios change often. Confirm current provider rates before using any figure in a forecast.

The Biggest Hidden Cost: Context Window Creep

If you take only one thing away from this Paper, take this: the biggest hidden cost in most production AI applications is context window creep.

Most model APIs are stateless; the model has no memory of past interactions. To hold a coherent, multi-turn conversation, the application must resend the entire conversation history with every new message. Imagine having to repeat a conversation from the beginning every time you wanted to add a sentence. That is what happens inside the API calls.

Consider a simple customer service chatbot:

• Turn 1 (user): “What’s your return policy?” (5 tokens)
• Turn 2 (bot): “You can return items…” (50 tokens)
• Turn 3 (user): “What about international orders?” (6 tokens)

To process Turn 3, the application does not send only the 6 new tokens. It sends the whole history, 61 input tokens in total. Each turn adds to the input of every call that follows, so the cost per call rises as the conversation goes on, and the total cost of a session grows much faster than the number of turns. In a long conversation, the cost of resending context can far exceed the cost of the responses themselves.

Several factors make this worse:

• System prompts and guardrails: long system prompts and in-context instructions are sent as input on every call.
• Agentic systems: “conversations” between models and tools follow the same pattern, and an agent can make many calls per user request. The Tokenomics Foundation’s Big-T Notation gives teams a shared way to describe how consumption multiplies with model calls per request and agent depth.
• Images and other media: an image sent in the first turn becomes part of the history, so its processing cost is billed again on every later turn. Because vision costs are usually combined with text tokens, this recurring charge is easy to miss.

The result is a paradox. Output tokens cost more per token, yet in conversational and agentic applications the volume of input tokens usually dominates total spend. Understanding how an application builds its context is essential to predicting and managing its cost.

GenAI Cost Optimization Techniques

Several techniques reduce token spend without changing what the application does. They fall under the Usage Optimization Capability, and most require Engineering to implement them, so FinOps Practitioners add the most value by knowing when each applies and how to measure whether it is working.

Prompt Caching

Prompt caching stores a processed prompt prefix so later requests that start with the same content do not pay full price to process it again. Providers offer two forms:

• Implicit caching: the provider detects repeated content at the start of the context window and caches it automatically. It requires no setup, but it only helps when prompts begin with the same content.
• Explicit caching: the application marks content to cache. Providers usually charge a premium to write content into the cache and a steep discount to read it back during the cache’s lifetime, which differs by provider.

Explicit caching creates a break-even calculation. If cached content is not read back enough times before it expires, the write premium costs more than caching saves. Three rules decide whether caching pays off:

• It matches from the start. Reuse begins at the first token and stops at the first difference, so place stable content (system prompts, tools, reference documents) first and changing content last.
• It matches exactly. A timestamp, a reordered JSON key, or an edited tool description breaks the match.
• It belongs to one model and expires. Switching models, including through a router, starts a new cache, and entries age out after a short idle period.

To check whether caching is paying off, track two metrics per workload: the cache hit rate (the share of input served from cache) and cache cost efficiency (the share of input cost caching actually avoided, after the write premium). The Tokenomics Foundation’s Prompt Cache Explainer defines both and shows how prompt structure changes the result.

Semantic Caching

Semantic caching stores model responses and returns them for requests with the same meaning, even when the wording differs. It uses vector embeddings to find similar past requests, and a cache hit avoids calling the model at all.

It works best when many different inputs should lead to the same output, such as a support chatbot answering frequently asked questions. Savings can be large in these narrow conditions, but semantic caching adds its own infrastructure cost and the risk of returning a cached answer that does not fit, so it needs quality checks.

Batch Processing

For tasks that do not need an immediate answer, such as analyzing a set of documents, many providers discount asynchronous batch processing by 50 percent or more. Batch results are returned as files, typically within 24 hours, rather than as a stream.

Batching requires designing applications to separate real-time and batch workloads, and it must be offered by the provider. It is also a low-cost way to compare models or prompts on a sample of nonproduction work before choosing one.

Why the “Cheapest” Model Isn’t Always the Most Economical

It is tempting to default to the model with the lowest per-token price. A less capable model, however, may need longer prompts, more retries, or more refinement to produce a usable result, and it may generate verbose output. A more capable model may understand a short prompt, answer correctly the first time, and respond concisely. The total cost of that single successful transaction can be lower than the accumulated cost of coaxing a usable answer from a cheaper model.

Compare models on cost per successful outcome, not cost per token. That requires defining what counts as a successful outcome for each workload, a quality bar that Engineering and Product should set together. The Tokenomics Foundation’s Classifying and Measuring Realized Value from AI calls this the quality floor and explains how to apply it.

The Provider Matrix: Where You Buy From Matters

The platform you use to access a model can change its entire cost structure, not only its per-token price. Common channels include:

• Direct model providers: the companies that build the models, billed per token, often with the newest models first.
• Cloud providers: model services from Amazon Web Services, Microsoft Azure, and Google Cloud wrap model access in enterprise security, compliance, and existing cloud billing and commitments. This convenience is often worth the cost for large organizations.
• Inference hosts for open-weight models: many vendors serve the same open-weight models. Prices for the same model have historically varied by as much as 30 percent between vendors.
• Routers and aggregators: one interface and invoice in front of many providers, which simplifies access but can hide which models and providers drove the cost.

Some differences do not show up in the price at all. Providers can differ in inference speed, model quantization, maximum context window, and how much reasoning effort they allow. Two offers at the same price per token may not deliver the same result.

Cloud providers and some model providers also sell provisioned capacity, with pricing that differs significantly between them. See Navigating GenAI Capacity Options for how to evaluate it, and AI Tokenomics: Managing AI Value in SaaS Model Token Costs for how each channel affects allocation and governance.

The Real Unit of Measure: Use Cases

Identical prompts can produce different numbers of output tokens on different models, and even on the same model from one run to the next. Tokens are only a small piece of the picture. The way to understand the TCO of AI for a business outcome is to measure the whole use case.

A use case is the complete process required to accomplish a result. It can be an autonomous agent, a data processing pipeline, or any workflow that combines GenAI calls and tools. A typical use case uses several models together, along with cloud resources such as databases, storage, and orchestration to host the data and run the workflow.

This shifts FinOps from token cost to the Unit Economics of a use case. Because token counts vary with every run, and reasoning models vary further, the cost per unit is not a fixed price. It is a distribution: some transactions cost a fraction of a cent, others far more. Report it that way, for example as median and high-percentile cost per run, rather than as a single average.

The unit economics of a use case are also sensitive to change. Editing a prompt, swapping a model, or changing the data architecture can shift the whole distribution. Understanding TCO therefore requires ongoing coordination between Engineering and Finance, so technical changes can be tied to their effect on unit costs.

From Token-Counting to Tokenomics to True Cost Ownership

To manage GenAI spend, FinOps needs to move beyond counting tokens. The path to real TCO runs through input and output premiums, context window creep, caching economics, and the different cost models of each provider. A useful approach shifts focus from the token to the use case, measures the unit economics of the whole business outcome, and relies on close collaboration between Finance and Engineering to track volatile cost distributions.

That cost data is also how FinOps contributes to the larger question of what AI spend returns. A cost per successful outcome, measured at the level of a use case, is the cost side of a value claim. Pair it with an outcome the business owns, and the conversation moves from what AI costs to what it is worth.

To go deeper:

• Pricing in billing data: FOCUS v1.5, scheduled for ratification in December 2026, is set to label token charges as input or output and as cached or uncached, and to name the model behind each charge. Input modality and reasoning tokens are not in v1.5. See What FOCUS 1.5 Does for AI Cost.
• The broader economics of AI: start with the Tokenomics Foundation’s What Is AI Tokenomics?

The next Paper in this series, Navigating GenAI Capacity Options, covers provisioned capacity and how to evaluate it.

Acknowledgments

We’d like to thank the following people for their work on this Paper: