Reduce LLM costs by measuring the work, not guessing

A low price per token does not necessarily produce a low cost per completed task. A cheap request that fails repeatedly may cost more than a slightly more expensive one that resolves the user’s problem. Start by measuring the actual workflow: context sent, output generated, tool calls, retries and whether the task was completed usefully. Then optimize the part responsible for avoidable work.

Define a useful unit of cost

For a support assistant, the unit might be a correctly answered request with an appropriate source. For a document feature, it might be an accepted summary. Count failed attempts and retries as part of the same task, instead of reporting only the cost of the final successful call.

Keep provider token charges separate from retrieval, storage, hosting and human review. Local inference may avoid per-request provider billing but still uses hardware, electricity and operational time. The right comparison depends on the workload you actually need to run.

Write down a transparent estimate

Use the provider’s current rates and units for the exact model and request type. The example below is a generic arithmetic helper. Its rates are deliberately fictional placeholders in currency units per million tokens; they are not a quote for any provider or model.

The helper also excludes cache writes, cache reads, tools and other fees. Expand your accounting only when those costs apply. Record the rate source and date beside an estimate so someone can reproduce it when the pricing changes.

def estimate(input_tokens, output_tokens, input_rate, output_rate):
    """Rates are fictional currency units per million tokens."""
    return (input_tokens * input_rate + output_tokens * output_rate) / 1_000_000

# An arithmetic example, not provider pricing:
example = estimate(2000, 500, 1.0, 4.0)
assert abs(example - 0.004) < 1e-12
print(f"Illustrative token charge: {example:.3f}")

Remove context that does not help the task

Inspect real requests. A short classification task may not need the whole conversation and every retrieved document. Send the relevant policy passage, not an entire handbook by default. Keep essential facts in application state so summarizing a conversation does not lose the one identifier that matters.

Measure quality after each reduction. A smaller context that removes a crucial exception can make the system less useful. For retrieval, compare source coverage as well as token usage; the goal is focused evidence, not the shortest possible prompt.

Cache the right thing with the right scope

Some providers offer prompt caching for eligible repeated input. Eligibility, cache behavior and billing depend on the specific API. Read the current documentation rather than assuming repeated text is automatically free.

Application-level response caching is a separate decision. Only reuse a response when its user scope, source version and freshness requirements match. A cache key that ignores authorization can turn an optimization into a data leak. Start with stable public content or tightly scoped results you can invalidate deliberately.

Reference: Anthropic — Prompt caching ↗

Route simple work and stop needless loops

A deterministic calculation should usually stay a calculation. A simple lookup should not become several rounds of model reasoning. For model-based tasks, compare a smaller candidate against your existing quality baseline before routing traffic to it.

Bound retries and distinguish transient transport failures from invalid input. Repeating a request unchanged will not fix a missing order number. Record why a task retried and whether recovery helped. A latency or usage field from a local service can inform measurement, but it is not a monetary invoice.

Reference: Ollama — Chat response usage fields ↗