LiteLLM 1.107 RC: Token Count Sources and Alternate Counting Models

LiteLLM 1.107.0-rc.1, published October 18, labels where token counts come from and lets deployments select another model for counting. The release candidate helps agent developers distinguish provider results from local estimates, but introduces response fields and counting dependencies that need testing before an upgrade.

The release is a prerelease. Its token-count changes are separate from the stable 1.106 response-cache and WebSocket defaults. A useful evaluation should focus on whether the count is suitable for the application's decisions, including when to shorten a conversation or reject an oversized request.

A token count now says where it came from

The tagged response types define three source values: provider, token_count_model and local_estimate. Provider-shaped responses can expose these through litellm_count_source; an alternate-model count also identifies its model through litellm_count_model.

LiteLLM's own /utils/token_counter uses count_source, while its existing model_used field identifies the counting model. The new metadata is optional, so clients should also handle responses where it is absent.

This gives an application more useful context than a bare integer. A provider-generated count and a local estimate might lead to different safety margins near a context limit. The label describes how the number was obtained, however, and does not guarantee that it will match the usage ultimately billed for inference.

Counting can use a different model from inference

The settings implementation resolves deployment overrides ahead of shipped defaults. model_info.token_count_model selects a counting model and, optionally, a region. An empty object disables a shipped alternate-model setting.

This is a direct choice of counter. It does not first try the inference model and then work through a fallback chain, and it does not change the model that answers the user's request. A failure from the selected counting model can therefore become a failure of the count endpoint.

Another setting, model_info.token_count_framing, supplies a nonnegative per-request adjustment to local message estimates. Setting it to zero removes that addition. Raw-text prompt counts do not receive message framing.

For a gateway operator, these settings make counting an explicit part of deployment configuration. Keep the chosen model, region and adjustment visible alongside the inference route, so an unexplained count change can be traced to the configuration that produced it.

Regional permissions can affect a working deployment

The feature pull request documents shipped Bedrock Sonnet and Haiku 5.5 counting defaults that use Opus 5.5 count access in supported regions. Applicable US entries use us-east-1, while Australian entries use ap-southeast-4; geography rules and explicit overrides affect how those defaults apply.

An account able to run the original inference model may still lack permission or regional access for the counter. This is worth testing with the actual deployment credentials. A successful chat response does not establish that the separate count request can run.

Local fallback behavior also needs attention. Bedrock Kimi K3 can use a pinned tokenizer downloaded on first use, but offline or cache-missing operation can take an estimate path. Tool content and provider-specific transformations can further separate an estimate from billed input. The maintainers document these limitations alongside their tests; they are not universal accuracy guarantees.

Test response schemas and failure paths in staging

The RC backport calls out intentional contract changes. Clients that reject unknown JSON keys must accept the optional source fields, and malformed tools on count routes now return HTTP 400 rather than an internal-server failure.

Start with representative text requests, then add the tool and file inputs your application actually sends. Compare counts with observed provider usage and record the source label. Include denied access to the alternate model, an unavailable region and a missing tokenizer cache in the test cases.

Also check how the application reacts when no trustworthy count is available. It might keep a larger margin, defer the request or ask the user to reduce the input. That behavior belongs in the application's policy rather than being inferred from a successful HTTP response alone.

The practical value of this RC is making a previously hidden assumption inspectable. The next step is to verify the selected counter and its failure behavior for each deployment before relying on the number to manage an agent's context.

I'm Paras. I work at the intersection of AI, engineering and distribution.

I help AI infrastructure teams explain API changes, deployment tradeoffs and product launches through technical writing that developers can use.

Book a 30-minute call.