Latest in AI
Claude Replies Stop at 4096 Tokens in LiteLLM: Check the Vertex Model ID
LiteLLM 1.108.10 fixes default output limits for versioned Vertex Claude IDs. Verify the outbound cap and provider route before increasing it.
A Claude response that ends mid-sentence through a gateway is not necessarily a broken stream. If the completion repeatedly stops at 4,096 tokens, check the output limit sent upstream before changing timeouts or replacing the model.
LiteLLM 1.108.10, released November 16, includes an LTS fix for default token limits on versioned Vertex AI Claude identifiers. The issue affects requests that omit max_tokens; it is not evidence that Claude itself has a universal 4,096-token output limit.
This is a source-based troubleshooting guide. The checks below are proposed for your staging deployment, not tests I ran against a model account.
Establish whether the answer hit a limit
Keep one harmless request that shows the problem. Record the gateway version, model alias, resolved provider model, requested output limit, returned completion-token count and finish reason. Preserve diagnostic fields without copying private prompt content or credentials into a ticket.
The LTS backport reports a versioned Vertex Haiku request ending with finish_reason=length at 4,096 tokens before the fix. Its comparison completed a longer response afterward. These are the contributor's observations, not a promise that every truncated answer has this cause.
If your application already sends an explicit 4,096-token cap, the gateway may simply be honoring it. Also check available context: finish_reason=length can indicate a context limit, not only the requested output cap. If the stream terminates without a final response, or the request fails with a transport error, investigate that evidence separately. Do not make a token-limit diagnosis from the last visible sentence alone.
Check the resolved model, not just its alias
A friendly application name can hide an identifier such as vertex_ai/claude-haiku-4-5@20251001. The suffix matters when the gateway looks up model capabilities.
The tagged 1.108.10 implementation checks the model's own mapping first, then the unversioned name for an identifier containing @, then its provider-qualified mapping. If none provides a value, it retains the configured fallback. The change does not make an unknown model's limit knowable.
Inspect the effective outbound max_tokens in an authorized staging diagnostic, alongside the resolved provider route. An alias in a client log is not enough to establish which mapping supplied the value. Also record relevant environment overrides so the comparison can be repeated after a restart.
Choose an explicit cap for the workload
LiteLLM's input-parameter documentation shows max_tokens as a request parameter and recommends checking supported parameters for the exact model and provider. The tagged transformation fills missing options rather than replacing an already supplied value.
For a short document summary, use a deliberate output budget instead of inheriting the largest available default. For a longer report, select a cap supported by the actual provider model and validate that the result is complete. Raising a ceiling permits more output and potentially more cost; it does not instruct the model to produce a useful answer of that length.
Keep the first comparison controlled: same deployment route, same synthetic prompt and same explicit cap before and after the candidate upgrade. Then test the omitted-cap case separately. Otherwise, an application change and a runtime change can disguise each other's effects.
Do not apply the Vertex result to every Claude route
The backport explicitly leaves versioned Bedrock Invoke identifiers at their existing 4,096 default on this LTS line. It also describes changes for some Azure AI routes. Those details rule out a blanket instruction to upgrade and assume every hosted Claude configuration now behaves identically.
For the route you actually use, verify both a short reply and a task that previously hit the boundary. Capture the final status and usage, inspect whether the answer meets the task's completion criteria, and retain the old deployment artifact until the comparison passes.
A larger default is not the same as a completed task. Make the requested cap, the provider's accepted cap and the final response visible in your upgrade review. That is more useful than repeatedly asking the model to continue without knowing why it stopped.
I'm Paras. I work at the intersection of AI, engineering and distribution.
I help infrastructure teams turn compatibility fixes and operational details into practical technical writing.