LiteLLM 1.106: Response Cache Limits and WebSocket Migration Checks

LiteLLM 1.106.0 changes two defaults worth checking before upgrading an AI gateway: which conversations can use its response cache, and how long a Responses API WebSocket connection stays open. Both can affect agent workflows that otherwise keep sending the same requests after an upgrade.

The stable release was published on October 18. LiteLLM provides a common interface to model providers through a Python SDK or a separately deployed proxy. For teams using it between applications and model APIs, these changes belong in the migration test plan. The underlying fixes merged earlier in October; their inclusion in this stable package is the new release event.

Response caching stops after four messages by default

The 1.106.0 cache implementation sets max_messages to four. When a request exceeds that count, LiteLLM skips both the cache lookup and storage of the resulting response. A request with exactly four counted messages remains eligible, subject to the other cache settings.

The count comes from entries in the request. A system, user, assistant or tool message in the messages array each contributes one. For the Responses API, LiteLLM counts conversation objects in the input list, excluding string entries and objects marked type: file. Four messages can represent fewer than four conversational turns, and their token length does not determine this limit.

The guard applies to exact and semantic response caching, across the Python and native Rust paths. Teams using exact caching for repeated longer conversations should therefore check their hit rates too. This setting governs reuse of a completed response through LiteLLM; a model provider's prompt-cache mechanism is a separate layer.

The cache fix explains the motivation: semantic caching could reuse an earlier answer when tool-result text was missing from the embedded prompt. The update includes that text, including block-form tool outputs. Short conversations remain eligible for semantic matches, so a permissive similarity threshold still deserves testing. Tool-call names and arguments are not fully represented in that embedded text.

For an existing deployment, the cost implication depends on what it cached before. Repeated requests above the limit may now reach a model provider, increasing upstream usage and latency. The change alone does not establish a universal spending increase. Operators can raise cache_params.max_messages or set it to null in proxy configuration; the Python Cache argument accepts None to disable the cap. Disabling it should be a deliberate decision about the workload's reuse behavior.

WebSockets get a total session lifetime

The Responses WebSocket change removes a 30-second deadline for receiving the first frame. A client that opens a connection in advance can now leave it waiting longer before sending its first response.create event.

In its place, the tagged endpoint code enforces a total connection lifetime measured from acceptance. The default is 3,600 seconds, or one hour. Operators can set general_settings.responses_websocket_session_limit_seconds between 60 and 7,200 seconds.

Traffic does not reset that clock. When the limit arrives, the proxy closes an active or idle connection with code 1000 and the reason Session duration limit reached. An application should recognize that closure and decide how to recover any unfinished work. Reopening a socket alone does not establish that an interrupted model response or tool operation has resumed safely.

What to check before rolling out

Start with requests representative of your application. Replay a short request and a longer conversation twice, then inspect whether each repeat reached the provider. Include a tool result that changes the answer the agent should give. Comparing those cases across the old and new versions will show whether the migration changes intended reuse or merely removes cache hits your application should never have trusted.

For WebSocket clients, use a short permitted session limit in staging. Test a delayed first request and a connection carrying work when the lifetime expires. Watch how the client reports unfinished output, and verify that any retry avoids repeating completed side effects.

LiteLLM's repository license makes the core code available under MIT terms, with the enterprise directory governed separately. Running the gateway still involves infrastructure and model-provider costs. For this upgrade, the useful acceptance criteria are predictable provider-call counts and recoverable session endings. Those checks connect the new defaults to the application's actual operating costs and reliability.

I'm Paras. I work at the intersection of AI, engineering and distribution.

I help AI teams turn technical changes and deployment decisions into clear stories for developers and enterprise buyers.

Book a 30-minute call.