Baseten Mixed-Mode Rolling Deployments: GPU Capacity and Availability Tradeoffs

Baseten announced mixed-mode rolling promotions on October 30, allowing inference teams to set maximum surge and maximum unavailable capacity together during a rollout. The feature adds flexibility when spare GPUs are limited. The documentation labels mixed mode a beta for selected organizations, not a generally available switch.

What changed in Baseten rolling deployments

The release note introduces a combined capacity policy for moving an environment to a new deployment. Instead of choosing only extra replicas or only reduced availability, a team can allow some of each.

A replica is a running copy of the model server. During a rollout, the old version still needs capacity to serve requests while copies of the new version start. The question is how much temporary overhead or temporary reduction the operator can tolerate.

The rolling-deployment documentation distinguishes the modes:

Mode	Capacity policy
Max surge	Add replacements before retiring previous replicas
Max unavailable	Allow previous replicas to retire before replacements are ready
Mixed beta	Permit both extra replicas and a temporary serving-capacity reduction

The budgets are percentages of a baseline replica count tied to current demand and the environment's configured limits. Setting both max_surge_percent and max_unavailable_percent above zero selects mixed mode where enabled.

Why allowing a small capacity reduction can help

Keeping every old replica alive until its replacement is ready is attractive, but it needs somewhere to start the replacements. If GPU capacity is tight, that requirement can become a constraint on the rollout itself.

Allowing some replicas to retire first changes that tradeoff. It gives the operator another way to make room, while accepting that less serving capacity may be available during the transition. A capacity budget is an operational choice, not a promise of unchanged latency.

The useful setting therefore depends on traffic headroom. A service with ample spare throughput can tolerate a different rollout policy from one already operating near its limits. The same percentage should not be copied between them just because both serve the same model.

Availability and configuration limits

Baseten says mixed mode is available to selected organizations by request. Rolling deployments apply to individual model deployments, not Chains.

The documentation also warns that omitted settings are preserved. When changing modes, explicitly set both budgets, including setting the unwanted budget to zero. Otherwise, a configuration intended to replace one policy can combine with the previous value instead.

That detail matters for automation. A deployment script should describe the complete intended policy, and a review should inspect the resulting settings rather than only the fields in the latest request.

What to measure during the rollout

For teams evaluating mixed mode, the useful comparison is rollout completion time alongside request latency, errors and available capacity. Finishing a release sooner is not necessarily an improvement if users experience a larger service disruption.

The feature makes the compromise more explicit. Teams can now choose how much temporary capacity to buy and how much temporary reduction to accept, then check whether production behavior matches that choice.

I'm Paras. I work at the intersection of AI, engineering and distribution.

I help technical teams turn infrastructure changes and AI product launches into clear stories developers can understand and use.

Book a 30-minute call.