PARAS MADAN
All writing

Latest in AI

Perplexity Orchestrator API: SDK Setup, Failover and Streaming Limits

Connect an existing SDK to Perplexity Orchestrator without confusing provider failover with model routing, or partial streams with completed answers.

Provider failover and model routing solve different problems. One keeps your chosen model available through another deployment. The other chooses a different model for the request. Treating them as interchangeable makes both debugging and cost attribution harder.

Perplexity Orchestrator supports both. For an existing application, I would migrate with an explicit model first, verify the response handling, and evaluate automatic model selection separately. That gives you one change to investigate at a time.

Configure the SDK URL correctly

The Orchestrator quickstart supports OpenAI Chat Completions, OpenAI Responses and Anthropic Messages formats. Use a Perplexity API key and the model's full catalog ID.

The base URL depends on the SDK:

  • OpenAI: https://api.perplexity.ai/orchestrator/v1
  • Anthropic: https://api.perplexity.ai/orchestrator

The Anthropic SDK adds /v1/messages itself. Copying the OpenAI base URL into that client would duplicate the version path.

For an initial server-side request with the OpenAI Python SDK installed and PERPLEXITY_API_KEY configured, the setup is:

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["PERPLEXITY_API_KEY"],
    base_url="https://api.perplexity.ai/orchestrator/v1",
)

response = client.chat.completions.create(
    model="perplexity/glm-5.3",
    max_tokens=2048,
    messages=[{"role": "user", "content": "Explain a cache miss briefly."}],
)
print(response.choices[0].message.content)

Before adding streaming or tools, confirm that your application can parse this basic response. Then introduce the features your actual workload needs rather than assuming format compatibility guarantees every provider-specific option.

Keep model choice separate from deployment choice

According to the routing guide, provider selection is automatic. When another deployment exists, provider errors, rate limits or timeouts can trigger a retry on that same model. You do not configure an individual provider.

Optional model selection uses perplexity/auto. That can change which model serves a request, so record the returned model field rather than inferring it from your preferred route.

My suggested evaluation has two stages. First compare the same model on representative prompts before and after migration. Then evaluate model routing against those prompts as a separate experiment. Otherwise, a quality change could be caused by the selected model rather than the new inference path.

Audit conversation state before migrating Responses

The Orchestrator FAQ says the API is stateless. Responses requests using store: true, previous_response_id or conversation return errors. Send the conversation explicitly instead.

I would inspect the existing client wrapper before changing its base URL. Look for hidden defaults, middleware that attaches state identifiers and code that assumes the server remembers earlier turns. A successful first message is not enough evidence that the second turn will work.

Use a two-turn test where the answer genuinely depends on the first turn. Keep the expected relationship simple enough to inspect without a subjective quality score.

Handle exhausted failover and partial streams

Invalid requests and overlong prompts return HTTP 400 without deployment retries. Exhausting the available deployments returns HTTP 429 with Retry-After; requests failing before output are not billed.

Once streaming output begins, a failure can arrive inside the stream. Delivered tokens are billed, and an unterminated stream should be treated as incomplete.

For a chat interface, I would preserve the partial answer with a clear failure state rather than quietly presenting it as complete. For a tool-using agent, do not execute a half-received tool call. Decide explicitly whether a retry replaces the partial generation or starts a new attempt, and keep that distinction in your logs.

Reconcile usage without double-counting reasoning

The FAQ documents per-token billing and no per-request fee. Responses report token usage, not a cost field. The served model's rates determine the bill.

In OpenAI-compatible formats, total input includes cached tokens and total output includes reasoning tokens. Do not add those subcategories to the totals again. Anthropic Messages uses separate input and cache-input counts, so a parser written for one format cannot simply be reused for the other.

My recommendation is to keep the original usage object alongside your normalized cost record. That makes an accounting discrepancy traceable without relying on a reconstructed summary.

Expand only after the failure cases work

Before increasing traffic, I would review a short checklist: correct SDK path, supported model ID, two-turn context, rejected input, incomplete stream and reconciled usage. Record failures alongside successful responses.

The useful migration is not the one with the fewest changed lines. It is the one whose behaviour remains understandable when the provider, stream or application does not cooperate.

I'm Paras. I work at the intersection of AI, engineering and distribution.

I help technical teams turn API capabilities and limitations into useful documentation and product storytelling.

Book a 30-minute call.