PARAS MADAN
All writing

Latest in AI

PolyAI Dialog-RSN-1 on SageMaker: What an OpenAI Realtime Migration Still Needs

Dialog-RSN-1 supports Realtime clients through a bridge, but you still need speech synthesis, compatible audio and a plan for session limits.

An audio-native model is not necessarily a complete speech-to-speech system. That distinction matters when moving a voice agent to PolyAI Dialog-RSN-1 on SageMaker: the model listens to audio, but your application still needs to turn its reply into speech.

AWS announced the Marketplace and JumpStart deployment option on November 19. It gives teams a SageMaker endpoint in their own AWS environment. It does not take ownership of their telephony integration, orchestration or business logic.

For an existing Realtime application, the useful question is which parts of the call path can stay unchanged and which need a deliberate replacement.

Separate the model from the voice application

The official PolyAI sample repository explains the output boundary: Dialog-RSN-1 returns text or a function call, not synthesized audio. Asking for audio output is rejected. A separate text-to-speech engine is required.

Draw the proposed call path before changing the client:

Caller audio -> audio conversion -> Realtime client
             -> bridge -> SageMaker model
Model reply text -> speech synthesizer -> caller
Model tool request -> application validation -> business system

Give each arrow an owner. In particular, decide where playback stops when the caller interrupts, where tool arguments are checked and which component records the final business outcome.

This prevents a misleading demo in which the model produces a sensible answer but the caller hears nothing, or hears an old answer continue after interrupting it.

The bridge is a transport adapter

PolyAI's sample bridge connects a local WebSocket to SageMaker's SigV4-signed HTTP/2 bidirectional stream. Credentials come from the standard AWS credential chain. SageMaker does not supply the WebSocket URL a Realtime client expects.

Treat that adapter as a distinct deployment component, not as a reason to assume every existing client feature works identically. Keep the first integration small: establish a connection, send a short recording, capture the reply text and close cleanly.

For a production design, decide where the bridge runs and how the client reaches it. A local development listener is not a finished multi-user service. Review connection ownership, access controls and failure handling before exposing the route to another application or device.

Check the audio before judging the model

The AWS guide specifies PCM16 mono input at 16 or 24 kHz. G.711 and Opus need conversion first. It also describes English-only support and connections lasting up to 30 minutes.

Start with a small set of permitted recordings that represent the input your application actually receives. Include a quiet call, background noise, a brief interruption and a caller spelling a reference number. Keep an untouched copy of each recording so you can distinguish a conversion problem from a model response problem.

Record the source codec, converted sample rate and channel count alongside each result. A transcript that drops digits is worth investigating at both stages. Do not compensate for poor input handling by making the prompt longer.

After the recorded examples behave sensibly, repeat the same scenarios through the live audio path. That comparison is more informative than testing unrelated questions in each environment.

Design for interrupted and long calls

The repository documents a 32,000-token context window, automatic conversion of older audio into its transcript, a five-minute idle timeout and the need to reconnect before the session cap. Reconnection requires replaying relevant context.

Build an application-owned handoff record for that transition. For a delivery-support assistant, it might contain the verified order reference, the caller's stated issue, completed lookups and the action still awaiting confirmation. Keep completed actions separate from proposed ones.

Then rehearse a reconnect between a lookup and the next question. The acceptance condition is not merely that the socket opens again. The assistant should continue the unresolved conversation without repeating an action already completed or treating an unconfirmed proposal as approved.

Use the same scenario to review interruption behavior: the next answer should respond to the caller's correction, not finish the abandoned response.

Count endpoint time and close the trial

PolyAI's README warns that the endpoint bills while it is InService, including when no client is connected. Stopping a notebook halfway through is therefore not a cleanup plan.

Assign someone to own the trial endpoint and its deletion. Record when it was created, which recordings were evaluated and when cleanup was confirmed. Keep the test window bounded rather than leaving a successful demo running until the next meeting.

Before moving live traffic, require separate sign-offs for audio handling, reply playback, reconnect behavior and tool execution. A compatible protocol reduces integration work, but the migration is complete only when the whole call behaves correctly.

I'm Paras. I work at the intersection of AI, engineering and distribution.

I help AI teams explain technical products through clear documentation, practical guides and developer-focused content.

Book a 30-minute call.