# Offline AI Evals vs A/B Tests: When Can You Ship on Evals Alone?

A practical release gate for AI teams that want to move quickly without mistaking higher benchmark scores for better user outcomes.

By Paras Madan | 2026-11-21


An offline AI eval can tell you that a new prompt answers your test cases better. It cannot, on its own, tell you that customers will resolve more problems. In a [November 19 engineering write-up](https://engineering.atspotify.com/2026/11/when-can-you-ship-on-offline-evals-alone), Spotify explains what would have to be true before an AI team could use the first result to skip measuring the second.

## The relationship to measure

Say you run an AI support assistant. Your offline eval checks whether answers cite the right policy and suggest the right next step. Your live metric is the share of tickets resolved without reopening within seven days. Those are related outcomes, but an answer can satisfy the rubric and still give the customer more work than they are willing to do.

What matters for release decisions is whether **changes that improve the eval also improve the live outcome**. A high agreement rate between an AI judge and a human reviewer on individual answers is not evidence of that relationship. The [underlying paper](https://arxiv.org/abs/2610.10142) calls this trial-level surrogacy. The unit you care about is the change you might ship, such as a prompt revision, not the individual answer you scored.

## Build a log of changes that ran both tests

Start by choosing one product surface, one class of changes, and one live outcome. For the support assistant, that could mean refund-policy prompt revisions and seven-day durable resolution. For each change, record what happened to the offline score and what happened in the A/B test. Store the uncertainty around both estimates, the test population, and the rubric or judge version. Include experiments that lost or were abandoned after testing, not only the launches that worked.

Do not put prompt edits, new retrieval systems, and full model replacements into one bucket simply to get more rows. For example, your offline score might reliably predict which refund prompts improve resolution, while a model swap changes response time enough to affect abandonment on its own. A strong average across those experiments would not tell you what happens for the next model swap.

## Why a simple correlation can mislead

Small eval sets and short online tests make both sides of the comparison noisy. Spotify's worked example gets a squared correlation of 24% with a naive estimate, against a known 60% in the example. That is an illustration of measurement error, not a finding that Spotify's production evals predict 60% of online results. The paper also applies its gate to 183 interventions from a separate Upworthy dataset, where the relationship is weak enough that the illustrative rule sends every change to an A/B test.

Watch for a different problem if you use an LLM as a judge. If it favors the new model's tone or answer format even when the underlying quality is the same, its mistakes can differ between the two variants. That can change the apparent direction of an effect. Blind human review of a sample from both variants is a useful check; simply increasing the number of automated judgments will not fix that kind of bias.

## What to ship, test, or stop

For an early team, use offline evals to decide what deserves user traffic before using them to replace an experiment. A candidate that fails known policy cases goes back for revision. A candidate that passes but changes which support requests are escalated should still face a controlled online test with escalation and reopens monitored. A small, reversible prompt adjustment is a more plausible candidate for an eval-only decision if your own paired history supports that narrow use.

Write down the maximum drop you can tolerate on the live metric before looking at new results. If the historical evidence is too thin to estimate the chance of crossing that line, run the online test. Do not set a universal offline cutoff like “ship anything above 90%”: the same score can mean different things on different tasks, and a threshold set after looking at one strong result is easy to fool.

## Keep a random audit after launch

If your release rule sends only borderline changes to A/B tests, you stop collecting online evidence for the obvious wins and losses. You can no longer tell whether the rule still works in those categories. Randomly audit some changes the rule would otherwise ship or stop, where testing is safe and appropriate. Use those results to check whether the offline and live relationship still holds. Revisit the gate if you change the model, judge, rubric, audience, or workflow.

The first worthwhile target is fewer bad ideas reaching an A/B test. Replacing some A/B tests is a later decision, once the team can show which kind of change the eval predicts and keep checking that it still does.

I’m Paras. I help AI founders explain technical decisions and product updates in language their customers can use. If you’re doing that work and want to tell the story behind it, I can help.

[Book a 30-minute call](https://cal.com/paras-madan/30min)
