Latest in AI
OpenAI Performance Factory: Design an Agent Workflow That Verifies Recovery
OpenAI's internal performance system offers a useful design lesson: a proposed fix is not a resolved incident. Here is a practical workflow to adapt.
OpenAI's Performance Factory uses Codex to investigate performance regressions, propose fixes and check recovery after deployment. Its November 19 engineering article describes an internal system, not a new public API you can switch on for your application.
The distinction worth borrowing is between producing an explanation and finishing the operational job. A convincing diagnosis can still be wrong, and a merged fix can still leave customers waiting.
The official account describes separate investigation and review steps, engineer-controlled merging, and a return to investigation when deployment fails to improve the problem. Here is how I would translate that idea into a small pilot rather than try to recreate OpenAI's infrastructure.
Pick one symptom with a measurable outcome
Start with one service and one recurring problem. For an AI support assistant, that might be slow first responses on ordinary text questions. Do not begin with an instruction to “improve performance everywhere.”
Google's monitoring guidance distinguishes externally visible behavior from internal signals and recommends tracking latency, traffic, errors and saturation. It also warns that averages can hide slow requests and that successful and failed requests need separate latency measurements.
For this pilot, I would record response latency by request type alongside errors and traffic. Otherwise, an apparently faster system might simply be failing quickly or receiving easier questions. Keep the original query definition with the incident so the closing measurement uses the same population.
Give the investigation a bounded evidence packet
My proposed starting packet would contain the alert time, metric query, affected route, deployment identifiers, a few redacted trace references and an owner. Avoid dumping every log into the prompt.
Ask the agent for two competing explanations and the evidence needed to distinguish them. For the support assistant, a slower retrieval call and a change in model routing are different hypotheses. The useful output is a testable claim, not a confident paragraph combining both.
An unsuccessful query should remain visible as missing evidence. If the agent cannot read a trace, it should record that limitation rather than treat the absence of retrieved data as proof that the dependency was healthy.
Make the state record useful after a restart
I would keep a small incident record outside the conversation. This is an illustrative application schema, not an OpenAI API format:
{
"incident_id": "support-latency-example",
"state": "awaiting_review",
"affected_route": "text-support",
"metric_query_id": "first-response-by-route",
"candidate_change": "reviewable-change-id",
"evidence_refs": ["redacted-trace-reference"],
"next_action_owner": "service-owner",
"verification_result": null
}
Use stable identifiers rather than copying credentials or raw customer messages into this record. A resumed worker should be able to tell what is known, what remains uncertain and who can authorize the next action.
I would also give each state a timeout and an escalation destination. A task waiting for unavailable telemetry should not retry forever or quietly disappear from the queue.
Keep a proposed change separate from permission to ship
For the first pilot, give the investigation worker read access and let it prepare a change in an isolated branch. Production settings and deployment credentials do not belong in its default toolset.
The review should name the affected behavior and a rollback plan. If the proposed optimization caches answers, ask what happens when the underlying support policy changes. A latency improvement that serves stale instructions is not a successful fix.
OpenAI reports that proposed fixes have been harder to get right than diagnoses. That is a good reason to keep the engineering decision explicit rather than interpret a successful test command as release approval.
Verify the deployed change against a control
Google's canarying guidance describes evaluating a limited rollout against the unchanged portion of a service before proceeding. It emphasizes reproducible artifacts and smaller changes that are easier to reverse.
For our example, I would compare the same request category under the changed and unchanged versions, with a pre-agreed observation window and minimum usable traffic. Record the actual deployed revision, not merely the branch that contained the proposed patch.
Keep the closure rule independent of the agent's narrative: the target metric recovers, errors remain acceptable and the selected checks pass. If traffic is too low to decide, mark the result inconclusive. Do not rename uncertainty as success to clear the queue.
Measure whether the pilot reduces review work
Before broadening access, examine a handful of completed and abandoned investigations. Count useful diagnoses, unnecessary changes, missing evidence and time the owner spent checking reports. Keep these as pilot measurements rather than projecting a savings claim from the number of agent runs.
My threshold for expanding this workflow would be straightforward: an engineer can understand the evidence faster, and the system reliably distinguishes a deployed patch from a recovered service. More automated investigations are valuable only when they make that decision easier.
I'm Paras. I work at the intersection of AI, engineering and distribution.
I help AI teams explain complex engineering systems through practical technical writing and developer storytelling.