PARAS MADAN
All writing

Latest in AI

OpenAI API Latency Troubleshooting: Match Dashboard Filters Before Changing Models

A slow request and a healthy aggregate chart can both be real. Compare the same project, model, tier and UTC window before diagnosing an API regression.

A slow OpenAI API request does not necessarily mean the model became slower. The request may have produced more output, used a different processing tier or waited inside your own application. OpenAI's updated troubleshooting guidance makes a useful distinction: service performance and token usage explain different parts of the incident.

I would start by aligning the evidence, not swapping models. Otherwise, a change that appears to fix the problem can leave you unable to explain why it happened or whether it will return.

Check which dashboard you can actually use

The Service Health guide says that dashboard is currently available only to Enterprise API customers. Data residency alone does not grant access. Without it, retain sanitized request identifiers, errors and timing records for investigation and support.

Where available, isolate one model, processing tier and affected project. Use HTTP Requests for status-code errors rather than reading the Uptime view alone. Compare latency percentiles rather than relying on an average.

For a proposed investigation, I would save the selected filters beside the screenshot. “Latency increased yesterday” is hard to act on when nobody knows whether the chart includes the affected service or a larger unrelated workload.

Use the same reporting window on both sides

The Usage guide places API activity under Analytics, with separate project and organization views. Dashboard timestamps use UTC. Clearing filters does not change that scope, and recent usage can take time to appear.

An unavailable value is not evidence of zero activity. The guide also notes that a permission error in a filter does not mean no requests were recorded.

Write down the incident interval in UTC and use it in application logs and dashboard exports. For an India-based team, that avoids comparing an evening local incident with the wrong reporting day. Include an unaffected interval with similar traffic as a baseline rather than choosing the quietest hour of the week.

Separate waiting for output from generating it

Service Health distinguishes time to first token, total request duration and token velocity. Reasoning and uncached input affect the first; output size and reasoning affect total duration. Those metrics answer different questions.

My application-side record would additionally separate time spent waiting in its own queue from time after the API call begins. Keep retry attempts visible instead of reporting only the final successful call. A user experiences the entire wait, even when the last attempt was fast.

For an illustrative comparison, suppose completed requests now produce twice as much output while their output rate stays similar. Longer total duration would not by itself establish slower generation. Check the actual token counts before treating that hypothesis as the explanation.

Export a narrow, reproducible comparison

Usage exports support Activity data, a capability selection, grouping dimensions and a reporting interval. Review the export filters explicitly; do not assume every filter follows automatically from the dashboard. The guide distinguishes activity from billed cost, including cases where an activity adjustment does not establish an invoice credit.

I would compare two small windows with the same project, model and tier. Record requests, output tokens, failures and user-visible duration. Calculate tokens per completed request separately from raw totals, and note whether retries are represented in the denominator.

Keep the resulting table with the application revision and configuration. That gives the next engineer a reproducible starting point rather than a screenshot whose settings have been forgotten.

Diagnose the error before retrying

The API error reference distinguishes request throttling from exhausted credits and enforced spending limits, even when they share HTTP 429. Inspect the error code. Retrying a billing limit does not restore access; transient throttling requires a bounded retry policy that respects Retry-After when present.

Do not combine a model change, larger timeout and higher retry count in the first experiment. Change one variable against the same test requests so you can identify what helped. Preserve any required regional routing while investigating.

I have not reproduced an incident here. The practical completion criterion is an explanation supported by matching records: where the time went, which requests were affected and what changed. A green chart without that evidence is not a diagnosis.

I'm Paras. I work at the intersection of AI, engineering and distribution.

I help AI infrastructure teams turn debugging workflows and product changes into clear technical writing.

Book a 30-minute call.