Latest in AI
Railway Security Incident: Rotate AI Keys, Check Usage and Request Refunds
Railway's full report gives affected teams a clearer recovery path, but replacing a secret and resolving the bill are separate jobs.
Railway's November 10 incident report explains how attackers obtained customer credentials during its October security breach. If you stored AI provider keys on Railway, the practical response is to identify potentially exposed keys, revoke them, deploy replacements and reconcile provider usage. A working application does not prove the old credentials stopped working.
This is a follow-up to the October incident, not evidence of a new intrusion in November. Railway says its investigation found no attacker access to customer data after October 21 at 09:25 UTC. The report also makes an important distinction: stopping access to Railway does not automatically stop someone using a copied API key elsewhere.
What the full report establishes
In its full incident report, Railway puts the upper bound of workspaces whose variables could have been decrypted at 1.85%, approximately 100,000. That is an upper bound derived from environment-level logs, not a confirmed count of individual secrets stolen. Railway also reports access to stored histories of 353 builds, including sealed variables.
Railway estimates unauthorized spending on stolen keys at roughly $740,000. Its forensic account says AI credentials were the primary target, while another query also matched cloud, database and payment credentials. The company says its forensics did not indicate that customer databases were accessed.
These are Railway's findings, with specialist forensic work still ongoing at publication. They support reviewing affected credentials and usage. They do not support telling every Railway customer that all of their data was stolen, or treating the absence of a large bill as proof that a key was never exposed.
Revoke the provider key, then update every consumer
Start with Railway's notifications and your own inventory of credentials stored there. Identify the provider account, project or workspace that owns each potentially exposed key, then the services and environments that consume it. Do not paste secret values into your incident document. Record key identifiers, owners and deployment names instead.
For OpenAI, Railway's recovery guide distinguishes project keys from organization-level or legacy user keys. Select the correct project before deleting the affected key, create a replacement, edit the Railway variable and deploy the staged change. Check exposed admin credentials separately rather than assuming a project-key rotation covers them.
For Claude, follow the equivalent process in the correct Console workspace. Railway's Anthropic guide also calls out shared variables: update the shared value, then deploy the services that reference it. A new value sitting in configuration is not enough if an old deployment still uses the previous value.
Revocation can interrupt the application until the replacement is deployed. Plan for that interruption, but do not leave a suspected stolen key active merely to avoid a brief outage. After deployment, make a small legitimate request through each affected application path and confirm that it succeeds without printing credentials into logs.
Consider an illustrative setup with a web service and a background worker sharing one Claude key. Updating and testing only the web service can leave the worker failing hours later when its scheduled job runs. The useful completion check is a list of consumers with a confirmed replacement deployment, rather than a single successful browser request.
Preserve usage evidence while stopping further spending
Provider usage needs its own review because copied credentials can be used outside your application. Compare key-level usage with application traffic, model selection and normal working patterns. Unexpected models, token volume or activity during an otherwise idle period are investigation signals, not automatic proof of who made a request.
OpenAI's usage view can help separate projects and keys. For Claude, compare the affected key's usage and costs, then inspect available request logs for mismatched timing or patterns. Save the relevant dates, key identifiers, amounts and exports so the billing investigation does not depend on someone's recollection.
OpenRouter needs a slightly different sequence. Railway's OpenRouter guide recommends disabling the key, which stops access while retaining its history, before deciding whether to delete it. Export the relevant Activity data before deletion, and give the replacement an appropriate spending limit and expiration.
Changing the Railway variable does not revoke the provider credential. If a stolen key remains active at its issuer, moving the legitimate application to a new key leaves the attacker's copy usable. Conversely, deleting a key at the provider without deploying a replacement stops both unauthorized access and the application's legitimate requests.
Request reimbursement without assuming approval
Railway's November 10 report describes different routes by provider. For OpenAI, it tells affected customers to open a support request and give the resulting case ID to Railway support; requests are reviewed case by case. For SpaceXSI, it directs customers to Railway support with their claim.
The same report says Railway is covering breach-related Anthropic costs while discussions continue, and breach-related OpenRouter costs after OpenRouter maintained its no-refund policy. It says AWS Bedrock and Google Gemini account teams have initiated refunds under their existing fraud policies. These are Railway's stated arrangements at the time of the report, not a promise that every submitted amount will be automatically approved.
Use the claims form linked from the report's unauthorized-spending section and retain the provider case reference, affected period and supporting usage evidence. Keep claimed, approved and actually refunded amounts separate in your records. That makes a partial reimbursement visible instead of accidentally marking the whole incident resolved.
Make the next stolen key less useful
The recovery work exposes a design question worth answering after service is restored: how much can one key spend, and how many workloads depend on it? Separate credentials by application and environment where the provider supports it, apply suitable limits, and assign an owner who can connect a spending alert to a real workload.
A useful recovery record should establish three things: the old credential is no longer accepted, every legitimate consumer uses its replacement, and unexplained usage has been accounted for. Until those are independently checked, “the deployment is healthy” is too weak a definition of recovery.
I'm Paras. I work at the intersection of AI, engineering and distribution.
I help technical teams explain incidents, infrastructure decisions and recovery workflows clearly enough for developers to act on them.