Northflank Security Incident: AI API Keys and Environment Variables at Risk

Northflank disclosed a security incident on October 30 and told all users to assume their environment variables had been decrypted. The hosting platform also said multiple users were reporting use of their AI keys. For teams running AI applications there, the immediate task is credential rotation, not waiting for a full postmortem.

What Northflank has confirmed

The official incident notice records a Wiz threat alert at 11:43 UTC, followed by network-policy changes and additional patches. Those are response milestones, not a confirmed start time for the intrusion.

At 13:58 UTC, Northflank instructed users to disable or rotate all keys. Its 15:12 UTC update said secret rotation was enabled for Postgres and MongoDB, and recommended disabling public exposure on addons where users had enabled it. The notice specifically calls out workload secrets, secret groups and global secrets.

At the time of writing, the incident remained marked Identified. Northflank had not published a complete root cause or an account-by-account exposure assessment. Its report about AI-key use does not establish which providers, how many accounts or what charges were involved.

Updating a secret is not the same as revoking it

An API key is a credential issued by another service. Replacing the copy stored in a hosting dashboard does not, by itself, make the old credential invalid at that service.

That distinction matters during this incident. An application can start using a replacement key while someone else continues using the old one. The old credential needs to be revoked at its issuer, alongside updating the workloads that depend on it.

OpenAI's API-key guidance recommends immediate rotation when a leak is suspected, checking usage and updating production applications. Teams using other providers should follow those providers' incident and revocation procedures rather than assume a Northflank setting covers every external account.

For an AI application, review the provider's usage and billing records as well as the application's own logs. Requests made directly with a stolen key would not need to pass through the application, so normal-looking application traffic is not enough to rule out misuse.

Check which workloads actually receive the replacement

Northflank's secret-group documentation explains that services and jobs inherit values from groups. Saving a group does not restart workloads; the documented restart-dependents action redeploys consumers with the new values.

There is another detail worth checking during recovery: variables set directly on a service or job override a same-named value inherited from a secret group. A team could update the shared group and still leave one workload using its old local value.

The API and CLI also replace the full variables or files collection when those fields are sent, including in a PATCH. An automated rotation script should preserve the other entries rather than treat that operation as a single-key edit.

These are configuration checks, not evidence about how the incident happened. They matter because a rushed rotation can leave either an exposed credential active or a legitimate application unable to authenticate.

What AI teams should take from this

The useful longer-term question is how quickly a team can identify every place a credential is stored, revoke it and verify its replacement. A secret can be hidden from source control and still depend on the security of the platform holding it.

After containment, separating credentials by workload and keeping permissions narrow can make the next incident easier to investigate. For now, successful recovery means checking both sides of the connection: the provider no longer accepts the old key, and the intended workload works with the new one.

I'm Paras. I work at the intersection of AI, engineering and distribution.

I help technical teams turn complex product updates and engineering decisions into clear stories developers and founders can use.

Book a 30-minute call.