Agent Substrate API Keys: Credential Injection and 403 Troubleshooting

Agent Substrate can let a sandboxed agent call an external API without placing the API key inside the sandbox. Its egress gateway replaces a request header with a credential obtained through an authorized provider. For teams running agents that execute code, this is worth understanding before defaulting to environment variables.

NVIDIA highlighted Agent Substrate in its November 10 Kubernetes infrastructure update. The practical question is how to configure that separation correctly, especially when the first request returns a 403 or a certificate error. This guide follows the project's documentation; it is not a report of hands-on testing.

What stays outside the agent

The project README describes isolated actors with individual identities and default-deny networking. Supported isolation uses gVisor or Cloud Hypervisor rather than ordinary Linux containers. Credential injection adds a separate boundary between the code making a request and the secret needed to authorize it.

The agent sends a placeholder header. The gateway asks the configured credential provider for the real credential, and the provider checks whether the actor's atespace is allowed to access it. The gateway then forwards the request with the replacement header. The application never needs the raw key in its environment or filesystem.

This reduces a particular exposure: code running inside the actor cannot read a key that was never placed there. It does not make every allowed API action safe. An agent may still misuse the permissions of an authorized destination, so narrow the destination policy and the service account's privileges as separate decisions.

The header must exist before it can be replaced

The credential injection guide makes an easily missed distinction: replacement operates on a header the actor sends. Declaring an Authorization replacement does not excuse the client from including that header.

For an HTTPS API, the relevant policy fragment can look like this. The names below are illustrative and must match your provider configuration and authorized namespace:

rules:
- https:
    hostnames: ["api.example.com"]
    effects:
      replaceHeaders:
      - header: Authorization
        prefix: "Bearer "
        credentialUri: ate-secret://k8s.io/default/ns1/example-api/token

The actor's request supplies the disposable value:

curl -H 'Authorization: placeholder' https://api.example.com/v1/items

The credential URI identifies a secret through the provider. It is not an internet endpoint, and it should not be replaced with a literal API key. Keep credentials out of the actor manifest as well as out of the application image.

A 403 can be the intended result

The reference Kubernetes credential-provider installation grants no namespace access by default. A correctly formed credential URI can therefore still be rejected until the atespace receives the appropriate grant. A refusal from the provider and an invalid credential reference are reasons to inspect the authorization configuration, not to copy the secret into the container as a workaround.

Debug the request in order: verify the allowed hostname, confirm that the client actually sends Authorization, inspect the credential URI, and check the provider's access grant. Then distinguish a gateway rejection from a response returned by the external API. Otherwise a team can spend time changing the remote API key when the request never reached that service.

HTTPS injection needs the gateway's trust bundle

The gateway must terminate TLS to replace an HTTPS header, then establish a new TLS connection to the destination. A tls_passthrough rule leaves the encrypted connection intact and cannot provide the same header modification.

The trust-bundle guide explains how to project the gateway CA and public roots into the actor through a SystemInfo volume. Configure the application runtime to use that bundle: Python requests uses REQUESTS_CA_BUNDLE, while Node.js supports NODE_EXTRA_CA_CERTS.

A certificate-authority error usually calls for checking that runtime configuration. Disabling certificate validation merely hides the missing trust configuration. Do not bake the cluster-specific CA into the application image either, because certificate rotation would leave the image tied to stale trust material.

Test the boundary, not just the successful request

Before relying on this design, verify that an authorized call succeeds, a request to a disallowed destination fails, and an actor without the provider grant cannot obtain a credential-backed request. Inspect application logs to ensure they do not accidentally contain sensitive response data even though the API key itself stays outside.

The core repository currently states a 1.0 stability policy. That does not automatically confer the same maturity on every integration discussed in launch coverage. Pin the version being evaluated and separate core credential behavior from experimental integrations.

The useful outcome is a smaller credential boundary, with permissions that can be reviewed without trusting the agent to keep a copied secret private.

I’m Paras. I work at the intersection of AI, engineering and distribution.

I help teams turn agent infrastructure and security decisions into technical stories developers can understand and use.

Book a 30-minute call