PARAS MADAN
All writing

Latest in AI

How to audit prompt caching and tool archives in Hermes Agent

Check what Hermes Agent 0.23.3 keeps in the prompt cache, when it shortens old tool output, and which data remains on disk.

A long tool result can cost you twice: once when an agent reads it, then again when the agent sends it back with later model requests. Caching can lower the second cost. But when an agent shortens its history, that does not necessarily remove the old result from disk.

Hermes Agent 0.23.3 brings together a new prompt-cache lifecycle and changes to how tool results are archived. If you operate long-running Hermes sessions, check the cache and the stored history separately before changing any settings.

Start with one long, reproducible session

Use a disposable profile and a generated fixture, such as a long test-results file containing no secrets or customer data. Have the agent read the fixture, then ask about one result. Ask a follow-up without adding another large file. That gives you the same early history to compare across requests.

Capture the Hermes version, model, provider route and the approximate pause between turns. If you change models while comparing two runs, you're changing the route and its caching behavior, not just the contents of the conversation. Start with one route before drawing conclusions about your fleet.

Read the cache clock, not just the token total

The versioned developer guide describes the cache clock and the new lifecycle. Start with /context to see the session's cache state, then compare it with the provider's reported usage. An unknown window isn't evidence that the next request will be a cache hit.

The tagged defaults set prompt_caching.cache_ttl to auto. For the applicable Anthropic-style cache configuration, Hermes chooses a longer request tier for human-paced sessions and a shorter one for machine-paced work. That isn't a guarantee that every route caches for an hour. Hermes can send eligible keep-alive calls, but it does not warm Responses API sessions this way.

Do not confuse cold and warm pruning

The release notes describe proactive pruning as on by default. The important distinction is timing. The configuration guide for this tag describes a default cold-cache path when Hermes determines the old prefix is no longer reusable. Before rewriting a still-warm prefix, the optional size-triggered path looks at compression.proactive_prune_tokens. Its tagged default is 0, so that trigger is off unless configured.

Leave the default in place for the first comparison. A shorter prompt after a prune can reduce later input, but rewriting a warm prefix can also forfeit a cache hit. Compare cached input, uncached input and total request cost across several representative turns, not just after a setting change.

Verify that archived results stay available

During compaction, Hermes can replace old tool output in the live prompt and write the original to cache/pruned/ under its home directory. The developer guide describes how this affects backends that do not publish that folder. Shorter live context isn't the same thing as deletion.

In your disposable fixture, check whether the expected archive exists and whether the profile can read back a known value from it. Don't ask the agent to rerun an old tool to reconstruct its result. Verify that your selected backend can read the fixture after compaction before relying on the archive for follow-up questions.

Review retention before repeating the test on real work

The tagged configuration guide says referenced archives can remain with the session, including after compaction, and automatic session pruning is off by default. Include archive files and session storage in your retention review. Don't treat compaction, a smaller context readout or an idle cache timeout as a cleanup policy.

Write down three outcomes before rolling out a change: how the second request was billed, whether the fixture was recoverable, and where the original result still lived. Those answers tell you more than a small context-window number alone.

I'm Paras. I work at the intersection of AI, engineering and distribution.

I help AI teams turn runtime changes and operational tradeoffs into technical writing and product stories developers can use.

Book a 30-minute call.