Raw-RAG vs Agentic Memory: What the New Benchmarks Show

An October 22 research paper finds that Raw-RAG, a retrieval baseline using original conversation history, leads compared agentic memory systems in six of seven benchmark-model settings.

How Raw-RAG works

The paper preserves timestamped history, retrieves overlapping text windows, reranks candidates and passes evidence to a reader model. It avoids LLM-based memory extraction during ingestion.

Reranking adds another relevance check after retrieval.

Consider a user changing a project deadline. A stored summary might retain the new date, while the original conversation also contains why it changed and which dependent tasks moved. Which representation helps depends on the question asked later. This is an illustrative product example, not a reported test from the paper.

What the benchmark does and does not show

The reported advantage ranges from 0.13 to 5.53 points in winning settings. Raw-RAG loses by 1.52 points on MEME with DeepSeek-V4-Flash. The authors limit their evidence to offline public conversation benchmarks and flag reader capability, storage, privacy and deletion constraints for further evaluation.

This does not establish that memory systems are unnecessary.

Code availability is not a deployment recommendation

The released repository carries an MIT license and includes evaluation code, configurations, tests and reproduction instructions. Its setup uses separate model, embedding and reranking services.

Teams can use a baseline like this to investigate where failures occur. Did retrieval miss the relevant passage? Did the evidence omit a relationship between two events? Or did the reader receive the needed information and still answer incorrectly? Those problems call for different fixes.

An accurate answer is not the whole memory contract. A product may also need to remove data, enforce access boundaries and explain where a remembered fact came from. Keeping an original conversation is not automatically the right retention policy merely because it contains useful evidence.

There is a similar distinction between reconstructing the latest state when asked and maintaining it continuously. The more often a product asks related questions, the more important it becomes to measure repeated work as well as ingestion effort.

A memory layer earns its complexity when it improves the behavior the product needs.

I'm Paras. I work at the intersection of AI, engineering and distribution.

I help technical founders explain AI systems and engineering decisions clearly, without making the work sound like marketing.

Book a 30-minute call.