# ToolSpace turns real MCP traces into harder agent training tasks

ToolSpace builds agent training environments from real tool traces. Its new benchmarks expose long-workflow failures, with practical limits for teams evaluating it.

By Paras Madan | 2026-11-05


ToolSpace, a November 3 research preprint, starts with records of real tool calls to build training environments for AI agents. For teams working with Model Context Protocol (MCP), the interesting question is whether training on those dependencies helps an agent finish a workflow after the first successful call.

Consider a hypothetical assistant updating a customer record and then creating an invoice. The invoice needs the correct customer identifier, and a failed update may change which action should happen next. Evaluating each call separately would miss much of what could go wrong between them.

## How ToolSpace builds a workflow

The researchers extract data, state and retry dependencies from MCP execution traces, then organize tools into a graph. They synthesize databases for environments that change state and reuse recorded outputs for query or stateless tools. State-aware walks generate executable chains, with rollback after failures. Generated tasks must replay successfully from their starting state before acceptance.

Across seven benchmarks, the authors report that Qwen3.6-35B-A3B's average rises from 23.0 to 41.0 after supervised training, reaching 43.2 with reinforcement learning. GLM-5.2 rises from 44.6 to 56.6. They also introduce harder AutomationBench Pro and tau3 Banking Pro tasks, reporting drops of up to 38.2 points among tested frontier models. These are author-reported preprint results, not independently confirmed findings. [Read the paper](https://arxiv.org/html/2611.05036v1).

A cross-benchmark average needs care. It compresses different tasks and scoring rules into one number. A team choosing an agent for invoice reconciliation should inspect failures on that kind of work before treating the aggregate improvement as evidence for a deployment decision.

## What builders can actually inspect

The [public repository](https://github.com/dongguanting/ToolSpace) includes seven benchmark harnesses and an AutomationBench graph example covering 47 services and 758 endpoints. That example is smaller than the full research graph. The released AutomationBench Pro package contains 300 frozen tasks across six domains; tau3 Banking Pro contains 98 tasks and 698 knowledge documents.

There is an offline demo that uses fixture model responses while executing the environment and replay machinery. Full online evaluation adds external dependencies, including auxiliary models, search services and dataset access. An offline demonstration therefore should not be presented as a reproduction of the reported model scores.

For a first evaluation, I would keep the scope narrow: choose one workflow that resembles the product, inspect its starting state, and check exactly what the verifier accepts. Then deliberately introduce a wrong identifier or an interrupted step. A useful test should expose those failures without allowing a plausible final message to count as completion.

## Early interest, with replication still ahead

ToolSpace's [Hugging Face paper page](https://huggingface.co/papers/2611.05036) was submitted to Daily Papers on November 4 and showed 34 upvotes when checked early November 5. That is an early attention signal, not evidence of broad adoption or independent validation.

The practical reason to examine this release is its treatment of dependencies between actions. For anyone building an agent evaluation suite, a concrete next step is to write down what must remain true after each tool call and make those conditions testable. Keep the original traces alongside failures so a reviewer can distinguish a bad model decision from an environment or scoring problem.

I'm Paras. I write about AI tools and the engineering decisions behind them.

I help teams turn technical work into clear writing, product stories and founder-led distribution that gives readers enough detail to judge the work.

[Book a 30-minute call](https://cal.com/paras-madan/30min).