# Microsoft Copilot Evaluation: What Synthetic Companies Can Test

Microsoft has described an internal evaluation platform built from simulated workplaces, but better test data is not the same as better Copilot answers.

By Paras Madan | 2026-10-30


Microsoft described its Industry Evaluation Data Generation Platform on October 30, explaining how it tests Copilot using synthetic workplaces rather than customer data. The announcement concerns internal evaluation infrastructure, not a new Copilot feature or a publicly released benchmark.

## How Microsoft builds synthetic enterprise tests

The [Microsoft Research account](https://www.microsoft.com/en-us/research/blog/from-synthetic-enterprise-data-evaluations-to-real-microsoft-copilot-ai-confidence/) describes simulated organizations containing users, emails, meetings and documents. Public occupational data, job postings and SEC filings help shape them. The environments cover pharmaceuticals, capital markets and software, with evolving communication histories.

Microsoft says the resulting datasets are available through an internal shared catalog. The post does not provide a public download of the platform or datasets.

## What the reported improvement measures

Microsoft reports a 39% improvement in its composite synthetic-data quality metric compared with its initial generation approach. 

**The 39% figure is not a Copilot accuracy improvement.** It describes the quality of the generated test environments, assessed using Microsoft's internal framework. 

## Why a realistic workplace matters

Consider a hypothetical team that changes a delivery date after three meetings and an email exchange. A useful assistant needs to identify the final decision, distinguish it from earlier proposals and support its answer with the right evidence. Testing only one clean document would miss much of that problem.

This is where the structure of an evaluation environment matters. Adding more questions to a simple document collection will not necessarily test conflicting versions, changing responsibilities or decisions made across several conversations. A smaller but carefully constructed scenario may reveal a failure that a larger collection of isolated questions overlooks.

There are also different failures to measure. Microsoft's [RAG evaluation documentation](https://learn.microsoft.com/en-us/azure/foundry/concepts/evaluation-evaluators/rag-evaluators) separates retrieval quality from groundedness, relevance and completeness. Retrieval asks whether the system found useful evidence. Groundedness asks whether the answer stays consistent with that evidence. Completeness asks whether it left out required information. A fluent answer can still fail any of those tests.

## What enterprise teams can take from this

The useful idea is to make the test environment part of the engineering work. For a team's own evaluation, that could mean preserving a known sequence of decisions, writing down the expected answer and deliberately including plausible but outdated information.

Synthetic tests still depend on what their designers think to include. A convincing simulated company can omit the awkward exceptions that dominate a real workflow. The evidence to watch next is whether improvements in the simulated environment predict fewer failures when the assistant faces unfamiliar work.

I’m Paras. I work at the intersection of AI, engineering and distribution.

I help enterprise AI teams explain technical work clearly, including what an evaluation proves and what it leaves unresolved.

[Book a 30-minute call](https://cal.com/paras-madan/30min)