Databricks Genie Ontology: How to Evaluate the 91% Benchmark Claim

Databricks made Genie Ontology generally available on November 11, 2026, alongside a benchmark reporting a 91% pass rate for Genie. The practical question is whether your company's definitions, dashboards and historical exceptions give it the same advantage. This is a context-system evaluation, not evidence that one underlying model wins every enterprise task.

What Genie Ontology adds

The GA announcement describes a shared business-context layer. It combines definitions people deliberately govern with knowledge learned from analytical work, then ranks that context by authority. An agent can use the result without rediscovering the meaning of every metric from raw tables.

The release includes generally available Genie Search and organizational directories. Power BI support for external knowledge sources is beta; Tableau and Sigma support are still coming. SharePoint search is also beta. Those distinctions matter when the argument for buying a platform depends on the systems your company actually uses.

What the benchmark proves, and what it leaves open

In the evaluation developed with Turing, Databricks used a fictional UK energy supplier with a populated workspace, including old analyses and business rules beyond table schemas. Each agent ran five times per task.

System	Pass rate	Average time	Average tokens
Genie	91%	100.3 seconds	431,829
Agent A	45%	434.6 seconds	889,019
Agent B	58%	306.5 seconds	1,051,070

The competitors are unnamed, and their Databricks toolbelt excluded Genie MCP. Databricks says the gap may be smaller in new workspaces or on questions answerable directly from schemas and tables. These are vendor-reported results, not an independent test performed here.

The exclusion is important when choosing a deployment. The experiment compares Genie with agents that must find context without Genie, rather than asking whether your preferred assistant performs better after connecting to it. It supports testing a shared context layer, not automatically replacing every assistant.

Start with the definitions that can change a decision

The Ontology documentation distinguishes modeled knowledge, such as certified metric views and Pages, from learned snippets. Modeled knowledge is more authoritative when the two conflict. The documentation also warns that snippets from external knowledge sources are available to all users in the account, a boundary worth reviewing before importing BI definitions.

For an evaluation, choose one consequential question and write down what a correct answer must contain before asking the agent. A useful fictional example is: "How many customers activated last month?" Specify whether activation means a first login, a completed setup or a first successful production job, and whether the date refers to account creation or the activation event.

Now ask the question using the everyday language a sales lead would use. Check whether the agent applies the approved definition, identifies the time period and exposes enough evidence for someone else to reproduce the number. A fluent explanation with the wrong denominator should fail.

Test the context layer against an ordinary integration

The new Genie plugins let external assistants obtain business context and cited answers from Genie. That creates a useful third condition for your pilot: the existing assistant alone, Genie directly, and the existing assistant using Genie.

Use the same questions and evaluation criteria in all three conditions. Include a historical definition that differs from today's rule, a term shared by two departments, and a question the user lacks permission to answer. Record analyst correction time alongside accuracy. A system that reaches the right number but needs ten minutes of evidence checking may not save much work.

Also separate the work of preparing the context from the cost of answering the question. If someone must repair a stale metric definition, that effort belongs in the evaluation even when it is useful maintenance the company should have done anyway.

The strongest reason to adopt an ontology is that a correction becomes reusable across workflows. If every team still needs its own prompt explaining what "activated" means, the company has added an agent without resolving the expensive part of the problem.

I’m Paras. I work at the intersection of AI, engineering and distribution.

I help enterprise AI teams explain technical capabilities, evaluation tradeoffs and product decisions without turning them into marketing claims.

Book a 30-minute call.