Gemini Robotics 2.5: Access, Safety Scores and How to Evaluate It

Gemini Robotics 2.5 is rolling out to Google's robotics partners, rather than as a general public robot-control API. Announced on November 10, it improves dexterity and adds safety steering. Teams evaluating it should separate three questions: whether the robot completes the task, whether it acts safely, and how often someone must intervene.

What is available today?

Google's release announcement describes Gemini Robotics 2.5 as a vision-language-action model, which converts visual information and instructions into robot actions. It is being supplied to robotics partners. The announcement does not publish general self-service access, token pricing or an open-weight download.

Google separately uses Gemini 4 Argon for robotics reasoning and research. Buying access to a general Gemini model should not be treated as access to Robotics 2.5 or a complete robot deployment. The model, supported hardware, control system and safety procedures have to fit together.

That distinction matters before starting a pilot budget. Ask the provider which robot configuration is supported, who owns integration, what happens when the model cannot complete a task and which operating conditions are covered. A convincing demo answers none of those commercial and operational questions by itself.

What the 96% factory result tells you

Google reports 96% success for an Apollo 2 automotive pick-and-pack proof-of-concept on Mercedes-Benz production lines. It also reports improvements across more than 200 evaluated tasks. These are vendor-reported results, not measurements from a deployment I tested.

The factory number is encouraging because it concerns actual work rather than a collection of visually impressive movements. But a success percentage needs a denominator and a failure definition before it can guide staffing or throughput plans. A misplaced part, a safe stop and an operator-assisted recovery can have very different consequences.

For a hypothetical 1,000 attempts, a 96% completion rate would leave 40 attempts needing another outcome. That calculation is illustrative, not a prediction about Mercedes-Benz's operations. If every unsuccessful attempt takes three minutes of human attention, it would create two hours of intervention work. If recovery is automatic and brief, the same headline percentage could describe a much more useful system.

Measure completed work after recovery, not just first-attempt success. Record the number and duration of interventions alongside cycle time, damage, rework and safe stops. Otherwise, the robot's apparent productivity can hide work transferred to the people supervising it.

ASIMOV-Jailbreak measures a different problem

Google also announced ASIMOV-Jailbreak, which disguises unsafe requests in robotics scenarios. Its current announcement reports 99.5% jailbreak resistance for Argon against an 82% Gemini Robotics ER 2 baseline. Those figures concern resistance to adversarial instructions, not a 99.5% guarantee of safe physical execution.

A system can correctly refuse a dangerous instruction and still fail a permitted movement. It can also execute a movement precisely while following a request it should have rejected. A useful evaluation must examine both decisions and actions instead of compressing them into one reassuring safety score.

Google's public ASIMOV Agentic documentation already separates physical constraints, uncertainty, feasibility and human-proximity monitoring. The published examples include decisions about payload or reach limits and responses when a gauge cannot be read reliably. This gives developers a useful structure for reviewing their own planner before connecting it to hardware.

Check benchmark access before promising reproduction

The Google dataset page asks users to accept conditions and share contact information to access its files. A public dataset description does not mean every file is downloadable anonymously. Review that access step through the official page rather than assuming a failed download is a broken benchmark.

The visible README documents the existing Agentic evaluation harness, but does not provide a dedicated ASIMOV-Jailbreak command in its task list. Do not substitute an older physical-constraints test and report it as a reproduction of the newly announced jailbreak score. Pin the actual dataset revision, model, evaluation task and scoring method once those details are available.

There is a separate, documented download issue worth recognizing: missing Git LFS data can produce a Parquet magic-bytes error because the local file is a pointer rather than the dataset. That is different from missing access permission. Resolve the cause before interpreting an empty or failed evaluation as a model result.

A practical pilot scorecard

Start with a tightly bounded task and representative operating conditions, with the robot disconnected during planner-only checks. Include normal requests, impossible requests and requests where the system should stop or ask for help. The desired response should be defined before running the evaluation, not decided after seeing what the model did.

For later controlled hardware trials, keep task completion, safe refusal, unnecessary refusal and intervention time as separate measures. Use qualified safety review and the hardware's protective controls; a language-model benchmark is not a substitute for them. Change one part of the stack at a time so that a performance shift can be traced to the model, hardware or integration.

Robotics 2.5 makes the evaluation conversation more concrete. The next useful question is whether a specific installation can maintain its performance while reducing the attention it consumes from people. That is the difference between a robot completing a demonstration and a robot doing dependable work.

I'm Paras. I work at the intersection of AI, engineering and distribution.

I help technical founders explain complex products and evaluation results clearly, including the limits a buyer needs to understand.

Book a 30-minute call