# Lion-ER-35B-A3B: Open Weights and a Benchmark Selection Caveat

LionRock's embodied reasoning model brings Apache-licensed weights, but its evaluation notes disclose a selected result that changes how to read the gains.

By Paras Madan | 2026-11-05


China Merchants' Lionrock AI Lab has released Lion-ER-35B-A3B, a vision-language model for understanding scenes, spatial relationships and task progress. [QbitAI covered the announcement on November 4](https://www.qbitai.com/2026/11/507586.html), reporting improvements over its base model across ten embodied-reasoning benchmarks.

The release gives developers something concrete to evaluate. It also comes with an unusually consequential disclosure: one displayed benchmark comparison was selected to show the largest advantage over the base model. Both details belong in any assessment of the launch.

## What developers can access

The [Hugging Face model card](https://huggingface.co/cmriat/lion-ER-35B-A3B) identifies Lion-ER as an independently fine-tuned Qwen3.6-35B-A3B derivative, with approximately 35 billion total parameters and three billion active parameters per token. The weights use BF16 Safetensors and carry an Apache 2.0 license. The card includes Transformers and vLLM usage examples, while Hugging Face currently lists no inference provider serving it.

The [repository files](https://huggingface.co/cmriat/lion-ER-35B-A3B/tree/main) total approximately 70.3 GB, including 26 Safetensors shards. The active-parameter figure therefore should not be used as the model's storage or GPU-memory estimate. Anyone budgeting a deployment needs to account for the full weights, the serving configuration and the workload.

For a robotics team, a sensible first experiment would be a separate scene-analysis service. Feed it recorded observations and compare its answers with known task states. That creates a place to measure errors before those answers influence physical actions.

## What the headline score measures

The lab reports an equal-weight mean of 65.29 across ten benchmark families. Its main protocol, avg@4, averages scores from four sampled answers per question. The model card also explains that 312,380 supervised training examples can include multiple reasoning trajectories for the same question, so the count should not be presented as that many unique questions.

The [evaluation notes](https://huggingface.co/cmriat/lion-ER-35B-A3B/blob/main/model-card-v10/EVALUATION.md) describe several different scoring systems. Where2Place measures placement quality; RealWorldQA combines multiple-choice scoring with short answers graded by Qwen3.8-27B. The reference models are a selected comparison group, and identical serving conditions are not established across that group.

An aggregate can help summarize a release. It is less useful for choosing a component until the team identifies which failures matter in its own system. Confusing two views of the same object, misreading task completion and misjudging a placement region each call for a different test.

## The selected RealWorldQA result

The most important disclosure concerns RealWorldQA. The lab ran four paired avg@4 evaluations of Lion-ER and its base model, then displayed the pair with the largest Lion-minus-base overall-score difference. Both models use that same selected run; results were not assembled from separate multiple-choice and short-answer runs. Other reference models use one avg@4 run each.

The notes explicitly caution that the selected difference is not an unbiased estimate of the average advantage or evidence of statistical significance.

This creates a straightforward follow-up for anyone reproducing the comparison: retain every run, report the average difference and show how much it varies. Selection can make an observed gain look more dependable than it is. Disclosing the procedure helps readers recognize that uncertainty, but does not remove it.

## What to test before connecting it to a robot

The model card says Lion-ER has not been validated for autonomous robot control or safety-critical decisions. It also warns that spatial outputs can be wrong and that comprehensive freedom from training-test contamination has not been independently established.

A useful evaluation should reserve scenes the team has not used to tune its prompts or select a checkpoint. Include ambiguous views and incomplete observations, then check whether the system can recognize when there is insufficient evidence. Keep the original observations alongside answers so reviewers can trace a confident mistake back to what the camera actually showed.

Measure the complete workflow: answer quality, latency, memory use and the consequences of a wrong answer. For task-progress monitoring, for example, a premature completion judgment may deserve a different penalty from an unnecessary extra check. The benchmark average cannot make that product decision for you.

Lion-ER is worth examining because its weights and evaluation details are available. The next useful evidence is a reproducible comparison on the scenes and decisions a team actually needs to handle.

I'm Paras. I work at the intersection of AI, engineering and distribution.

I help technical founders turn model releases and engineering decisions into clear technical writing and stories that explain what a product can reliably do.

[Book a 30-minute call](https://cal.com/paras-madan/30min) if you want help telling that story.