Latest in AI
CulturalRiddles: Cohere's Cultural Reasoning Benchmark, Results and Limits
A new multilingual benchmark shows why an overall model score can hide the communities where an AI product still needs work.
Cohere Labs and its research community released CulturalRiddles, a benchmark that tests whether AI models can solve culturally grounded riddles. The paper, submitted on October 21, compares 28 configurations. Its useful lesson for multilingual products is how much the aggregate score can conceal.
What CulturalRiddles measures
The released repository contains 6,191 original riddles from 62 communities across 51 languages and 45 countries. The evaluated snapshot is smaller: 6,090 riddles and 170,520 model answers. Those are different counts, not interchangeable descriptions of the test set.
These questions depend on local references, traditions and wordplay. The evaluation combines rule-based scoring, native-speaker assessment and agreement between two model judges. The distinction matters because a valid answer can differ from the reference wording. An exact text match is a convenient scoring method, but it can penalize understanding expressed differently.
For a product team, the relevant question is whether its existing tests capture that kind of variation. A translated English test can check language handling while missing the references that users actually bring into a conversation.
CulturalRiddles benchmark results
The paper reports these overall accuracies for the tested configurations:
| Model configuration | Accuracy |
|---|---|
| Gemini 3.1 Pro Preview, high reasoning | 79.4% |
| GPT-5.5, default configuration | 76.3% |
| Claude Opus 4.7 Thinking, high reasoning | 66.0% |
| DeepSeek V4 Pro, high reasoning | 63.8% |
The sharper result is that no tested model exceeded 22% in the Lumasaba and Mongondow communities. Translating riddles from five difficult communities into English closed only about one-fifth of the gap to high-resource communities.
A strong overall score does not establish readiness for every audience. If a product serves a particular community, its performance there matters more than its position on a pooled leaderboard. That is a reason to inspect the breakdown, not a reason to discard the overall measurement.
What developers can use
The repository includes a benchmark runner, released answers and analysis code. Its documented reproduction workflow can rebuild the published outputs without making fresh paid model calls. Running a new model or new judge requests is a separate activity that can require provider credentials and inference spending.
The dataset card also makes an important licensing distinction. The code is Apache-2.0, but the data has separate evaluation-only terms. Those prohibit training, fine-tuning, distillation and using the material to seed synthetic data. This is not a ready-made training corpus for fixing the failures it reveals.
A useful first pass would be to inspect the communities relevant to a product, then examine incorrect answers alongside the scoring evidence. That can help distinguish a missing reference from a language problem or an answer the scoring system handled poorly. It still needs to be followed by tests of the actual product workflow.
Where the benchmark needs caution
Each configuration answered once with one prompt, so this is not a robustness study across many prompting strategies. The paper also identifies shared judge errors, including acceptance of some human-rejected answers and rejection of valid alternatives. Human-informed evaluation improves the measurement without making it infallible.
The dataset warns that generated model answers are released without filtering and may contain offensive or fabricated content. Teams should treat them as evaluation material, not reuse them directly in customer-facing examples.
Riddle performance also cannot establish that an assistant will handle customer support, translation or local recommendations correctly. Those tasks have different context, retrieval and interaction requirements. CulturalRiddles adds a useful diagnostic, rather than replacing task-specific evaluation.
The practical change is to make the audience visible in the evaluation plan. A product can support a language in its interface and still need much more evidence before it works well for the people speaking it.
I'm Paras. I work at the intersection of AI, engineering and distribution.
I help technical teams turn research and product updates into clear explanations that developers and founders can use and share.