Latest in AI
Memdoor Qwen Benchmark: What 49 Out of 50 Runs Actually Shows
A new multi-agent coding experiment reports strong results on ten repeated problems, but the workflow failures are as useful as the score.
Memdoor published an October 31 coding experiment in which a Qwen3.8-27B workflow solved 49 of 50 runs on a small LiveCodeBench Hard subset. That is a useful result for developers building agents around open models, but it needs a narrower reading than “Qwen beats Fable 5.”
The experiment used ten problems repeated five times, not 50 different problems and not the full benchmark. Its value is that the project published the runs, costs and failures alongside the headline.
What Memdoor tested
Memdoor's team-solve workflow rebuilds the GVS5H method using fresh model contexts and shared files. A planner establishes an approach, two ideators develop alternatives, and a solver produces code. A manager reviews progress, with a worker making further changes.
The benchmark replication used the original authors' prompts and loop, with up to ten rounds. Agents saw public examples; the final scores used hidden tests. This separates the evidence available while solving from the tests used to judge the answer.
Memdoor's results and comparison limits
The published run records show the following totals across the same ten problems and five passes:
| System | Successful runs | Where the results came from |
|---|---|---|
| Memdoor Qwen3.8-27B workflow | 49 of 50 | New replication |
| Original Qwen3.8-27B multi-agent harness | 48 of 50 | Original authors' regraded files |
| Fable 5 single-call baseline | 40 of 50 | Original authors' regraded files |
Fable 5 was not rerun inside Memdoor. The original research covered 100 problems; this replication used its first ten. Neither the small difference between the two Qwen workflows nor the larger baseline gap establishes general coding superiority.
There are other limits. The problems came from contests held in March and April 2025, so training contamination cannot be ruled out. Memdoor used OpenRouter providers, while the original Qwen harness used a pinned, self-hosted FP8 backend. These were not identical serving conditions.
The cost includes time and coordination
The 50 completed runs cost $35.79, rising to $38.77 when 16 interrupted or stopped attempts are included. Median model time was 40 minutes per run, with the longest taking four hours. These are reported experiment costs, not a product subscription or a promised cost per coding task.
The workflow can connect to local runtimes, but these results were produced through hosted inference. They do not demonstrate the same latency or throughput on a laptop.
For a developer, that changes the comparison. A smaller model can make more attempts and coordinate through files, but the relevant measure is the cost and time to reach a verified solution, not simply the price of one model call.
Why the failed run matters
In the unsuccessful run, the manager proposed an incorrect counterexample. The team replaced an initially correct program with one that passed the public samples but failed some hidden tests. In two other runs, later rounds overwrote correct solutions before eventually recovering them.
More deliberation can therefore make a working answer worse. A practical next experiment would preserve previously verified candidates and independently check a manager's proposed counterexamples before allowing a rewrite. The project describes these safeguards, but they were not part of the reported results.
For teams testing multi-agent coding, Memdoor offers an inspectable workflow rather than a reason to stop evaluating models. The next useful result would show whether the gain survives newer unseen problems without turning extra coordination into avoidable delay.
I’m Paras. I work at the intersection of AI, engineering and distribution.
I help technical founders turn benchmarks, launches and engineering decisions into clear stories that developers can understand and share.