Memdoor’s October 31, 2026 results report that a five-role workflow using Qwen3.8-27B solved 49 of 50 coding-problem runs. Across five passes over the same ten LiveCodeBench Hard problems, the workflow averaged 9.8 out of 10, compared with 8.0 for Fable 5 alone in the original researchers’ result files.
The useful part is the material behind that headline. Memdoor publishes requests, replies, generated programs, grader output, hosting details and model costs, alongside a workflow developers can run themselves. It is a community replication of an existing multi-agent method, with enough documentation to examine where the implementation matched and where it differed. The project README provides the results and setup instructions.
Five Roles Coordinate Through Files
Memdoor’s team-solve workflow implements GVS5H, a training-free orchestration method introduced in a paper first submitted on August 27, 2026, and revised on September 21. The researchers evaluated eleven open- and closed-weight models on 100 hard LiveCodeBench problems. Their central idea is to give fresh instances of one model separate responsibilities, with a shared filesystem preserving work between them. The paper also reports that some models gained nothing or performed worse, so orchestration is not a universal upgrade. The GVS5H abstract describes that broader study.
The Memdoor implementation uses:
- A planner.
- Two ideators.
- A solver.
- A manager.
- An additional repair worker.
These are fresh-context roles using the same Qwen3.8-27B model, not five different model families. Files carry information between stages instead of one continuously growing conversation. That separates planning, idea generation, implementation and management while keeping persistent artifacts available to subsequent calls.
A shell script checks sample-test success. For the published benchmark results, generated programs are additionally graded against every public and hidden test using the original researchers’ corrected grader. A model’s approval is not the final correctness measurement.
The reproduction allows up to ten rounds and uses a 128K reply cap, meaning an output allowance, not simply a context-window specification. Reported temperatures are 0.3 for planning, 0.4 for ideation, and 0.2 for the manager and worker, with top_p set to 0.95 and top_k to 20. These settings matter when comparing implementations because they affect generation behavior. Memdoor documents the configuration.
What the Published Scores Actually Compare
Each pass below covers the same first ten problems from the researchers’ LiveCodeBench Hard list.
| Pass | Memdoor Qwen workflow | Original Qwen harness | Fable 5 alone |
|---|---|---|---|
| 1 | 10/10 | 9/10 | 7/10 |
| 2 | 10/10 | 10/10 | 8/10 |
| 3 | 10/10 | 10/10 | 10/10 |
| 4 | 10/10 | 9/10 | 8/10 |
| 5 | 9/10 | 10/10 | 7/10 |
| Mean | 9.8/10 | 9.6/10 | 8.0/10 |
The original-harness and Fable columns come from the GVS5H authors’ regraded result files. Memdoor did not run Fable 5 inside its workflow. Fable received a single call per problem in those files, while the Qwen workflow could use multiple roles and rounds. This compares two coding systems with different inference budgets, not equal-compute model ability. The repository explains the comparison.
The evidence has a clear boundary: ten distinct problems, repeated five times, are still ten distinct problems. These results have not been independently reproduced in the reviewed sources. The contests ran from March 29 to April 6, 2025, and Memdoor explicitly acknowledges that training-data contamination cannot be ruled out.
I’d treat this as evidence worth reproducing, especially because the traces are public. I would not use it to choose a general-purpose coding model without testing the kinds of work that choice needs to support.
Costs Favor Inspection, but Latency Matters
Memdoor reports $35.79 in model charges for the 50 counted runs, generating 15.3 million output tokens. Including 16 additional attempts interrupted by time limits, fixes or pauses raises the total to $38.77.
The reported median run cost was $0.39. Dividing the counted total by 50 gives an arithmetic mean of approximately $0.72 per run. That difference matters for budgeting: the typical run was cheaper than the average, with expensive runs pulling up the total.
Time is another constraint. Median model time was 40 minutes, and the longest run took four hours. Those figures make the workflow more plausible for asynchronous problem-solving than for an interactive coding assistant expected to respond in seconds.
All reported runs used OpenRouter. The original researchers used their own FP8 vLLM deployment; Memdoor used multiple providers, and Modal’s precision was undisclosed. These are hosted API costs, not measured local-inference economics. The cost and host records are published with the runs.
Reproducing the Workflow
The README supplies Memdoor installation instructions. Review its shell installer before running it, since that installation method downloads and executes a script.
After installing Memdoor, the following commands connect OpenRouter, copy the workflow into Memdoor’s shared workflow directory, and prepare the supplied sample problem:
memdoor connect openrouter
git clone --depth 1 https://github.com/guregodevo/memdoor
mkdir -p ~/.memdoor/workflows
cp -r memdoor/examples/workflows/team-solve ~/.memdoor/workflows/
mkdir my-problem
cp -r memdoor/examples/workflows/team-solve/sample/. my-problem/
cd my-problem
memdoor workflow run team-solve --wait
The sample includes task.md and input/output files under tests/. The final command waits for the workflow; according to the README, it exits successfully when every supplied sample passes.
That exit status is narrower than the published benchmark score. The loop sees public samples, while hidden tests determine the benchmark result. A successful sample run therefore cannot certify correctness on unseen cases. Reproducing the score requires the benchmark procedure and grader, not merely running the bundled example.
Memdoor also documents a local connection option, but matching the published setup requires attention to more than the model name. The team found that its initial problem-text wrapper changed planning behavior; switching to the researchers’ exact problem text was consequential.
Frequently Asked Questions
3 questions
1Does this require five different models?
No. The reported workflow uses fresh contexts of Qwen3.8-27B for five specialist roles, plus a repair worker.
