An open-source implementation of the Memdoor multi-agent framework has outperformed leading frontier models on complex competitive programming tests, using a coordinated group of smaller open-weight models. In a benchmark recreation published on 25 October 2026, a team of five 27-billion-parameter Qwen models solved 49 out of 50 runs across hard problems, beating the single-call baseline of the proprietary Fable 5 system.
Key points
- Five Qwen3.8-27B models running inside Memdoor achieved an average score of 9.8 out of 10 across five passes on LiveCodeBench Hard problems.
- The entire multi-agent system relies on plain text files on disk for communication, giving each model role a clean context window.
- All 50 benchmark runs consumed 15.3 million output tokens and cost $35.79 in cloud inference fees, averaging $0.39 per run.
- The automated workflow was written by an AI coder directly from research notes rather than crafted by human engineers.
How the multi-agent harness works
The workflow replicates the GVS5H approach described by researcher Lee earlier in 2026. Instead of using a single giant model with an expanding context window, the system splits complex tasks across five dedicated roles: a planner, two independent algorithm ideators, a solver, and a manager. A separate repair agent steps in only when validation tests fail.
Rather than using conversational threads or model-to-model dialogue, each agent runs in isolation. When an agent starts work, it receives a completely fresh context window, reads the relevant task files and code saved on disk, and writes its conclusions back to storage. The creators call this ledger discipline, pointing to it as the main reason smaller models avoid confusion on longer assignments.
The test verdict does not depend on an AI judge. A standard Linux bash script executes the written Python script against public input-output samples and inspects the exit code. If the program fails the basic checks or times out, the repair agent attempts to correct the implementation before a final verification step checks the work.
Benchmarking on LiveCodeBench Hard
The benchmark test targeted the first ten problems in the LiveCodeBench Hard set, evaluating five separate passes on each problem. Across the 50 runs, the Memdoor team achieved 49 full solutions on hidden test suites. That gave the setup an average score of 9.8 out of 10, compared with 9.6 achieved by the original GVS5H custom harness and 8.0 recorded by Fable 5 running as a single model.
The entire suite consumed 15.3 million output tokens, costing $35.79 across successful runs, or $38.77 when including 16 aborted or interrupted attempts. The median run took 40 minutes of model processing time and cost $0.39 on hosted Application Programming Interfaces (APIs). The longest single problem run lasted four hours.
The project authors noted an important caveat regarding test validity. The ten benchmark problems originated from competitive programming contests held between 29 March and 6 April 2025. Because the base model, Qwen3.8-27B, was trained later, the researchers stated that data contamination cannot be ruled out, as the model could have encountered the contest tasks or similar solutions during pre-training.
Inference hosts and pipeline tuning
The original research team ran their experiments locally using the vLLM inference engine in 8-bit floating point (FP8) precision. By contrast, the Memdoor reproduction routed calls across several cloud providers via OpenRouter, querying Modal, CoreWeave, Parasail, and Wafer.
Network shifts were common during the test. Modal handled 203 calls before token throughput dropped to 14 tokens per second, prompting the system to route queries exclusively to FP8 providers. Wafer processed 115 requests and CoreWeave took 27, with three runs migrating to different host providers mid-execution while retaining completed disk steps.
Four specific adjustments to the Memdoor framework were required to replicate the researchers’ harness. Developers added support for extracting internal reasoning tokens when a model returned an empty text field, allowed custom top-p and top-k sampling parameters, implemented cutoff flags when answers hit the 128,000-token limit, and introduced four-hour execution timeouts.
Where the pipeline failed
Across the 50 evaluations, the pipeline produced only two distinct failure modes, both inherited directly from the original study’s control loop. The single unsolved pass occurred on problem abc399_e during its fifth run. The solver produced a working program on its first attempt, passing 32 of 44 hidden tests.
The manager agent then fabricated an invalid counterexample containing an incorrect mathematical output. The team rewrote the code to satisfy the manager’s flawed test case. While the altered program satisfied the basic public problem samples, it failed on the hidden evaluation suite.
A second issue appeared in pass 4 on problems arc196_c and arc196_d. Both runs produced code that satisfied all hidden tests, but the manager failed to emit an explicit completion signal. The loop continued running, overwriting the valid code before eventually recovering working programs in later rounds. These extra steps drove total inference costs for those two runs to $3.44 and $4.24.
Local execution and open development
The entire orchestration system runs on open-source tooling, controlled through step definitions written in YAML. The pipeline relies on standard system utilities, executing Python 3 through command-line scripts without requiring custom orchestration servers or proprietary platforms.
Developers running their own workflows can bind tasks to local models rather than external cloud APIs. The system supports direct connections to Ollama, llama.cpp, vLLM, and LM Studio. Users setting up private infrastructure can follow project documentation on how to connect one without API keys.
The team-solve workflow remains available under the project’s public code repository, allowing engineers to swap prompt templates or adjust role definitions for custom software development tasks.
