Benchmarking Terminal Coding Agents: 920 Public Trials and 460 Private Trials [Part 2]

This is Part 2 of a three-part series on running, benchmarking, and scaling local coding agents. Part 1 covers the local Qwen setup; Part 3 explores running a larger MoE model on the same desktop hardware.

In Part 1, I got Qwen 3.6 35B A3B NVFP4 running locally on an RTX 3060 with a 64k context window and wired it into Crush, Pi, and Oh My Pi. I wanted to know how that setup compared with the terminal coding agents I already had on subscriptions. Could it finish the same tasks, and would I want to merge the patches?

We started with four configurations and kept adding comparisons as the results raised questions: different models in one harness, different harnesses on one model, a workspace flag, and changes to reasoning effort and output limits. The current public matrix has 23 configurations, eight tasks, and five runs per task: 920 trials. The private matrix adds 23 configurations across four tasks in a production Go monorepo: 460 trials.

That is 1,380 recorded trials across two different task sets. The private repository and patches remain private; this post uses aggregate results and generic task descriptions.

Update September 17, 2026: Three Harnesses and One Runner Bug

I added Claude Code on the Flash-Next endpoint, plus DeepSeek Harness on Flash-Next and on the local Qwen. That adds 120 public and 60 private trials.

More importantly, I had to retract yesterday’s migration-review correction. A broad bin/** exclusion in the runner removed the required bin/es.js file from captured patches before review and replay. Fifty surviving workspaces show that the agents had written it; their patches passed when regenerated without the bad exclusion. Sixty-five older workspaces were already reclaimed, so their earlier event-store review scores remain in the tables. The reports record that limitation.

Update September 16, 2026: Claude Code on Other Models and a Superseded Review Correction

I added Claude Code 2.1.272 to four more lanes: forty public DeepSeek trials, twenty private DeepSeek trials pinned to the same endpoint as the other private DeepSeek rows, twenty private DeepSeek trials with ordinary provider routing, and twenty private trials against the local Qwen. The public row makes the total 800; the three private rows bring that set to 400.

The same review round appeared to find a defect in a migration task that the hidden tests did not exercise. The September 17 investigation found that the captured diffs had silently omitted the command-line file. The correction in this section is superseded by the update above.

Update September 15, 2026: Flash-Next in Three Harnesses

The September 14 version covered 640 public and 280 private trials. We added 120 public and 60 private trials of Qwen3.8-Flash-Next-FP8, a 177B MoE model served by vLLM on a colleague’s cluster, using Pi, Crush, and Oh My Pi. The tables below include those runs; the scoring formula is unchanged.

This gave me a useful follow-up to the local experiment: how would the larger model behave with enough serving capacity? Public correctness reached 99.7%-99.8% in all three harnesses. On the private set, Pi passed all 20 runs, while the blind reviewer preferred the patches from Crush and Oh My Pi. I have added that comparison and the shared-endpoint and rerun caveats below.

Update September 14, 2026

I re-scored both matrices after reviewing the blind-quality weight. Correctness now carries 72% of the score, compliance 15%, and blind quality 13%. The usability threshold also moved from 300 to 450 seconds. The trial records did not change; the tables and conclusions below use the updated scorecards from both reports.

The Contenders

Every trial ran headless: one prompt in, a git patch out, and no human editing during the run. The public matrix includes eight subscription configurations, six local configurations, four metered API configurations, and five configurations on the shared Flash-Next endpoint. These are the model identifiers and CLI versions recorded for these runs.

The local configurations all used the same Qwen model and RTX 3060 with 12 GB of VRAM. The -lo configurations requested low thinking and allowed 16,384 output tokens per message; baseline Pi and Crush used an 8,192-token cap. Oh My Pi also used low thinking and the 16k cap.

The subscription rows have no per-token bill for these runs. Claude Code’s reported $26.00 for 40 Opus trials and $10.73 for Sonnet are API-equivalent estimates. The OpenRouter rows are metered: the public Pi runs report nominal costs of $1.61 for Qwen and $0.18 for DeepSeek. Claude Code’s own cost field is an Anthropic list price when it drives DeepSeek, so I do not use it. The 40 public and 20 private unpinned Claude Code/DeepSeek trials used $0.89 of actual OpenRouter credit. Local inference avoids API charges; electricity and hardware still cost money.

The five Flash-Next configurations used the same direct vLLM endpoint at no charge to me. They measure remote inference, so their timings do not describe what the 177B model can do on my RTX 3060. No zero-retention policy is established for that endpoint by these results; the provider settings described for OpenRouter below apply only to those OpenRouter runs.

What We Asked Them to Do

The public tasks are small repositories, with 26 to 129 lines of source each, spanning Python, Node/TypeScript, and Go. Each includes visible tests and a separate hidden suite.

Each configuration ran every task five times in a fresh workspace. A single run would have hidden several failures that only appeared on one attempt.

How a Trial Works and How It Is Scored

1. Copy the task into a fresh workspace outside the benchmark tree and create a starting git commit.

2. Launch the agent with the task prompt. Permissions are auto-approved; external integrations are disabled where supported.

3. Record the patch, wall time, exit status, and available token usage.

4. Run hidden tests and compliance checks through a separate grader account.

The score now gives 72 points to correctness, 15 to compliance, and 13 to blind quality, then subtracts a usability penalty of up to 10 points.

Correctness is the mean of each trial’s hidden-test pass fraction. It is not a pooled count of every assertion across tasks. Compliance covers checks such as preserving visible tests, adding requested regression tests, and passing go vet. A fully passing run means all hidden tests passed; it can still miss a compliance check.

Quality is scored from 1 to 5 on anonymized patches, then normalized with (quality - 1) / 4. Claude judged the patches before the author mapping was revealed. Blind review reduces author bias, but it does not remove the preferences of the judge. The September 13 revision assigned quality 25 points. Reducing that to 13 limits the judge’s influence, though quality can still outweigh a small correctness difference.

The scoring code computes the penalty from the configuration’s overall median wall time and timeout fraction:

slow_fraction = clamp((median_seconds - 450) / 750, 0, 1)
penalty = 10 * (1 - (1 - slow_fraction) * (1 - timeout_fraction))

score = 72 * correctness
      + 15 * compliance
      + 13 * ((quality - 1) / 4)
      - penalty

With no timeouts, there is no penalty at or below 450 seconds, and the penalty reaches 10 points at 1,200 seconds. Timeouts add a penalty even when the median is fast: two timeouts in 40 runs cost crush-ords 0.5 points despite its 114-second median.

The old formula awarded 15 points for relative efficiency, scaled by the fully passing fraction. That counted correctness again and made scores depend on whichever configuration was fastest in the table. Adding a configuration could move everyone else’s score without changing any patches. The intermediate 60/15/25 weighting corrected the relative-speed problem but let one reviewer’s stylistic preference outweigh small correctness differences, which is why the quality weight was reduced.

The new formula removes that dependency. Selecting a subset of configurations leaves their scores unchanged. Both task sets use the same scale, though different tasks still mean different difficulty. A higher private score does not imply that a model improved between runs.

What Broke in the Benchmark Runner

Getting reliable results took several changes to the runner.

1. Agents Leaving Their Workspace

In our first smoke tests, Antigravity CLI did not edit the copy of the repository it was launched in. It walked up the directory tree, found the master task under tasks/, edited the master source directly, and reported success. Its diff in the workspace came back empty, and it contaminated the starting point for every subsequent run. Making tasks/ read-only did not stop it.

The fix was a tree verification guard: before each trial, the runner hashes the git tree of the freshly copied workspace and compares it with a recorded hash of the pristine task. On a mismatch, it restores from a stored tarball, logs the event, and snapshots the tampered copy for forensics. Over the first 160 trials the guard fired 9 times, all attributable to Antigravity, and zero trials started from a contaminated tree.

A working-directory probe showed why. In headless print mode, the CLI attaches to a persistent “default project” whose working directory is a scratch folder under its own config directory, rather than the directory you launched it from. The agent is told about files it cannot see, searches the disk, finds the master copies, and modifies those instead. Adding the launch directory to the trusted-workspace list did not help. Passing --new-project forced it to stay in the target workspace. That single flag accounts for the leap between agy (68.8) and agy-np (95.9) in the public results.

2. Agents Reading the Answer Key

In the pilot runs, agent transcripts showed another surprise: agents were locating and opening the hidden test files and reference solutions stored in a neighboring directory. Nine of sixteen pilot patches reproduced the reference solution down to the docstrings. File permissions do not help when the agent runs as your own user account.

So the hidden material moved to a directory owned by a separate unprivileged Unix user, benchgrader, with permissions 0700. Grading runs as that user through a single sudo rule allowing one script. The agent’s workspace is piped to that script over stdin, the script injects the hidden tests into a private copy, runs them, and returns only the pass counts. The public report’s strict audit found no hidden-material contamination in the final 920-trial matrix.

3. Quotas and Subscription Halts

Three subscriptions ran out during run 5: Antigravity as shipped, Antigravity with the flag, and Codex. The second Antigravity lane paused execution for two hours and sixteen minutes before resuming automatically; Codex required waiting for its reset window.

A local model has no provider quota, though it still depends on the machine staying healthy.

4. The Disk Filled

Late on the second night, the Go build cache reached 12 GB and the home partition hit 100%. The local model server also died during the incident, leaving six trials to fail in under 40 seconds against a dead endpoint. The full disk made the contamination guard fire twice on an empty git tree and destroyed one completed trial’s token record.

Everything affected was re-run or annotated. Two lessons came out of it: FreeToken’s /health endpoint returns OK for several minutes before it can actually answer a completion, so the runner now verifies a real completion before starting a lane; and any multi-day benchmark runner needs a disk-space check in the loop.

5. Our Own Bugs and Tooling Quirks

A leftover line in the runner aborted every trial right after grading for the first 45 minutes of run 1; 68 timing records had to be reconstructed from commit and file timestamps. The quota-wait script compared 0827 as a number in bash, which treated the leading zero as octal. Its quota detector also grepped the whole trial record for “rate limit exceeded”, and one task’s source code happened to contain that exact string, so two healthy lanes deleted passing trials and slept for an hour.

One Sonnet trial was killed by an untraced SIGTERM after writing a complete, fully passing patch. And Crush’s internal database only logged tokens for the final turn in a session; an early draft printed them as 26k in and 0.2k out per trial, an order of magnitude too low. We reconstructed accurate totals by re-tokenizing each trial’s message history with Qwen’s tokenizer (~354k input tokens per trial).

6. The Runner Removed a Required File

The September 16 review correction was wrong. The runner captured patches with a broad bin/** exclusion to avoid compiled binaries from another task. That glob also matched the event-store task’s required bin/es.js command-line file. The file was removed before the patch reached review or replay.

Fifty workspaces still existed. Every one contains a modified bin/es.js, and regenerating those patches without the bad exclusion produced 50 passing test suites out of 50. Their blind-review mean is 4.30 rather than the 2.00 caused by reviewing a truncated patch. Sixty-five older workspaces were already reclaimed, so I cannot recover their real diffs. Their original event-store review scores remain, and the reports identify those rows.

The hidden grader ran against the full workspace and therefore never saw the bug. The review saw a missing file. The replay check confirmed the same missing file. That is a useful reminder: benchmark instrumentation needs the same suspicion as the agents it measures.

Public Benchmark Results: 920 Trials

stab is the mean per-task standard deviation of the pass fraction across five runs. Zero means repeatable results, including repeatable failures. Token figures are per-trial averages; the local Crush figures were reconstructed from transcripts. Hosted Crush token totals are unavailable in the current aggregate, so they are shown as n/a.

Reading the Top of the Table

Opus leads at 98.6, followed by Astra at 98.5. Their quality scores are almost identical: 4.67 and 4.65. Both pass 35 of 40 runs, and both miss the same intervals edge case in all five attempts.

Luna is the only configuration with 40/40 fully passing public runs. Terra passes 39/40 and finishes at a 68-second median; Luna takes 78 seconds. Astra has the highest blind quality among the four Codex configurations, while Sol trails them at 4.03. The Codex composite spread is 2.2 points.

Sonnet is fastest overall at 60 seconds, compared with Opus’s 110 seconds, but its quality score is lower: 4.25 versus 4.67. Both are well below the usability threshold, so speed does not decide their composite ranking. Luna’s perfect correctness still ranks below Opus’s 99.5%, which shows that quality remains part of this trade-off.

There is still a useful lesson in the intervals task. Opus and Astra fixed merge, then rewrote subtract and lost a zero-length boundary case. Sonnet and the local configurations passed all five attempts. Both Terra and Luna also passed all five.

Low Thinking and a Larger Output Cap

We changed two settings together: low reasoning effort and a cap increase from 8,192 to 16,384 tokens. These runs cannot tell us which setting caused the difference.

The blind reviewer preferred the reconfigured patches in both harnesses. Correctness stayed flat for Pi and fell one percentage point for Crush. The patch review found shorter changes with fewer speculative options and less dead code.

The larger cap did not eliminate runaway reasoning. One reconfigured Crush run still used the entire allowance on a thinking block and produced no patch. Pi’s low-thinking runs did not hit the 16k limit across either task set.

Hosted Models in the Same Pi Harness

The public Pi scores are 94.8 for local Qwen, 95.6 for hosted Qwen 27B, and 92.3 for hosted DeepSeek. That is a 3.3-point spread, with hosted Qwen 0.8 points above local Qwen.

Hosted Qwen has the highest quality of these three at 4.03. DeepSeek is faster and cheaper in the recorded runs, but its SQL task shows why a pass count needs explanation.

One DeepSeek SQL run scored zero. The report’s patch replay found that its output differed from a fully passing run by four section headers. The calculations matched after removing those headers, but the prompt required them and the grader located each section by its header. Omitting the labels made all four checks fail.

That is a real output-contract failure. It also tells us much more than saying the model could not do SQL.

Scaling to a Production Monorepo: 460 Trials

The private set uses a production Go monorepo with about 3.7k lines of Go across 15 packages, plus web and mobile applications. Agents received a bug report or feature spec and had to locate the relevant code. The four tasks cover metadata-import debugging, signed-URL security, configuration validation, and a typed job-status refactor. Their hidden suites contain 16, 9, 13, and 8 tests respectively.

The expanded matrix has 23 configurations, each with 20 trials. It includes five Flash-Next configurations, three additional Claude Code configurations, and DeepSeek Harness on Flash-Next and local Qwen.

Private OpenRouter runs pinned providers with fallbacks disabled and requested zero data retention and no data collection. Qwen 27B used Reka fp8; DeepSeek used Reka fp4. The IDs below distinguish them from the unpinned public runs.

Claude Code used 2.1.268 for Opus and Sonnet and 2.1.272 for the three new rows. Antigravity used 1.2.1 rather than the public set’s 1.2.0. Those version changes limit direct comparisons between sets.

The private table was regenerated from the saved records. For pi-ords-reka, those records give 9.3k output tokens per trial; the report’s table currently lists 9.8k. I have used the regenerated figure here. Missing Crush/DeepSeek usage remains n/a in both tables. claude-ft usage is averaged over 18 trials because two timed-out trials had no usage record.

Opus, Antigravity with --new-project, Astra, and Pi with Flash-Next all passed 20/20 private runs. Astra was fastest of those four at 103 seconds. Opus had the highest blind quality at 4.90, followed by Sonnet at 4.80 and Antigravity at 4.75.

Astra’s private quality was 4.05, despite perfect hidden-test results. That difference matters under the revised formula: passing everything does not guarantee the highest composite. Antigravity now ranks second, with no usability penalty at the 450-second threshold.

The three original local-model configurations reach 89.1%-93.2% correctness, depending on the harness. All three swept the typed-status refactor, but none fully passed the metadata-debugging task. Pi low reached 90.9 overall, compared with 87.0 for Crush low and 83.7 for Oh My Pi. Claude Code on the same local endpoint reached 94.7% correctness and 100% compliance, but a 1,810-second median gave it the full 10-point usability penalty. DeepSeek Harness reached 92.7% correctness at a 444-second median.

Hosted Qwen under Pi reaches 97.3% correctness and 4.20 quality, compared with local Pi’s 93.2% and 3.75. Hosted DeepSeek under Pi reaches 97.9% correctness at a 37-second median, though its quality is only slightly higher than local Pi’s: 3.85. These are useful gains without moving to a frontier subscription model.

Does the Harness Ranking Survive a Model Change?

On the public tasks, baseline Pi beats baseline Crush by 0.2 composite points on local Qwen: 93.0 versus 92.8. With hosted DeepSeek, Crush leads 94.1 to 92.3.

The private runs let us compare all three harnesses on both local Qwen and pinned DeepSeek:

The first explanation was that harnesses sending more context benefited more from a hosted model. The follow-up did not support that prediction: Crush sent less context than Pi locally and gained much more.

The largest gains belong to the configurations with the lowest starting scores. That is consistent with convergence toward a ceiling, but it does not establish a cause. Model, endpoint, configuration, and tool differences still matter.

Blind quality makes the comparison easier to inspect:

Oh My Pi’s quality advantage over Pi on private DeepSeek did not repeat on hosted Qwen: both scored 4.20. The private report puts the DeepSeek difference at about 1.93 standard errors, so it is weak evidence for a general preference.

In the September 14 version, I stopped short of calling the harnesses equivalent and found no consistent hosted-model winner between Pi and Oh My Pi. Recovering the Flash-Next patches retracts the September 16 quality correction. The Qwen 27B tie still matters: I would test a harness with the model I intend to use before carrying a preference across models.

Cost still differs substantially. On private hosted Qwen, Oh My Pi used about 1.65 million input tokens per trial, compared with Pi’s 325k. Its 20 trials cost a nominal $7.77, versus Pi’s $2.11, with equal mean quality and lower correctness. Total input tokens count repeated conversation content across requests, so they are not the size of a single context window.

Flash-Next: Five Harnesses on One Model

The three original harnesses remain close on public Flash-Next: Crush scored 97.4, Oh My Pi 97.0, and Pi 96.1. Claude Code joined them at 97.5 and DeepSeek Harness at 95.9. Claude Code was fastest at an 89-second median, passed 39 of 40 runs, and tied Crush on blind quality at 4.25. The one public miss came from the intervals task, which the other four harnesses also found difficult in some runs.

The private set changes the order. Claude Code on Flash-Next passed all 20 runs at 100% correctness and compliance, scoring 99.2. DeepSeek Harness passed 19/20 and scored 98.5. Pi also swept 20/20 at 97.1. The Flash-Next rows are all remote inference on one shared endpoint, so they measure the harness and service together, not what a 177B model does on the RTX 3060.

The five Flash-Next lanes shared an endpoint during part of the run. The reports measured 10%-20% lower output throughput at busy times. That is small compared with the difference between the slowest and fastest harnesses here, but it means the median wall times are not isolated performance measurements.

DeepSeek Harness had two operational surprises. Its Bash sandbox blocks the shared Go build cache, so agents spent turns moving GOCACHE into their workspace. In two trials the redirected cache was included in the captured diff. The harness also sends a full session log and plugin inventory on DeepSeek API requests by default. The report verified that disabling both options removed them from test requests to a local recording endpoint.

Claude Code on Local Qwen and Hosted DeepSeek

Claude Code can talk to any service that implements the Anthropic Messages API. FreeToken provides that API for the local Qwen, and OpenRouter provides it for DeepSeek. For the unpinned public DeepSeek row, setting ANTHROPIC_BASE_URL and passing --model deepseek/deepseek-v4-flash-0731 was enough. Claude Code logged unrecognized_model and still sent the request. I set its context limit explicitly because the CLI otherwise assumes a 200k window for an unknown model and does not compact soon enough.

On public DeepSeek trials, Claude Code scored 85.7, behind Crush at 94.1 and Pi at 92.3. Five of 40 runs produced no patch even though the agent read the repository and exited successfully. The same endpoint produced tool calls in direct probes, so these were counted as model-and-harness results rather than infrastructure failures. Claude Code used 306k input tokens and a 192-second median per trial, compared with Pi’s 100k and 98 seconds on the same tasks and model.

The private DeepSeek results made the same failure mode clearer. The pinned Claude Code lane had five empty patches in 20 trials and scored 84.5; the unpinned lane had four and scored 87.1. The comparable pinned Pi, Crush, and Oh My Pi rows scored 94.6, 94.8, and 96.0 with no empty patches. Pinning gave Claude Code a 92-second median instead of 286 seconds, but it cost about three times as much per trial and did not improve correctness.

The local Qwen row tells a different story. Claude Code had the highest local correctness, 94.7%, and perfect compliance. It was still last by composite because every turn re-prefilled almost the entire prompt, leaving it with a 1,810-second median and the full usability penalty. The report traces this to how FreeToken moves Claude Code’s changing trailing system message into the leading system block, which defeats its prefix cache. That is a server-side cache defect, not evidence that local Qwen needs thirty minutes per task.

The Endpoint and Tools Are Part of the Configuration

The first private DeepSeek endpoint attempt, OpenInference fp8, repeatedly hit the turn watchdog without producing usable edits. Switching to Reka fp4 made the same signed-URL task complete in 19 turns and 44 seconds with all nine tests passing.

The report checked repeated file reads and found identical tool output, despite the model claiming it was garbled. That supports an endpoint-specific problem; it does not isolate its technical cause. These two deployments do not establish that lower precision is generally better.

Web access was another difference. Private Crush on DeepSeek made 94 web-tool calls, all on the metadata-import task. The audit found public-format research and no project identifiers in the queries checked. That configuration achieved one fully passing run on the task, but five attempts are too few to attribute the improvement to search.

One public Crush/DeepSeek trial made five Sourcegraph queries that returned no results. Audited Claude, Codex, Pi, and Oh My Pi logs showed no web use. Two gaps remain: the baseline and low-thinking Crush workspaces were reclaimed before the audit, and Antigravity’s saved transcripts have no tool log. We cannot claim that every configuration had identical external-tool conditions.

What I Would Not Over-read

Five runs per task expose repeatable mistakes and occasional failures, but they do not make a two-point difference decisive. The tasks are few, the quality judge is a single model, and several comparisons changed more than one variable. CLI versions auto-updated; the local tuning changed two settings together; public OpenRouter endpoints were unpinned. Crush changed from 0.92.0 locally to 0.93.1 on DeepSeek and 0.94.1 on Flash-Next, so those model swaps also include a harness-version change.

The final private Pi-low row is a complete 20-trial rerun under randomly named work roots. An earlier trial had worked inside a sibling’s already-completed workspace. Fixing that artifact changed the result from 13/20 to 11/20 fully passing runs while mean correctness rose slightly, from 92.9% to 93.2%. Counting only the repaired trial would miss the variance in the other nineteen.

One private Antigravity run returned success with an empty response and patch and was rerun. Its final row therefore includes 19 first attempts and one replacement. A stray Sonnet smoke test was removed so that every private row has 20 trials. Hidden-test defects were also corrected and affected patches regraded; the reports retain those incidents.

Cloud lanes sometimes ran concurrently, while local lanes shared one GPU and ran serially. Some early timings were reconstructed, and one private Opus usage record was imputed after the disk filled. The timing and token columns deserve those caveats.

The Flash-Next run exposed another launcher bug: an orphaned shell started a duplicate public lane, which wrote into result directories already used by a live lane. We checked the recorded working directory, transcript directory, workspace link, and patch for every Flash-Next trial. Five public records were archived and rerun; all 60 private records passed the audit. The final counts exclude the superseded records.

Review depth also varied. The September 16 command-line check appeared to show a missing requirement, but the runner had removed that command-line file from the recorded patches. The September 17 re-review used regenerated diffs for the 50 surviving workspaces. Sixty-five reclaimed workspaces cannot be re-reviewed, so their original scores remain. Anonymizing authors does not make a review or its input infallible.

The unpinned Claude Code/DeepSeek lanes also differ from the pinned private lanes in provider routing and data policy. They were intentionally run without the private set’s pin or zero-retention request to test the native OpenRouter Messages endpoint. Their results are a blend across providers and are not a private-code deployment recommendation.

The scoreboard can be regenerated from existing public records without launching new agent trials:

cd ~/projects/ai-bench
PYTHONDONTWRITEBYTECODE=1 python3 tables.py
PYTHONDONTWRITEBYTECODE=1 python3 tables.py --agents claude,codex,crush,agy

Wrapping Up

The local 35B setup completed 37 of 40 public runs under Pi low and all five private status-refactor runs. That gives me a concrete reason to keep a local agent available. The harder debugging task also showed its limits: none of the three local configurations completed it perfectly.

The hosted comparisons add another option. Pi with hosted Qwen improved both correctness and blind quality on the private tasks. Pi with pinned DeepSeek was much faster. Flash-Next under Pi now joins Opus, Antigravity with the workspace flag, Astra, and Claude Code on Flash-Next in passing the entire private set.

Claude Code and DeepSeek Harness on other models are a reminder that API compatibility does not make agent behaviour portable. Claude Code achieved the best private local-Qwen correctness but was held back by a cache defect; on DeepSeek it produced empty patches far more often than the other harnesses. I would read the individual columns, inspect a few patches, and verify the benchmark captures every required file before picking a configuration.

In Part 3, we explore fitting a 177B MoE model onto the same desktop hardware. The cluster runs show what the model can do with faster serving. Making that useful on a 12 GB card remains a separate problem.