01 Overview
A homemade bake-off that runs 14 AI coding-agent CLIs against one fixed task — build and deploy a small "Word Counter" Node.js web app into my real 50-plus-service homeserver stack — and scores each attempt 0–100 on whether the container actually builds, boots, and answers correctly, how closely it followed the rules, and an LLM judge's read on plan and code quality. It exists to answer one practical question: which agent, at which price tier, should my self-building deploy pipeline reach for?
It grew in three layers of increasing rigor, sharing one target task and one set of executor adapters.
Speed pass
The simplest harness: run each agent once, time it, count the files it created, and print a markdown table. A quick "does this even work" pass.
Matrix + judge
The full agent × plan-model × research-model × deep-mode comparison, with a hybrid claude-haiku judge and a self-contained HTML report with sortable, filterable tables.
Deploy-grade scoring
Actually docker compose builds and boots each agent's output, then black-box-tests the live HTTP API with exact-value assertions — a 0–100 score per (agent × price tier).
02 Why I built it
My wider home dev pipeline (the pipeline-dash orchestrator, credited under the platform work) can hand a build task to any of a dozen interchangeable coding agents. That raised an obvious question I had no data for: which one, and at which model tier, is actually worth using for a given job? I needed measurements, not vibes.
So the harness normalizes every agent across four hardcoded price tiers — free local coder ($0), claude-haiku (~$0.02/run), claude-sonnet (~$0.20), claude-opus (~$1.50) — and its analyzer emits named "speed / quality / cheap" mode configs that feed straight back into the pipeline's mode selector. A second push came from vendor churn: when one agent went pay-walled for headless use, a one-day research sprint added six replacement agents, and the harness is how I told which of the newcomers were any good.
03 What I built & how it works
Seed the task once, fan it across the matrix, defend the host, then score what survived.
Fig. 1 — the harness drives a separate orchestrator over REST; the two scorers share one target app but weigh it differently (infra-and-judge vs. functional-HTTP).
- Seed once — run research and planning a single time and reuse the same plan for every combination, so the only variable under test is the execute phase.
- Dispatch — POST each (agent × tier) to the pipeline over REST, auto-approve its gates for a fully unattended run, poll to completion, and resume-skip anything already scored.
- Execute — each of 17 wrappers (14 benchmarked) shims one third-party agent CLI into a uniform plan-in → files-out contract, reading its price tier from a shared mode file.
- Defend — snapshot the 50-plus-service compose before the run; a graft step restores it if the agent overwrites it (see below — this is the hard part).
- Score — build and boot the container, probe the live HTTP API with exact-value assertions, and fold infra checks + rule compliance + a claude-haiku plan/code judge into one 0–100 number.
- Report — emit a self-contained HTML dashboard plus a CSV, then rank agents by score, cost-per-score, and speed to pick the pipeline's default tier.
04 🛠 Skills & tech used
05 Notable challenges & decisions
The interesting engineering wasn't the scoring — it was making an untrusted, code-writing agent safe to run against a live production stack.
The day an agent flattened the production compose
One executor overwrote my 52-service homeserver docker-compose.yml with a 48-line stub, taking real services down. The fix wasn't a better prompt — it was treating every agent as untrusted: snapshot the compose before each run, then a compose-graft.py step notices when the service count drops below half of the snapshot and grafts the new blocks back onto the original. On top of that sit a contaminant blacklist that strips services agents like to hallucinate (postgres, redis, grafana…) and a four-case YAML auto-repair loop that restores from backup and re-injects the test service. Most of the harness's code is defense, not scoring.
Talking to Amazon Q without its desktop app
The kiro agent normally needs a GUI daemon. To run it headless, its wrapper reads the OIDC bearer token out of kiro-cli's own SQLite store, POSTs directly to the Amazon Q endpoint, and hand-parses the AWS binary event-stream response with struct.unpack — then falls back to LiteLLM if any of that fails. Several wrappers needed this kind of spelunking; each agent came with its own auth scheme and its own favorite failure mode to counter.
14 agents behind one plan-in → files-out contract
Each wrapper shims a different third-party CLI into the same interface and maps one shared four-tier mode file onto that agent's model selection — a LiteLLM alias, a direct Anthropic model name, aider's architect/editor pair, or an OpenHands config patch-and-restore. Getting there meant surviving a lot of Bash string handling: a brace-parse bug corrupting JSON, backticks in a double-quoted prompt executing as shell commands, ANSI codes breaking an embedded Python literal — six harness bugs found and fixed in a single session before any number could be trusted.
Isolating the one thing under test
Research and planning run once, and that same plan feeds every combination, so only the execute phase varies. Scores normalize across four price tiers (free coder up to ~$1.50/run), and the analyzer reports cost-per-score and score-per-second — the numbers that actually decide which agent the pipeline defaults to. When a mid-sweep review found the judge deducting unfairly in three spots, I fixed those checks and re-ran the affected cells rather than trust the first pass.
The theme, stated plainly. A benchmark of code-writing agents is mostly a containment problem. Snapshot, graft, blacklist, redirect, restore — the scoring is the easy 20%; keeping the host intact across hundreds of runs was the other 80%.
06 Results
In the 56-run price-tier sweep (14 agents × 4 tiers) the average score was 94.7/100, and eight agents — goose, codex, continue, forgecode, openinterp, kilo, openhands, and kiro — scored a perfect 100 at every tier; opencode, claude-code, oatmeal, and aider-auto landed in the mid-90s. The reliable set the pipeline now reaches for is kiro, kilo, openhands, codex, goose, and opencode. cline and autogen were the flaky ones — autogen's free-coder tier and cline's haiku tier each produced zero files and scored 2/100. Fastest clean run was kiro at 46 seconds. The complete theoretical matrix is 1,152 runs (the script estimates 6–12 days to run it in full); the weekly cron instead ran a smaller 192-combination tier for six consecutive Saturdays.
Sources: exec-quality-results.csv and the 2026-03-08 sweep summary. Numbers are from my own homeserver runs, not an external benchmark.
07 Media
No polished screenshots to share — the harness is a terminal-and-report tool. These are placeholders describing the real surfaces.
winner banner · per-agent bar charts · sortable 15-column results table
report.html): stat cards, a winner spotlight, avg-score bar charts by agent and plan-model, and a sortable, filterable results grid — zero external assets, per my homeserver no-CDN rule.█░ score bars · cost-efficiency · 14×4 score matrix
12 rival compose fragments · install.sh full of markdown headings
08 Honest status
This is a harness, not a product — a set of shell and Python scripts on my homeserver plus, for a while, a weekly Saturday cron that re-ran a 192-combination matrix. It is dormant now: the cron entry was removed around mid-April 2026 and the test container is cleaned up between runs by design, but the scripts still work on demand. Two honest limits. First, the six weekly full-matrix campaigns were archived (185 MB of runs) but I never fully mined them — the findings above come from the 56-run tier sweep I did analyze. Second, a fresh benchmark costs real money at the higher tiers and touches the production compose file, so it is not something I leave running unattended. There is no live demo here; this page is a write-up.