The bug that made my evaluation meaningless, and the fingerprint that found it

I spent a week building Agent Change Gate, a tool that measures agent prompt changes. Then I found out it had been measuring nothing.

This is the write-up for Agent Change Gate, built on the TrueForge harness for the WeMakeDevs hackathon. The part worth writing about is the two hours where the tool was reporting numbers about specs that were never in effect.

Two and a half minutes of it running: the fan-out across subagents, the sandbox doing the statistics, the approval gate stopping three writes one at a time, and the pull request that came out of it.

The job

Changing an agent's prompt is a deploy. Rewrite a line of instructions, ship it, and nothing turns red, because nothing is measuring. If the bot starts mislabelling things a week later, there is nothing linking that back to the edit.

So: re-run a frozen scenario set against the current spec and the candidate, tell the difference between a real regression and model noise, print the comparison, and stop. A human decides whether the change reaches GitHub.

Sixteen issues from microsoft/vscode, split eight bug and eight feature-request, with the maintainers' own labels as ground truth. Scoring is deterministic string checking, never a model judging a model. A grader that disagrees with itself between runs makes the whole comparison worthless.

What the harness does

The judging line for this hackathon says the harness has to be doing the work "rather than sitting underneath a thin wrapper", and lists what that looks like: real tools through MCP, code running in a sandbox, a pause before anything irreversible, work handed to subagents, a session that survives a reconnect.

My first version had three of those and faked a fourth.

Subagents. Scenarios go out in batches. Each batch is one turn, and the coordinator hands every scenario to its own subagent. Fan-out, scheduling, context isolation: all the harness's. Batches run one after another, so there is no concurrency in my code at all.

The version before this drove a ThreadPoolExecutor over one session per scenario, which is the thin-wrapper shape the organisers warn about: the harness reduced to an HTTP endpoint being called concurrently.

Code run in a sandbox. The statistical read of the results (Wilson intervals per arm, whether they overlap, which scenarios diverge) is code the agent writes and runs in its sandbox. This is not TrueForge's Code Mode, which is generated Python calling MCP tools through an in-sandbox mcp_client. The pass counts are already in memory and routing them through a tool would buy nothing, so borrowing the term would have been a lie of convenience. On the first run it reached for scipy, got ModuleNotFoundError, rewrote against the standard library, and ran. The verdict deliberately stays out of the sandbox: fixed, broken and flaky are settled by deterministic checks before the matrix is handed over, so the same outputs always score the same.

Real tools. Landing a change goes through the GitHub MCP server: branch, commit, pull request. Calling the REST API directly would have been easier and would have bypassed the thing that matters.

Approval. Landing a change is three separate tool calls, and every one of them comes back for its own decision. The part I like: I did not write that policy. require_approval_for_tools: ["@write", "@destructive"] is the factory default, and GitHub's MCP server is what annotates these calls as writes. My contribution was declining to turn it off.

Reconnect. A batch takes minutes and the turn keeps running server-side, so a dropped connection resumes from the last sequence id it saw.

What broke

The one that mattered

I wrote this in a docstring, and never checked it (translated from the original Chinese, removed in 5050e9b):

Subagents inherit the same instructions, so what gets measured is still the spec under test.

Qodo's review flagged something adjacent on that pull request: if the coordinator's "hand every item to a subagent, never answer one yourself" rules are inherited, workers might fan out again instead of answering. Reasonable. Also based on a premise nobody had tested.

So I tested it. Give the spec instructions that demand a fingerprint token at the end of every reply, then fan out:

coordinator output : "...ZX9QQ"
subagent 1 output  : no fingerprint
subagent 2 output  : no fingerprint

Subagents inherit nothing. Which is worse than the recursion Qodo worried about: both arms of my comparison were running with neither spec in effect. The baseline and the candidate differ only in their instructions, and the instructions were never reaching the thing doing the work.

The report from that period looks healthy. no change outside noise, token cost 1.01x. Of course it does. It was comparing an empty spec against an empty spec, sixteen scenarios at a time, three times each, and printing a verdict about it.

A dynamic subagent takes no instructions of its own (AgentInfo is name, input and model), so the fix is that the spec travels with each item and the coordinator carries dispatch rules only. Same fingerprint check afterwards: every subagent returns the token.

That change moved the numbers immediately. Token cost went from 1.01x to 1.07x on the three-repeat runs of the day, because the candidate's longer instructions were finally being paid for.

The one about placement

Putting the spec in the item text is not the same as putting it in the instructions field, which is where it lives in production. That is a real difference, so I measured it rather than waving it through. Same spec, same scenarios, paired per run, ninety-six pairs:

instructions field : 29/32   59/64
inside the prompt  : 28/32   57/64
disagreeing pairs  : 1       2

All three disagreements are the same scenario, one that is unstable under both placements. The other fifteen never disagreed. Not proof of equivalence, and I do not claim it is. Enough to say the fan-out is not visibly costing accuracy, with a script anyone can re-run.

The one where I fixed a bug and introduced its twin

A dropped connection before turn.created left no turn id, so a turn that kept running server-side was paid for and thrown away. The session lists its turns, so it is recoverable. My first fix took data[0] from that listing.

The listing is oldest-first. data[0] is the oldest turn on the session, not the newest. Three turns created in order confirmed it. The test I wrote alongside the fix had a single-element list, so it could not have caught the mistake.

Qodo then pointed out that even taking the newest is wrong: nothing tied it to the turn being recovered, so a concurrent turn on the same session would be resumed instead. No path in the codebase does that today, but "nothing does that today" is not a reason to leave a lookup that picks by position. It now matches on the input the turn records.

The one I found on the last day

Hours before the deadline I ran the test suite on a machine with no TrueForge server running, which is the machine a judge has. Three tests failed. On my own machine, with the harness up, the same suite had been green for days.

The three had been reaching the network without meaning to. _request turned HTTPError into a HarnessError and let URLError straight through, and the two callers written to degrade gracefully, _find_turn and describe_call, both catch HarnessError. A connection-level failure skipped their guard entirely, so looking up the tool name behind a pending approval would take down the approval gate mid-write-back instead of falling back to the raw call id.

Same shape as the fingerprint bug, one layer down. A test that quietly depends on a service you happen to have running is green for a reason that has nothing to do with your code. The fix is four lines; finding it needed an environment I did not have until I went looking for one.

The smaller ones

• The SSE stream sends data: first and id: after. My parser emitted each event as soon as it saw data:, so every event carried the previous event's sequence number. The resume cursor was one short, which the exclusive cursor turned into a replayed event rather than a lost one. Nothing looked broken.
• HTTPError subclasses URLError. After adding reconnect-on-drop, a 400 from a bad manifest fell into the retry path, failed again, and surfaced as "this batch produced no output" with no error anywhere.
• A harness outage was being scored as wrong answers. One failed batch read as "this change broke sixteen scenarios". Failed runs now leave the denominator and the scenario is marked incomplete.

What it says about a real change

The candidate spec spells out the classification criteria the baseline leaves implicit. Five runs per scenario, both specs. The whole report is PR #23, opened by the gate itself:

Verdict: no change outside noise.
0 fixed, 0 broken, 1 flaky, 0 unproven, 0 incomplete.  Token cost 1.02x of baseline.

passed        baseline 71/80 (89%)    candidate 74/80 (92%)
tokens        baseline 312,661        candidate 317,791
issue-332082  baseline 1/5            candidate 4/5            flaky

That is a decision, not a scoreboard. On this run the change cost 1.02x the baseline, and two other five-repeat runs cost 1.08x and 1.09x, so the size of the increase is itself unstable. The one scenario that moved is one the baseline cannot answer consistently either.

And the repository has the other reading of the same change sitting next to it. PR #8 was opened by the gate itself from a one-run-per-arm evaluation:

Verdict: improvement. 1 fixed, 0 broken, 0 flaky. Token cost 0.94x of baseline.

Same two specs. Same sixteen scenarios. One run per arm calls it an improvement that also saves tokens. Five runs call it no change outside noise, at 1.02x the token cost. The single run got both the direction and the sign of the cost wrong, and nothing about that report reads as uncertain.

Both pull requests are still open, because the gate's verdict on this change is that it is not worth paying for. Merging them would make the report a decoration.

What I would tell myself on day one

"Subagents inherit the parent's instructions" was one experiment away from being tested, and it silently invalidated every number the tool produced until somebody's review made me look.

The fingerprint trick generalises: when a system is supposed to carry your configuration somewhere you cannot see, put a token in it that can only have come from there. If the token does not arrive, the configuration did not either, whatever the output looks like.

Everything here re-runs. probe/ has the standalone scripts behind every claim: the fan-out with its fingerprint check, the reconnect that cuts a live connection, the sandbox analyst, and the placement comparison. The suite is at eighty-five passing tests. One thing is not pinned by a test at all: the approval deadlock is closed structurally, because the window is a few instructions wide and 200 paired rounds never reproduced it against the old code. The README says so rather than shipping a test that would pass on both versions.

Check it yourself

• Code: github.com/cyh7789/agent-change-gate. uvx --python 3.11 --from pytest pytest -q tests needs no harness running.
• Demo: a two and a half minute run, from the evaluation through the approval gate to the pull request.
• The two verdicts: PR #23 (five runs per arm) against PR #8 (one run per arm). Both were opened by the gate itself and both are still open.
• The review trail: PR #10 carries the disposition of all 28 Qodo findings, including the two I judged not to be defects.
• The hackathon: The Agent Harness Hackathon, run by WeMakeDevs with TrueFoundry and Qodo.