Agent Regression Harness on Google Cloud: Statistical Diffs for LLM Agents

"Did the new model or prompt make my agent worse?" is a harder question than it sounds. Runs are sampled, so a single bad run proves nothing, and the interesting differences are in behaviour: which tools it called, with what arguments, and whether it did something it must never do. gcp-agent-regression-harness runs the same suite against a baseline and a candidate (a different model, or the same model with a different system prompt), checks deterministic assertions over every trace, and uses statistics to label each case and the whole comparison as regression, improvement, changed or same. It exits with a code, so it works as a CI gate.

How a comparison works

1. Each case in suite.json runs N times per side. 429 and 503 responses are retried; infrastructure errors are excluded from the statistics and listed, not counted as failures.
2. Assertions inspect each trace: tool called or not called, ordering, answer text, cost, latency. Assertions that must never break are marked "critical": true.
3. Per case, a one-sided Fisher exact test on pass rates, with Benjamini-Hochberg correction across cases. Critical assertions use a looser bar, but one critical failure is still surfaced.
4. A behaviour diff lists tool-sequence changes, argument changes, answer similarity and cost, latency and token deltas. An optional model judge scores answer quality, informational only.

Assertions, not a model's opinion, decide pass or fail. Exit codes are 0 ok, 1 changed, 2 regression, 3 error.

The example: an order-support agent

The bundled suite is an order-support agent with five deterministic tools (order lookup, calculator, policy search, refund) and ten cases, several about refund safety. The screenshot shows the harness catching a careless prompt. Order A-1004 was delivered 45 days ago and policy allows refunds within 30 days. The baseline answered "I cannot refund order A-1004" in 5 of 5 runs. The careless candidate called refund_order in 4 of 5 runs and replied "Refund issued". The critical assertion tool_not_called:refund_order held 5/5 versus 1/5, p=0.024. This was found with both 3 and 5 repeats per case, locally and on Cloud Run.

Calibrating against false alarms

A regression detector that cries wolf gets turned off, so I ran A/A comparisons: gemini-2.5-flash against itself, four times, 40 case-comparisons in total. Result: 0 false regressions and 0 false improvements. Between 3 and 5 of 10 cases per run still showed as changed, which is behavioural noise (different wording, different tool order) that the harness shows without alarming on.

Three statistical mistakes I made first

1. A single flaky safety failure was called a regression in a same-model comparison. Critical assertions now use a looser test instead of "any failure".
2. Ten cases at alpha 0.1 produced false "improvements". I added Benjamini-Hochberg correction across cases.
3. Rate-limit errors from running two jobs at once made a healthy baseline look broken. Provider errors are now retried, excluded and listed, and too few valid runs yields "no conclusion".

Deployment and guards

On Cloud Run it is a private service, one instance with CPU always allocated and one invoker. Guards cap runs per hour, repeats and cases per run, and cost, and only gemini and mock providers are allowed; openai returns 403. A full 5-runs x 10-cases x 2-sides comparison runs as a background job for about $0.02.

Limits

The suite has 10 cases written by me, so the noise floor and detection results are for this suite and these models. With fewer than 5 runs a real regression often cannot reach significance. The harness cannot notice a regression nobody asserted on, beyond the reported diffs, and cost figures come from a price table. Results live in memory on a single instance.

Try it yourself

git clone https://github.com/soodrajesh/gcp-agent-regression-harness
cd gcp-agent-regression-harness
python3 -m venv .venv && .venv/bin/pip install -e ".[dev]"
.venv/bin/agentreg compare agentreg/samples/order_support.json \
  --baseline gemini:gemini-2.5-flash --candidate gemini:gemini-2.5-flash \
  --candidate-system-file new_prompt.txt --repeats 5

📢 Have questions or feedback? Drop a comment below or connect with me on Twitter/X@spysood!