Once an LLM agent goes from "notebook experiment" to "thing running in production," three questions stop being optional: what is it actually costing per call, is it safe from someone hiding instructions inside its inputs, and is the quality holding up as the underlying model or the prompts change. Most agent projects answer none of these until something goes wrong — a surprise bill, a jailbreak that made it into a screenshot, a quality regression nobody measured because nobody was measuring quality in the first place.
gcp-agent-cost-observability-platform is the answer I built: an observability and cost-governance platform for a real 3-agent (planner → tool-using worker → reviewer) Gemini pipeline on Cloud Run. Every call is logged to BigQuery with token counts, computed cost, latency, and agent/step identity; input and output guardrails catch prompt injection and data exfiltration; and a scheduled golden-set evaluation scores quality with an LLM-as-judge. It's the token/quality/safety complement to its sibling repo, gcp-finops-cost-anomaly-detector — that one watches infrastructure and cloud spend broadly, this one watches LLM spend specifically. It deployed live in europe-west1 against the Vertex AI global endpoint, and a from-scratch teardown-then-rebuild passed its 18-check test suite 18 of 18.
A real 3-agent pipeline, not a single prompt
The pipeline is planner → tool-using worker → reviewer: the planner breaks down the request, the worker actually calls tools against a synthetic spend-data fixture, and the reviewer checks the worker's answer against tool ground truth rather than re-deriving the answer from scratch. That last detail is the structural mitigation behind ADR 0001's trade-off: the pipeline runs on gemini-2.5-flash, a fast, cheap model with weaker reasoning than a pro-tier alternative, and the reviewer's job of comparing against ground truth rather than independently re-solving the problem is what makes that weaker-reasoning trade-off safe to take.
Every call, accounted for
A logging middleware wraps every single Gemini call — prompt, response, input/output/thinking token counts, cost computed from published Vertex AI pricing, latency, and which agent and step made the call — and writes it to a BigQuery llm_calls table. The live test confirms this isn't aspirational: a real /ask call produces at least three logged rows with nonzero cost, one per agent in the pipeline. That per-step granularity is what actually makes a cost investigation possible later — "the pipeline cost $0.04" is a number; "the reviewer step cost 3x the planner step because it re-reads the full tool output" is an insight you can act on.
Guardrails that stop spend before it happens
A heuristic input guardrail runs before any call reaches Gemini — a matched prompt-injection attempt is blocked with zero Vertex AI spend, which the live test confirms directly: a canonical injection attempt produces zero logged Gemini calls, not a logged call with a refused answer. That distinction matters for cost governance specifically, not just safety — a guardrail that blocks after the model call still spent the tokens.
test §4: canonical prompt-injection attempt
→ blocked at input stage, 0 logged Gemini calls, $0 spendOn the output side, a second guardrail checks Vertex AI's own safety ratings plus an explicit secret-shaped-string rubric against the final answer, catching output-stage exfiltration risk that an input-only check would miss entirely. ADR 0003 is candid about the limit here: a heuristic, regex-based input guardrail catches known injection phrasings, not novel ones — which is precisely why there's an independent output guardrail as a backstop rather than betting everything on catching bad input before it's processed.
Quality measured, not assumed
A separate Cloud Run Job runs a fixed 8-question golden set against the deployed service — not a local test double — and scores each answer with an LLM-as-judge, writing results to a BigQuery eval_scores table over time. In the live run behind this post, all 8 questions scored 5/5/5 across every judged dimension. To give the resulting trend chart history on day one, 20 days of clearly-labelled synthetic evaluation data (is_synthetic = true on every row) seed the backfill — and the live test explicitly asserts is_synthetic is never null on any row, so a synthetic backfill and a genuine live-judged run can never blur together by accident months later when someone's looking at the trend line and needs to know which is which.
Six decisions
| Decision | Trade-off |
|---|---|
gemini-2.5-flash on the global endpoint, verified live before writing any pipeline code | Weaker reasoning than pro-tier — mitigated by the reviewer comparing to tool ground truth, not re-deriving answers |
| Synthetic fixture for the agent's task data, synthetic seed history for the eval trend | Proves the method, not a genuine multi-week quality trend |
| Heuristic (regex) input guardrail, not an LLM classifier | Limited to known injection phrasings — the output guardrail is the real backstop |
| Threshold alert policies, not a trailing z-score baseline | No adaptive baseline — a single session has no history to build one from yet |
| No GitHub-Actions Workload Identity Federation; CI is lint/validate/scan only | CI never touches the live project, by design |
| Authenticated-only Cloud Run service; a Job, not a Function, for evaluation | Requires an ID token to demo — deliberately no bare public URL |
Eight real bugs, each with a name
Getting this live took eight bugs a design review would never have caught: a quota-project gotcha in how the Vertex AI SDK resolves billing; log-metric label mismatches that silently broke alert routing; a Cloud Monitoring restriction where DISTRIBUTION-type metrics can't be used the same way as scalar ones in an alert policy; Google's own front end silently reserving the /healthz path (the same gotcha that shows up in the serverless-orders build); two separate Vertex AI SDK host/version bugs; a rule about human-identity versus service-account ID token audiences that isn't obvious until you hit it; and log-based-metric propagation lag that made an alert look broken when it was just slow to catch up. Each one is documented with root cause and fix in runbook 03 — the kind of list that's only honest if you actually hit every item on it live.
What's deliberately not proven here
Golden-set evaluation quality is proven against 8 fixed questions over a small synthetic fixture — a real deployment needs a larger, evolving, human-reviewed benchmark, and this repo says so rather than implying broader coverage. The input guardrail is curated heuristics, not a trained classifier, so novel injection phrasing can slip past it to the output backstop. Alert policies are simple thresholds, not adaptive baselines. There's no VPC Service Controls perimeter around BigQuery, no rate limiting beyond Cloud Run's own concurrency ceiling, and indirect prompt injection via tool output plus live exfiltration provocation weren't adversarially tested end to end — both are named explicitly in the threat model's known-gaps section rather than left implicit.
Try it yourself
git clone https://github.com/soodrajesh/gcp-agent-cost-observability-platform
cd gcp-agent-cost-observability-platform
gcloud config set project <your-project>
./scripts/up.sh # infra → build both images → deploy service+job+scheduler → seed history → golden set → live tests
./scripts/down.sh # delete everything, including the BigQuery dataset and its dataWell under $0.01 in Vertex AI spend for a full build-test-teardown session — gemini-2.5-flash bills at $0.30 per million input tokens and $2.50 per million output tokens, and a full /ask call plus an 8-question golden-set run together total a few thousand tokens. Cloud Run, BigQuery, Monitoring, and Scheduler usage don't come close to their free tiers, and a Terraform-managed Billing Budget bounds total spend as a backstop regardless of what the pipeline itself does.
📢 Have questions or feedback? Drop a comment below or connect with me on Twitter/X@spysood!