Before a change ships, how risky is it, and what should the rollout look like? gcp-deploy-risk-scorer scores a change (a git diff, or a commit in a repo) with a model trained on real commit history, adds deterministic flags that no model can lower, and turns the result into rollout gates: first canary percentage, bake time, reviewers, deploy window and rollback plan. A model may explain the result, but it never sees code or file names and cannot change the band.
Why two layers
The history model ranks ordinary changes by how they historically turned out. The flags catch the rare change that is catastrophic however small it looks. The first screenshot is the example: a two-line migration (DROP TABLE legacy_users; ALTER TABLE orders DROP COLUMN notes;) is in the 1st percentile of risk by history, because in the training data migration files and brand-new files are rarely followed by fixes, and the change is tiny. The model reads it as low. Flag C001 (destructive SQL in a migration) raises it to critical, and P003 (migration files changed) adds a medium. The final band is the maximum of the model band and the highest flag. The gates for critical: 5% first canary, 60 minutes bake, 2 reviewers, a rollback plan, and a scheduled window with the owner on call.
The pipeline
- Features: size, spread, test ratio, file categories (infra, IAM, migrations, dependencies, CI, config), deletes, binaries, message hints and time of week. With a repo it adds how often the touched files changed and were fixed before, and author familiarity, using history only, never the future.
- Model: logistic regression with pure-Python inference, predicting the probability the change is followed by a fix or revert. Bands come from rank among the 16,730 training changes: low below the median, medium below p80, high below p95, critical above.
- Flags: path rules (Terraform deletes, IAM infra, migrations, CI, auth code without tests, very large) and content rules on added lines (DROP and TRUNCATE in migrations,
0.0.0.0/0andallUsers,deletion_protection=false, keys in the diff, dependency major bumps). - Gates for the band, with notes for weekends, Fridays and nights.
- An optional model note from structured metadata, rejected if it has a link, a number not in the score, or argues the band down.
How good is the model, honestly
The label is a proxy, not a measured incident: a commit is positive if a later commit whose message says fix, revert, hotfix or regression touches one of the same files within 30 commits and 14 days. I trained on five public repositories (flask, requests, click, werkzeug, httpx) with 16,730 labelled commits.
| Diff-only (hosted) | Full (adds history) | |
|---|---|---|
| Chronological holdout AUC (train earliest 70%, test latest 30%, 5,022 commits) | 0.655 | 0.667 |
| Lift in the top 10% (base rate 20.8%) | 1.58x | 1.65x |
| Brier score (base rate alone: 0.1827) | 0.1771 | 0.1735 |
| Leave-one-repo-out AUC range | 0.63 to 0.74 | 0.66 to 0.72 |
| Baseline: lines changed alone | 0.60 to 0.71 | |
That is real but modest skill. It beats "bigger diffs are riskier" in every split, and it is a triage signal, not an oracle. It is weakest on requests (AUC 0.53 on the time-ordered split, diff-only). What it learned: files changed often before and files with a fix history are riskier, authors who know the files are safer, docs-only changes are safer, and a commit message saying "fix" raises the score, because in this data fixes are often followed by more fixes. I deliberately removed repository age as a feature after it improved headline numbers but turned out to be a time-trend artifact.
Verification and deployment
24 offline tests cover diff parsing, features with no look-ahead, labels and horizons, the model (calibrated; numpy and pure-Python training agree), every flag, floors the model cannot lower, gates, note validation, and the guarantee that no code or paths go to a model. In the note eval (mock, gemini-2.5-flash, gpt-5-nano) the band never changed and one gemini note was rejected. On Cloud Run the service account has only aiplatform.user and logging.logWriter: it reads no project data, the diff arrives in the request, is parsed in memory and is never stored. The deployed service scored the destructive migration critical; unauthenticated and openai requests return 403. In CI the exit code (0 low, 1 medium, 2 high, 3 critical) can fail the job or require approval.
Limits
The label over-counts normal iteration and under-counts incidents fixed elsewhere, so it does not measure outages. The bundled model is trained on five public Python libraries; retrain on your own history with deployrisk train and check calibration before trusting the probabilities. The flags are patterns and miss destructive changes expressed another way, such as an ORM migration that drops a column. Gates are a policy table by band, not a measured reduction in incidents, and message-based features are easy to game.
Try it yourself
git clone https://github.com/soodrajesh/gcp-deploy-risk-scorer
cd gcp-deploy-risk-scorer
git diff main... | .venv/bin/deployrisk score --diff - --message "refactor billing"
.venv/bin/deployrisk score --repo . --rev HEAD # full model with this repo's history📢 Have questions or feedback? Drop a comment below or connect with me on Twitter/X@spysood!