GKE Platform Engineering: Binary Authorization & Canary

Most "GKE demos" stop at a cluster with a Deployment and a Service. That proves Kubernetes works. It doesn't prove the cluster is safe to hand to five other teams. I wanted to build the thing an actual platform team owns: a cluster where nobody can run an unsigned image, nobody can break the org's rules even by accident, deployments roll out gradually and roll back on their own, and the edge is protected by a WAF — all provable from the command line, not just described in a README.

So I built gcp-gke-platform-engineering: a GKE Autopilot cluster with a signed-image-only supply chain, GitOps for the platform layer, progressive delivery with automatic canary and rollback, a WAF-protected Gateway, and golden-signal observability. Everything is provisioned by one script and destroyed by one script. It went live in europe-west1 on 2026-09-25/26, and its own test suite — 28 checks against the running cluster — passed 28 of 28.

Three gates, not one

The core idea is that "policy enforcement" shouldn't be a single YAML file you hope catches everything. Every pod that tries to schedule on this cluster has to clear three independent, structurally different gates:

• Pod Security Admission — the built-in Kubernetes control. Every application namespace runs at the restricted profile: no privileged containers, no host namespaces, no running as root.
• ValidatingAdmissionPolicy — native CEL rules running inside the API server itself, not a separate webhook. No :latest tags, a team label is mandatory on every workload, and no bare LoadBalancer Services (everything goes through the shared Gateway). A 9-case behavioural suite runs against a real API server in CI, not a mock.
• Binary Authorization — the supply-chain gate. Cloud Build builds, scans, and signs each image digest with a Cloud KMS key; Binary Authorization admits only digests carrying that signature. Tags are immutable, so there's no way to swap what a tag points to after the fact.

The live test suite proves this isn't decorative. An unsigned image is rejected with an explicit No attestations found message; a deploy that references a mutable tag instead of a digest is denied; and the pipeline's own build — signed the normal way — is admitted without any manual override.

Soft multi-tenancy, the way it actually works

Each team gets a namespace with Pod Security restricted, a ResourceQuota, a default-deny NetworkPolicy, and its own identity via Workload Identity Federation for GKE — IAM roles are granted straight to ns/<namespace>/sa/<serviceaccount>, no service-account keys anywhere. The live checks make the isolation concrete rather than assumed: team-a's workload can read its own GCS bucket; team-b, using the exact same code path but a different namespace identity, gets a 403. That's the test that actually matters — not that the IAM binding exists, but that the wrong identity is refused.

Releases that catch their own mistakes

Google Cloud Deploy owns the application pipeline: staging deploys automatically, production requires a human approval, and once approved, traffic shifts 25% → 50% → 100% with a soak period between steps. To prove this isn't just configuration, I ran a real bad-release drill: a deliberately faulty build was pushed through to the 25% canary stage, and the live test suite watched 25% of responses come back faulty while the other 75% stayed healthy. Rolling back was a single command — and in production that rollback itself requires the same approval gate as a forward release, which is a detail easy to get wrong (make rollback "safe" and you've just built an unaudited backdoor around your own approval process).

The platform layer itself — namespaces, network policies, the shared Gateway, monitoring — is managed the GitOps way, through an Argo CD app-of-apps with self-heal and prune turned on, scoped to its own AppProject so it can't reach outside its lane. If someone hand-edits a NetworkPolicy on the live cluster, Argo CD notices the drift and heals it back to what's in Git — I exercised this directly in runbook 03.

Edge protection and observability that isn't a dashboard nobody looks at

Traffic reaches the cluster through a global external Application Load Balancer, provisioned via the Kubernetes Gateway API, sitting behind Cloud Armor with OWASP SQL-injection and XSS rule sets plus rate limiting. The live suite fires actual attack-shaped requests at the edge and confirms they're blocked before they reach a pod.

Behind the edge, Google's Managed Prometheus scrapes RED metrics (rate, errors, duration) from the demo "shop" service and feeds Cloud Monitoring PromQL alerting and a dashboard. The live test confirms the metrics are actually queryable through PromQL against the deployed workload — not just that a ServiceMonitor object exists somewhere.

Eight design calls, each with a trade-off I'm not hiding

The repo's ADRs are where the honest trade-offs live, and I think that's the part most architecture write-ups skip:

What a live deployment actually caught

None of the seven bugs this build hit were things I could have found by reading documentation. Autopilot enforces resource-request limits that Standard doesn't warn you about. GKE injects fields into admitted objects that then trip your own ignoreDifferences config in Argo CD if you don't account for them. NodeLocal DNS listens on 169.254.20.10, and a default-deny egress NetworkPolicy will silently break DNS resolution cluster-wide unless you explicitly allow it. Managed Prometheus runs in a namespace called gke-gmp-system, not the name you'd guess. And GitHub's OIDC subject claims are immutable in a way that broke the Workload Identity Federation trust relationship until I adjusted the attribute condition — documented as L9 in the troubleshooting runbook. Every one of these is the kind of thing that looks obvious in retrospect and invisible in a design doc.

One thing I want to be straightforward about: the full CI/CD release job — the one that would push a build through Workload Identity Federation end to end against a live cluster — was proven for keyless authentication and service-account impersonation, but was not run end to end against a live stack in this session. I'd rather say that plainly than imply more coverage than the evidence supports.

What's deliberately left out

The cluster's control-plane endpoint uses an authorized-networks allow-list rather than a fully private endpoint with private connectivity pools. There's no domain in this demo, so the Gateway serves HTTP only — no managed certificate. Binary Authorization checks for a valid signature but doesn't yet require a separate SLSA provenance or vulnerability attestation (that pairing shows up in the companion Cloud Run secure-delivery build). Tenancy is soft — no service mesh, no gVisor sandboxing between namespaces. All of these are named in the repo's threat model rather than glossed over.

Try it yourself

git clone https://github.com/soodrajesh/gcp-gke-platform-engineering
cd gcp-gke-platform-engineering
gcloud config set project <your-project>
./scripts/up.sh    # infra → cluster → GitOps → signed build → staging → prod canary → live tests
./scripts/down.sh  # 94 resources, nothing billable left

Autopilot bills per pod request, so a session left running for a day costs a few euros in pods, Cloud NAT, the load balancer, and Cloud Armor — bounded by a Terraform-managed budget with alerts. down.sh specifically removes the load balancer and network endpoint groups the Gateway controller creates outside Terraform's view before it destroys the VPC, because otherwise teardown just hangs.

The point of building this the hard way — live, torn down, rebuilt from scratch, every claim checked against the running system — is that "it should work" and "it worked, here's the test output" are different sentences. This repo only makes the second kind.

📢 Have questions or feedback? Drop a comment below or connect with me on Twitter/X@spysood!