Hub-and-Spoke Network Landing Zone on Google Cloud

Landing zones are the least glamorous thing you can build on a cloud and the thing everything else quietly depends on. Get the network foundation wrong and no amount of good application security fixes it — a compromised dev environment that can route to prod is a landing-zone failure dressed up as an app-security incident. I wanted to build a landing zone shaped the way a real platform team would build one: environment isolation that's provable, not just diagrammed; admin access with no bastion host and no public IPs; and every claim checked against the running network, not against a Terraform plan.

gcp-network-landing-zone is three VPCs — hub, prod, dev — joined by Network Connectivity Center in a star topology, default-deny firewall policies with logged denials, private-only VMs reached through Identity-Aware Proxy, centralized private DNS, Cloud NAT egress, and flow logs landing in BigQuery. It applied 71 resources on the first from-scratch run in europe-west1, and its live test suite passed 42 of 42.

Isolation, proven three separate ways

The claim "prod and dev can't reach each other" is easy to state and easy to get subtly wrong — a peering misconfiguration, a firewall rule that's broader than intended, a route that leaks. So this repo doesn't prove it once. It proves it three independent ways that would each have to be wrong simultaneously for the isolation claim to be false:

1. Google's Connectivity Tests — a managed service that traces the actual forwarding path a packet would take and reports whether it's deliverable, independent of anything this repo wrote.
2. The VPC route tables themselves — confirming there's no route for prod's CIDR present anywhere inside the dev VPC (and vice versa), which is the mechanism NCC's star topology relies on: edges only get routes to the center, never to each other.
3. Real packets — a live curl from inside one spoke to a service in the other spoke, which returns 000 (connection refused, not even a TCP handshake).

That last one is the test I'd trust most, because route tables and managed connectivity checks can both be technically correct while still missing something a live packet would catch — an overly permissive firewall rule that a route table check doesn't evaluate, for instance. The spoke firewalls independently admit only the hub CIDR, which functions as a second, structurally different isolation layer beneath the NCC topology itself: even if NCC's routing had a gap, the firewall policy is the backstop.

No bastion, no public IPs, still auditable

Every VM in this landing zone has no external IP address, runs as a Shielded VM, and has project-wide SSH keys blocked at the metadata level. Administrative access goes exclusively through Identity-Aware Proxy with OS Login, so "who can SSH into this box" is answered entirely by IAM — there's no separate bastion host to patch, harden, and audit as its own attack surface. Every live check in the test suite runs over an actual IAP tunnel, which is a meaningfully different claim than "IAP is configured": it means the entire test suite itself only works because IAP-based access actually functions end to end.

Denials you can see, not denials that vanish

The default, implicit deny in a GCP firewall policy is silent — nothing logs it, nothing tells you a request was dropped. This repo replaces that silence with an explicit deny-all rule at the bottom of every firewall policy, and that explicit rule is logged. The logs feed a log-based metric, which feeds a Cloud Monitoring alert. In other words: instead of trusting that the implicit deny is doing its job, the repo makes the deny an event you can query, graph, and alert on.

DNS as one place, not three

Rather than each spoke owning its own DNS zone (which drifts, inevitably), there's a single private DNS zone in the hub, DNS-peered into both prod and dev. One name, one source of truth, plus query logging so you can see who resolved what. ADR 0004 in the repo puts it plainly: a name is not a route — DNS peering resolves names across the hub, but it's the NCC topology and firewall policy, not DNS, that actually decides whether the resulting IP is reachable. It's a subtle distinction that's easy to blur if you're not careful about which layer is doing which job.

Four decisions, stated plainly

The NCC-star choice is the one worth dwelling on. VPC peering doesn't transit — hub-and-spoke peering alone would leave you managing N² relationships as spokes grow, and peering can't be chained through a hub. Shared VPC collapses the isolation boundary you're trying to build in the first place. A full mesh scales even worse than peering. NCC's star topology gives you a hub that intentionally can reach every spoke, and spokes that structurally cannot reach each other, without hand-managing routes — the isolation is a property of the topology itself, not something reinforced only by policy.

Getting the test suite honest

The build hit a subtlety worth naming: on the first pass, one of the isolation checks passed — but for the wrong reason. A DNS resolution failure between spokes was being read by the test as "connection blocked" when what had actually happened was that the query never resolved at all, which says nothing about whether the underlying network path was actually closed. It's the kind of false positive that's dangerous precisely because it looks like a passing security test. The fix, and the general lesson, is documented in runbook 08: when a security check passes suspiciously easily, verify it isn't passing for the wrong reason before you trust it.

What's deliberately not here

There's no on-prem connectivity (Cloud VPN or Interconnect) — this landing zone models multi-environment isolation inside GCP, not hybrid connectivity. There's no centralized egress inspection via Cloud NGFW Enterprise or a network virtual appliance. VPC Service Controls and organization policies are out of scope — they need Organization Admin, which this project doesn't have. It's single-region, and the workloads used to prove reachability are tiny HTTP probes, not real services.

Try it yourself

git clone https://github.com/soodrajesh/gcp-network-landing-zone
cd gcp-network-landing-zone
gcloud config set project <your-project>
./scripts/up.sh    # state bucket → 3 VPCs + NCC hub → firewall/NAT/DNS/logging → live tests
./scripts/down.sh  # delete everything

Running costs an estimated €0.5–1/day from list prices — Cloud NAT, flow-log ingestion into BigQuery, the NCC hub itself — bounded by a Terraform-managed budget alert, and €0 once down.sh finishes. For a landing zone specifically, I think the estimate-vs-measured distinction matters enough to say plainly: this is a list-price estimate, not a bill I measured, and the repo says so rather than implying more precision than it has.

📢 Have questions or feedback? Drop a comment below or connect with me on Twitter/X@spysood!