Streaming pipelines fail in boring, expensive ways: a producer ships a malformed event and it either crashes the consumer or silently corrupts a downstream table; a transient error causes a message to vanish instead of retrying; an analyst querying "for a quick look" ends up with a customer's raw phone number in a spreadsheet. None of these are exotic failure modes — they're the default outcome of a pipeline that doesn't design for them up front.
gcp-streaming-data-platform is built around a simple principle: enforce structure at the edge, never lose a message even when something downstream is broken, and never let raw PII reach a human who doesn't need it. Pub/Sub with an enforced Avro schema feeds a BigQuery subscription for a raw, replayable landing zone; a Cloud Run enricher with retries and dead-lettering produces curated tables; PII-masked authorized views are what analysts actually query. It went live in europe-west1 with data resident in BigQuery EU, and its 30-check test suite passed 30 of 30 — with the dead-letter replay in runbook 04 executed for real, not just described.
Bad data stopped at the door
The schema lives on the Pub/Sub topic itself, not in application code somewhere downstream. A publish attempt with the wrong shape or the wrong field type fails immediately with INVALID_ARGUMENT — before the message is accepted, before it can propagate anywhere. That's a meaningfully different guarantee than validating in a consumer, where a bad message has already been accepted and now has to be handled as an exception case somewhere down the chain.
Nothing lost, exactly-once results from at-least-once delivery
Every message the topic accepts lands untouched in a raw BigQuery table via a native BigQuery subscription — no custom code, no Dataflow job. The live test is blunt about this: 316 messages published, 316 rows landed. But Pub/Sub only guarantees at-least-once delivery, which means duplicates are a certainty over time, not an edge case. Rather than fight that at the ingestion layer, the raw table keeps every duplicate, and de-duplication happens at read time by event_id when curated tables are built — the test sequence makes the arithmetic visible: 316 raw rows in, 306 after removing exact publish-retry duplicates, 300 in the final curated mart. Each number is checkable against the last.
Poison doesn't block the pipeline and doesn't vanish
A message that's schema-valid but semantically broken — a foreign key that doesn't resolve, a value the enricher can't process — gets a 422 from the enricher, is retried a bounded number of times, and if it still fails, lands in a dead-letter topic and a raw.dead_letters table, with an alert fired. Crucially, BigQuery write failures dead-letter too, not just enricher failures — the failure mode is handled symmetrically wherever it originates in the pipeline, not just at the one place that was easiest to instrument.
And the replay path isn't theoretical — I ran it for real against the live system: pull dead-lettered messages back out, fix or reprocess, and republish. The point of dead-lettering instead of dropping is that a bad message is recoverable data, not a permanently lost event.
PII never reaches an analyst
Analysts query authorized views, not the underlying tables, and there are no direct table grants at all. The customer_key field is hashed before it's exposed through the view layer. The live test proves this from the analyst's actual vantage point: querying as an impersonated analyst identity returns the masked, hashed data — there's no path through which the raw column is reachable from that identity, because the identity never has table-level access to check.
Fast enough to matter, not fast for its own sake
Event-to-enriched-row latency measured p50 2.2 s and p95 5.6 s warm — numbers that matter for setting a realistic SLA, and numbers I'd rather publish honestly (including the p95 tail) than round down to a headline figure. A pipeline that's "usually fast" but occasionally takes 5+ seconds behaves very differently for a downstream consumer with a tight timeout than one that's consistently mediocre.
Four decisions
| Decision |
|---|
| BigQuery subscription for raw landing — no Dataflow job, no custom code |
| Enforce the Avro schema at the topic, not in a downstream consumer |
| Retry → dead-letter → replay, with attempt counts treated as approximate, not exact |
| Authorized views for analysts, never direct table grants |
The dead-letter attempt-count caveat is worth calling out specifically: Pub/Sub's delivery attempt counter is a best-effort value, not a precise ledger, and this repo says so rather than building alerting logic that silently assumes more precision than the platform actually gives you.
What's out of scope, on purpose
No CMEK, no VPC Service Controls, no BigQuery column-level policy tags — all organization-level controls this project-scoped build doesn't reach. The customer_key hash is unsalted, which is fine for a demo and wrong for real PII (a keyed hash is the documented fix). It's single-region, streaming inserts are best-effort deduplicated rather than exactly-once (the curated mart is treated as the source of truth, not the raw stream), and the event producer is synthetic.
Try it yourself
git clone https://github.com/soodrajesh/gcp-streaming-data-platform
cd gcp-streaming-data-platform
gcloud config set project <your-project>
./scripts/up.sh # infra → image build → Cloud Run + subscriptions → live tests
./scripts/down.sh # delete everything, including dataset contentsWell under €1 for a full build-test-teardown cycle — Cloud Run scales to zero between events, and a Terraform-managed budget with alerts bounds the total regardless. The infrastructure is cheap; the discipline of never losing a message and never leaking a column is the part worth the effort.
📢 Have questions or feedback? Drop a comment below or connect with me on Twitter/X@spysood!