MLflow 3.17 UI on Cloud Run showing the taxi-high-tip experiment with a sweep of training runs and a registered model v1
MLflow running on Cloud Run: the taxi-high-tip experiment, one successful sweep of 6 child runs, the registered model v1, and two earlier failed sweeps left visible.

I wanted a concrete GCP counterpart for each Databricks concept, one I had actually run rather than read about. gcp-lakehouse-mlops-lab builds the same lakehouse and MLOps workflow twice: as Databricks notebooks (Free Edition) and on GCP with Terraform, Dataproc Serverless, BigQuery, Cloud Run and Vertex AI. Two of six planned projects are built and verified. One caveat up front: the Databricks notebooks are written but were not executed here, because I had no workspace token. The GCP side is what ran and what I measured.

Project 1: a medallion lakehouse on Delta

One month of NYC yellow-taxi trips, 2.46M rows, taken through bronze, silver and gold on Delta Lake.

StepDatabricksGCP
Spark runtimeServerless compute, spark existsDataproc Serverless batch, runtime 2.2, 2 x 4-core executors
Delta LakeBuilt indelta-spark and delta-storage jars staged in GCS, plus two Spark configs
Storage / catalogUnity Catalog managed tablesDelta files in GCS, exposed as BigLake external tables (format = DELTA_LAKE)
Raw sourcesamples.nyctaxi.tripsEXPORT DATA from the public BigQuery taxi table

What the run proved: bronze 2,463,900 rows, silver 2,291,506 after removing invalid and duplicate trips, gold 5,886 rows (day by pickup zone). A second run executes silver as a MERGE (Delta version 2) and the count stays identical, so it is idempotent. Time travel with versionAsOf = 0 returns the original snapshot and DESCRIBE HISTORY lists WRITE, OPTIMIZE, MERGE. Appending a frame with fare_amount as a string is rejected with DELTA_FAILED_TO_MERGE_FIELDS. And BigQuery reads the same files Spark wrote, with no copy and no load job.

Things that bit me

  1. Scala version. Runtime 2.2 is Spark 3.5 on Scala 2.13. The delta-spark_2.12 jar fails with NoClassDefFoundError: scala/Serializable; use _2.13.
  2. --jars does not give you the Python API. With jars staged that way the delta.tables module is not on the path, so the job uses SQL MERGE and DESCRIBE HISTORY. On Databricks DeltaTable just works.
  3. TIMESTAMP_NTZ. BigQuery's Parquet export arrives timezone-less, which enables a Delta table feature BigQuery's reader may not support. The job casts to plain timestamp first.
  4. No internet. Dataproc Serverless needs Private Google Access on the subnet and an internal-allow firewall rule.

Project 2: experiment tracking and registry

The task: predict whether a card-paid trip tips 20% or more, with features from the silver table read through BigQuery. The same 8-model sweep (logistic regression, random forest, hist-GB, XGBoost where available) is tracked three ways.

ConcernDatabricksGCP
Tracking serverBuilt into the workspaceSelf-hosted MLflow 3.17 on Cloud Run, private, IAM
Backend storeManagedCloud SQL Postgres (db-f1-micro)
ArtifactsManagedGCS, proxied by the server (--serve-artifacts) so clients need no storage credentials
AuthWorkspace login or PATCloud Run IAM, MLFLOW_TRACKING_TOKEN from an identity token
RegistryUnity CatalogMLflow registry on the server
Managed alternativen/aVertex AI Experiments (train_vertex.py), no server at all

Metrics were identical on both trackers (same seed, same split). The best model was hist_gb with AUC about 0.641 and F1 about 0.887 on a 90k-row sample. The AUC is modest because tipping behaviour is noisy; the point here is the tooling, not the model. taxi-high-tip v1 is registered with the alias candidate, and the screenshot shows it next to the run that produced it.

Things that bit me

  1. Cloud Run returned 429s while logging models. A 1 GiB, 1-CPU instance was overwhelmed by large random-forest artifacts proxied through the server. 4 GiB and 2 CPU fixed it. The two failed sweeps (26 and 20 minutes) are still visible in the UI, which I think is the honest way to leave them.
  2. MLflow 3.x defaults sklearn serialization to skops, which refuses tree types. Use serialization_format="cloudpickle" for your own models.
  3. --allowed-hosts: MLflow 3's middleware rejects unknown Host headers. "*" is acceptable here only because Cloud Run IAM already gates every request.
  4. Vertex AI Experiments quietly creates a TensorBoard instance outside Terraform. down.sh deletes it.
  5. The first Cloud Run deploy takes about 6 minutes because Alembic migrates a fresh Cloud SQL.

Cost and process

Project 1 takes about 10 minutes and a few cents. Project 2 adds Cloud SQL f1-micro at roughly 0.01 EUR an hour while it exists; Cloud Run scales to zero. Terraform is always applied from a saved, reviewed plan, never a blind -auto-approve, and down.sh destroys everything.

What is still planned

Model serving and promotion, a feature store with batch inference, governance with Dataplex and BigQuery policy tags, and an end-to-end capstone are listed in the repo as planned, not built.

Try it yourself

git clone https://github.com/soodrajesh/gcp-lakehouse-mlops-lab
cd gcp-lakehouse-mlops-lab
./scripts/p1-up.sh   # ~10 min: Terraform, jars, raw export, Spark batch, BigQuery tables, checks
./scripts/p2-up.sh   # ~10 min: Cloud SQL + MLflow on Cloud Run
gcloud run services proxy mlflow --region us-central1 --port 8099
./scripts/down.sh

📢 Have questions or feedback? Drop a comment below or connect with me on Twitter/X@spysood!