Dataproc Spark and Prefect on GKE for continuous Postgres-to-BigQuery sync — users, payments, and gameplay configuration at production scale.
Problem
Analytics needed near-realtime copies of core Postgres tables while backfills had to stay reliable. Streaming state needed backup before destructive changes.
Approach
Build PySpark jobs on Dataproc for batch backfill and incremental reads, orchestrate continuous merge loops on Prefect with tiered GKE work queues. Evaluate Databricks for dedup at scale — recommend native Spark on Dataproc for cost and control.