Solution2022–2024PrizePicks

Compliance data tiers & GDPR pipelines

Hot, warm, cold, and frozen storage tiers for terabyte-scale gaming transactions — with finance backup and retention pipelines for GDPR.

Problem

Regulatory compliance and finance audit needed governed retention on terabyte-scale transactions. A single warehouse tier could not balance realtime access, analytics freshness, and long-term archive with row-level PII controls.

Approach

Define a four-tier lifecycle modeled on Elasticsearch data tiers plus compliance requirements. Pair tier design with finance backup and retention pipelines, governed cold storage, and warehouse row-level PII access.

Architecture

Lifecycle flow

Streaming sources through hot, warm, cold, and frozen tiers — governed reads for finance, retention exports for GDPR.

React Flow

Drag nodes to explore the flow

Data lifecycle

Operational

Hot

CDC streams, APIs, warehouse raw — seconds to minutes fresh.

Analytics

Warm

dbt and Prefect models — the layer analysts and DS query daily.

Governed read

Cold

Read-only snapshots with access controls for finance and audit.

Retention

Frozen

Archive exports for compliance holds and long-term retention policy.

Access model

Why tiers beat a single warehouse for regulated gaming data.

Before

One BigQuery dataset with mixed PII, operational tables, and ad-hoc analyst copies — hard to audit and expensive to retain.

After

Tiered storage with row-level PII controls, governed cold access, and frozen exports that finance and compliance can point to.

Solution highlights

Hot tier

CDC, APIs, and warehouse raw for operational and near-realtime use.

Warm tier

dbt and Prefect models — the analytics-ready layer teams query daily.

Cold tier

Read-only snapshots with governed access for finance and audit.

Frozen tier

Retention and archive exports (CSV, Parquet, BigQuery) for compliance holds.

GDPR pipelines

Finance backup and retention pipelines with row-level PII controls in the warehouse.

GDPRBigQueryGCSCompliancePII