Building a GCP-native data platform from zero

Architected and operate a GCP data platform processing millions of events per month, with cross-cloud CDC ingestion and SOC2-aligned governance.

• GCP
• BigQuery
• Dataflow
• Dataform
• Cloud Composer
• Datastream CDC
• Terraform
• Terraform Cloud

Context

At Second Nature, I own the data platform: a GCP-native analytics stack ingesting operational data from multiple source systems into a unified BigQuery warehouse.

Challenge

Build a production-grade data platform from an empty GCP project — reliable ingestion from systems living in a different cloud, a governance model that satisfies SOC2 without slowing analysts down, and infrastructure that a small team can actually operate.

What I did

• Designed a Medallion architecture (Bronze/Silver/Gold) using BigQuery, Dataflow, and Dataform.
• Deployed Datastream CDC for real-time replication from AWS-hosted PostgreSQL/MariaDB into BigQuery over a private network path.
• Orchestrated ingestion and transformation with Cloud Composer (Airflow).
• Made Terraform (via Terraform Cloud, GitHub-OIDC authenticated — no long-lived cloud credentials) the only way infrastructure changes ship, with CI gates for linting, security scanning, and coverage.
• Built the SOC2/PCI-DSS-aligned governance layer: least-privilege IAM, Zero Trust review, audit-ready logging.

Stack

GCP (BigQuery, Dataflow, Dataform, Dataplex, Cloud Composer, Datastream), Terraform + Terraform Cloud, GitHub Actions, AWS (source-side RDS).

Outcome

A platform that runs the company’s operational analytics with high reliability, passed SOC2 controls review, and cut infrastructure deployment cycles from days to hours.