Airflow and dbt, built from code
The pipelines were stored procedures and Python scripts run by cron. I got a server on Google Cloud, set up Airflow and dbt with Terraform and Ansible, and made changes deploy themselves on merge. Old jobs moved over by running old and new side by side until they matched exactly. Now the pipelines are easy to see, test and rebuild.
How it works
Tap a step for the details.
Stored procedures and Python scripts kicked off by cron.
A Google Cloud server set up with Terraform and Ansible.
Airflow runs the jobs, dbt shapes the data.
Merge a change and it deploys itself.
Old and new ran together until the results matched exactly, then we switched.