Industrial-grade data pipelines
Use cases in production, not POCs: tests, CI/CD, lineage, on-call that sleeps at night.
A pipeline that works in a notebook is a prototype. In production it has to run every day, fail loudly, recover cleanly and be changed without fear. Most teams we meet have the first part and struggle with the rest:
- -No tests on the data, so errors are found by the business users in their dashboards.
- -Changes go to production by hand, and a fix on one pipeline breaks another.
- -Nobody can say which reports depend on a table before changing it.
- -Alerts fire at night for failures that could have retried on their own, and the real incidents get lost in the noise.
- Data code is code. It is versioned, reviewed, tested and deployed through CI/CD like the rest of your software.
- Declarative where possible. dbt models or Spark Declarative Pipelines describe what the data should be, and the engine works out how to get there.
- Tests and contracts at the boundaries. What a source sends and what a consumer expects are written down and checked on every run.
- On-call is designed, not endured. Each alert says what broke, what it affects and what to do, and transient failures retry on their own.
- 01
Framing the use case
Who consumes the data, how fresh it must be, what happens when it is late or wrong. That gives the service level and the tests that matter.
- 02
Pipeline as code
dbt, Spark on Glue or EMR, or Spark Declarative Pipelines, orchestrated with Amazon MWAA or Step Functions. Incremental by default, idempotent so a rerun never duplicates data.
- 03
Tests & data quality
Unit tests on the transformations, data quality rules on the outputs (freshness, volume, uniqueness, business rules) and contracts with the sources.
- 04
CI/CD & environments
Every change runs in an isolated environment on a sample of real data before it reaches production. Deployments are automated and can be rolled back.
- 05
Lineage, monitoring & runbooks
Lineage from source to dashboard, alerts routed to the team that owns the pipeline, and a runbook per failure mode so the person on call knows what to do.
- -A 2-hour discovery on one use case and the pipelines behind it.
- -That use case taken to production with tests, CI/CD and monitoring, as the template for the others.
- -The other pipelines moved onto the template, the most fragile ones first.
The production-ready pipeline checklist
What we check before a pipeline goes to production: tests, idempotency, contracts, alerts and runbooks, with the failure modes we see most often.
Coding agents now write a good share of pipeline code. With tests, CI/CD and contracts in place, their changes are reviewed and validated like anyone else's.
Developer Agentic Platform →