Build & optimize your data platform on AWS

Most data platforms we review on AWS work. The questions come later: the bill grows faster than the data, nobody can say which team or which job costs what, and the architecture still reflects the choices made five years ago. We usually find:

• -Clusters and warehouses sized for the peak and running all the time.
• -Full reloads where an incremental job would do, and data scanned again and again by the same queries.
• -No cost per team, per pipeline or per use case, so nobody owns the bill.
• -A mix of formats and catalogs that makes every new use case slower to start.

• We start from the bill and the workloads, not from a target architecture. The first memo says where the money goes and what we would change first.
• Open table formats by default. Apache Iceberg on S3, with one catalog, keeps the data readable by every engine you use now or later.
• Serverless where the load is irregular, provisioned where it is steady. The choice is made per workload, with the numbers.
• Cost is a metric like latency. Every team sees what its pipelines and queries cost, and it is reviewed with the rest of the platform.

1. 01

   Cost & architecture review

   Cost and Usage Report, job history, query history and the current architecture. Within a few days, a written memo: what is solid, where the money goes, what we would change first and what it saves.

2. 02

   Quick wins

   Right-sizing, auto-pause and auto-scaling, storage classes and lifecycle rules, partitioning and compaction, incremental instead of full loads. Most of these changes are reversible and pay back within weeks.

3. 03

   Target architecture

   Storage, table format, catalog, engines and governance for the next years, with the trade-offs written down. Typically S3 and Iceberg (or S3 Tables), the Glue Data Catalog, and Athena, Redshift, EMR or Glue depending on the workload.

4. 04

   Build, workload by workload

   Infrastructure as code, CI/CD for the data code, and each workload moved when the target is ready for it, not in one big switch.

5. 05

   Cost ownership

   Tags and cost allocation per team and per pipeline, budgets and alerts, and a monthly review so the bill stays explained.

• -A 2-hour discovery on your platform and your last bills.
• -A cost and architecture memo within the first week.
• -The quick wins first, then the target architecture, workload by workload.