Platform selection & architecture

Choosing a data platform is a decision for the next five to ten years, and it is often made on a demo, a vendor relationship or the preference of the last person hired. The questions that matter usually come up after the contract:

• -The cost at your volume and your usage patterns, not the list price per unit.
• -What your team can run and hire for, with the skills it has today.
• -How easily the data can move to another engine if the choice turns out wrong.
• -How the platform fits with your security, governance and AI plans.

• We are independent from the vendors. We have built on AWS native services, Databricks and Snowflake, and we recommend what fits your case.
• The decision is tested on your workloads. A few representative jobs and queries run on the candidates, and the cost and performance come from those runs.
• Open formats reduce the stakes. With Apache Iceberg and an open catalog, the engine can change later without moving the data again.
• The trade-offs are written down. The final document says what you gain, what you give up and why, so the decision can be explained and revisited.

1. 01

   Requirements

   Workloads (batch, streaming, BI, machine learning, AI), volumes, the team and its skills, security and residency constraints, and the budget. Agreed with the stakeholders before any comparison.

2. 02

   Shortlist & criteria

   Usually two or three options among AWS native (S3, Iceberg, Glue, EMR, Athena, Redshift), Databricks on AWS and Snowflake on AWS, with weighted criteria agreed in advance.

3. 03

   Benchmark on your workloads

   A few representative pipelines and queries run on each candidate in your AWS account. We measure performance, cost at your volume and the effort to build and operate.

4. 04

   Costed decision

   A written recommendation with a three-year cost estimate, the trade-offs, the risks and the reference architecture for the chosen option.

5. 05

   First steps

   The landing zone, governance and the first workload on the chosen platform, so the decision turns into a running system.

• -A 2-hour discovery on your workloads, constraints and the options on the table.
• -Requirements and criteria agreed within the first week.
• -A benchmark on your workloads and a costed recommendation in 4 to 6 weeks.