— Stack · Integrations

Databricks on AWS

A lakehouse platform on AWS. Data comes in through managed connectors, files or streams, compute runs in your VPC or in Databricks' account, and agents reach governed data through MCP. Tables sit in your S3 unless you choose Databricks default storage.

How it fits

Ingest from your AWS account, govern in Unity Catalog, serve tables and agents.

Lakeflow Connect pulls from databases and SaaS apps with built-in CDC, so AWS DMS is one option among others. Default storage lives in Databricks' account, is reachable only from serverless compute, and external Iceberg clients cannot read its files. Unity Catalog exposes UniForm tables to Athena and EMR through its Iceberg REST endpoint.

Four ways in

Connect it, load files, stream it, or query it in place.

Managed

Lakeflow Connect

Managed connectors for databases like SQL Server and apps like Salesforce, with CDC, retries and schema evolution handled. Databricks recommends starting here.

Files

Auto Loader

Picks up new files as they land in S3 and loads only those. Fits exports, DMS change files and partner drops.

Streams

Structured Streaming

Reads Kinesis or Kafka on MSK with exactly-once processing. Lakeflow pipelines wrap it with orchestration and data quality checks.

In place

Federation

Unity Catalog reads tables registered in Glue Data Catalog or in external databases without copying them. Federated Glue tables are read-only.

The three choices that matter

A few decisions set the bill and who can read the data.

Classic or serverless compute

Classic compute runs in your AWS account: you keep VPC and instance control and pay EC2 directly. Serverless runs in Databricks' account, with compute bundled into the DBU rate.

Where tables live

Managed tables in your bucket get Unity Catalog optimizations; external tables leave file layout to you. Default storage is simplest on serverless workspaces, but only serverless compute reaches it.

How other engines read

UniForm writes Iceberg metadata beside Delta, so Athena, Trino and Spark read the same Parquet files through Unity Catalog. In the other direction, Unity Catalog can read what Glue already catalogs.

Tag values describe mechanisms, not measured figures.

What you pay

A Databricks bill, in one line.

Databricks compute

DBUs × DBU rate

DBUs consumed per hour times a DBU rate that varies by product, tier, region and compute type

Classic compute

DBUs + EC2

The DBU charge is on top of the EC2 instances running in your own AWS account

Serverless

bundled DBU rate

Compute is included in the DBU rate; no separate EC2 line

Storage

your S3, or Databricks

S3 storage and requests on your AWS bill; default storage billed by Databricks

What we see in production

Clusters left running with no jobs on them.Set auto-termination, or move bursty work to serverless so idle time is not billed.

Billing split across Databricks and AWS invoices.Subscribe through AWS Marketplace to consolidate Databricks charges onto the AWS bill and draw down an EDP commitment.

External engines need to read Delta tables.Enable UniForm so Iceberg clients read the same files instead of maintaining a second copy.

Shared tables created on default storage.External Iceberg clients cannot read those files; put tables other engines need in a catalog on your own S3.

Agents writing SQL against raw tables.Start with Genie, which resolves metric definitions and table relationships before it queries.

Glue and Unity Catalog both claiming the same tables.Federated Glue tables are read-only in Unity Catalog; decide which catalog owns writes for each domain.

How we help

Platform selection & architecture

AWS native, Databricks or Snowflake. A costed decision, trade-offs written down.

Read the approach →

PDF · coming soon

The one-page Databricks on AWS field guide.

Best practices, cost traps and performance checks, in the order you should run them. Printable, no fluff.