dataonaws
— Stack · Integrations

DuckDB on AWS

An open-source analytical database that runs inside your own process. On AWS it turns a Lambda function or a Fargate task into an ETL job: read raw files from S3 with SQL, transform them, and write Iceberg tables to S3 Tables. You pay for the seconds of compute and the storage, with no cluster and no engine licence.

How it fits

A file lands, a small job starts, DuckDB writes tables every engine can read.

TRIGGERSYOUR COMPUTESTORAGEREADERSFile in S3object created eventScheduleEventBridgeStep Functionsmulti-step pipelineLambdaup to 10 GB, 15 minFargate taskbigger, longer jobsCPU, memory you set20 GB disk includedSpot up to 70% offDuckDBSQL engine in-processreads Parquet, CSV, JSONstraight from S3SQL joins, aggregateswindow functionsspills to local diskwrites Iceberg or ParquetS3 Tablesmanaged IcebergS3 Parquetcurated filesGlue Data CatalogIceberg RESTAny enginesame tablesAthenaRedshiftAurora PostgreSQLEMR, Glue SparkSageMaker Unified StudioDuckDB on a laptop112234numbers follow the order of the workflow below
  1. 1A new file lands in S3, or a schedule fires.
  2. 2Lambda or a Fargate task starts and runs DuckDB with your SQL, reading the raw files directly from S3.
  3. 3DuckDB writes the result as Iceberg through an attached catalog (S3 Tables or Glue), or as plain Parquet files.
  4. 4Athena, Redshift, Aurora PostgreSQL or Spark read the same tables, with no copy.

Writing Iceberg from DuckDB requires an attached Iceberg REST catalog, such as S3 Tables or AWS Glue; reading files directly needs no catalog. DuckLabs, the team that maintains DuckDB, joined AWS in September 2026; Aurora PostgreSQL is the first AWS service to embed DuckDB. DuckDB itself stays open source.

Where it runs

Three places to run the same SQL.

Event

Lambda

One file in, one table out. Up to 10,240 MB of memory, up to 10,240 MB of /tmp for spills, 15 minutes per run, billed by the millisecond.

Batch

Fargate

A container task with the vCPU and memory you choose, for jobs that outgrow a Lambda run. Fargate Spot cuts the price for jobs that can restart.

Laptop

Local or CI

The same binary runs on a developer machine and in a pull request check, so the transformation is tested before it ships.

The three choices that matter

Three decisions keep it cheap and keep it working.

01

Lambda or Fargate

Lambda fits per-file and hourly jobs and costs nothing between runs. Move to Fargate when a run needs more memory, more time, or a steady batch window.

02

Files or Iceberg tables

Plain Parquet is enough for append-only outputs. Iceberg adds updates, deletes and schema changes that other engines see; S3 Tables also runs compaction for you.

03

When to hand over to another engine

DuckDB runs on one machine. When a job's working set no longer fits in memory plus local disk, or many users query the same tables, Athena or Spark takes over on the same Iceberg data.

Tag values describe mechanisms, not measured figures. Athena price for US East (N. Virginia).

What you pay

A DuckDB-on-AWS bill: compute seconds and storage.

Lambda
requests + GB-seconds
Billed per millisecond at the memory you set; 1 million requests and 400,000 GB-seconds a month free
+
Fargate
vCPU + GB per second
From image download to task end; 20 GB of ephemeral storage included; Spot up to 70% off
+
S3
storage + requests
GET about $0.0004 per 1,000; many small files cost more in requests than in storage
+
S3 Tables
storage + maintenance
Per GB-month, plus monitoring and compaction per object
+
DuckDB
free
Open source under the MIT licence; no engine or seat fee
What we see in production
A Lambda run killed for memory on a large file.Set a memory limit in DuckDB and give it /tmp to spill to; past 10 GB of working data, run it on Fargate.
Two jobs writing the same Iceberg table at once.Commits conflict and one fails; give each table one writer, or serialise runs with Step Functions.
One Lambda run per small file, thousands of tiny outputs.Batch on a schedule instead, and let S3 Tables compaction merge what is left.
Extensions downloaded on every cold start.Bake the iceberg, httpfs and aws extensions into the container image or a layer.
AWS keys pasted into the job's SQL.Use DuckDB's credential chain secret so the job uses the IAM role of the Lambda or task.
A job that keeps growing until it no longer fits.Keep the SQL portable; the same Iceberg tables can move to Athena or Glue Spark without a migration.
How we help

Industrial-grade data pipelines

Use cases in production, not POCs: tests, CI/CD, lineage, on-call that sleeps at night.

Read the approach →
PDF · coming soon

The one-page DuckDB on AWS field guide.

Best practices, cost traps and performance checks, in the order you should run them. Printable, no fluff.

One email when the PDF is ready. Nothing else unless you tick the box.