Data Firehose
A managed pipe from a stream to a destination, with no code to run. You point producers at a Firehose stream; it buffers, can transform and reformat, and delivers to S3, a warehouse, a search cluster or an HTTP endpoint. You pay per GB ingested. It moves and lands data; it does not do stateful stream processing.
Producers write in. Firehose buffers, shapes, and delivers out.
Sources: Direct PUT (SDK, Lambda, Kinesis Agent, Fluent Bit, and vended logs from CloudWatch Logs, WAF, VPC Flow Logs and others), an existing Kinesis Data Stream, or an Amazon MSK topic. Destinations include Amazon S3, Apache Iceberg tables (including S3 Tables), Redshift, OpenSearch Service and Serverless, Snowflake, Splunk, and any HTTP endpoint, with built-in integrations for Datadog, New Relic, MongoDB, Dynatrace, Elastic and others. Delivery is at-least-once, so sinks should tolerate duplicates.
Buffer, reshape, and land, with knobs for each.
Buffer size and interval
Firehose holds records until a buffer fills (size in MB) or an interval passes (up to 15 minutes), then writes one object. Bigger buffers mean fewer, larger files and higher latency; smaller buffers mean fresher data and more objects.
Transform with Lambda
A Lambda function can filter, enrich or correct each batch before delivery. You own the function; Firehose invokes it and can back the untransformed source up to a second S3 bucket.
Format conversion
Convert JSON to Parquet or ORC in flight, using a table schema from the Glue Data Catalog. This makes the landed data cheaper to scan with Athena, Redshift Spectrum and EMR.
Dynamic partitioning
Route records to S3 prefixes by a key read from the record (a JQ expression or your Lambda), so a lake lands already partitioned by date, tenant or type, ready to query.
A few settings set the latency, the file layout, and the bill.
Buffer size vs delivery latency
A larger buffer produces fewer, bigger objects that are cheaper and faster to query later, at the cost of up to 15 minutes of delay. Small buffers deliver sooner but scatter many small files across S3.
Which source, which billing shape
Direct PUT and KDS ingestion rounds each record up to the next 5 KB, so many tiny records cost more than their raw bytes. MSK and vended logs bill on actual GB. All three are tiered: the per-GB rate drops past 500 TB a month.
Which extras you turn on
Base ingestion has no extra delivery charge. Format conversion adds a per-GB rate. Dynamic partitioning adds a per-GB and a per-object charge, plus a per-hour charge if you use the JQ parser. Delivery into a VPC adds per-GB and per-AZ-hour charges.
Bar lengths are illustrative of the latency-vs-file-size trade-off, not measured figures.
A Firehose bill, in one line.
Industrial-grade data pipelines
Use cases in production, not POCs: tests, CI/CD, lineage, on-call that sleeps at night.
Read the approach →The one-page Data Firehose field guide.
Best practices, cost traps and performance checks, in the order you should run them. Printable, no fluff.