Amazon MSK
Apache Kafka as a managed service. You keep the Kafka API, clients and ecosystem; AWS runs the brokers, patching and storage. Producers write events, consumers read them, and MSK keeps the log. How you run it, on provisioned brokers or serverless, decides how you pay.
Events in from producers, out to consumers. AWS runs the log in between.
Clients talk the standard Kafka protocol, so existing producers, consumers, Connect and Streams code works unchanged. MSK handles broker provisioning, patching and storage. Tiered storage moves older log segments to a cheaper tier that clients still read through the Kafka API. MirrorMaker 2 still works for migration, but MSK Replicator is the managed path: it replicates topic data and metadata, synchronises consumer offsets, and supports bidirectional replication for phased cutover from on-prem, Confluent, Aiven, RedPanda, WarpStream or AutoMQ. S3 delivery (streaming tables as Iceberg, or raw files) is a native Express-broker capability, not a separate connector.
Same Kafka API, three deployment shapes.
Provisioned, full control
You pick the broker instance type and count, set partitions and retention, and tune Kafka configs. The most control for steady, large or latency-sensitive workloads.
Provisioned, less to manage
Up to 3x more throughput per broker than Standard, according to AWS, with faster scaling and recovery. Storage is unlimited and managed, and you pay per GB written in on top of the instance hours.
No brokers to size
You create a cluster and topics; MSK scales capacity. You pay per cluster-hour, per partition-hour, and per GB in and out. Fits bursty or unpredictable traffic.
A few decisions set the bill and the operating load.
Which MSK flavour
Standard gives the most control for large steady workloads. Express raises throughput per broker and manages storage, with a per-GB data-in charge. Serverless skips sizing entirely and bills per partition and per GB.
How clients authenticate
IAM authenticates clients and authorises Kafka actions with AWS policies, so there are no Kafka credentials to rotate. SASL/SCRAM and mTLS stay available for clients that cannot use IAM; Serverless supports IAM only.
Migrating off self-managed or Confluent
MSK Replicator moves topic data and metadata with no infrastructure to run, synchronises consumer offsets so you cut over producers and consumers at your own pace, and replicates both ways for rollback. MirrorMaker 2 still works if you want to run the tooling yourself.
Tag values describe mechanisms, not measured figures.
An MSK bill, in one line.
Industrial-grade data pipelines
Use cases in production, not POCs: tests, CI/CD, lineage, on-call that sleeps at night.
Read the approach →The one-page Amazon MSK field guide.
Best practices, cost traps and performance checks, in the order you should run them. Printable, no fluff.