dataonaws
— Stack · Streaming

Managed Flink

Managed Apache Flink for stateful stream processing: windows, joins, enrichment and exactly-once delivery that a per-record function cannot hold. You write the job in Java, Python or SQL; AWS runs the Flink cluster, scales it, and keeps the state. Capacity is measured in KPUs, one vCPU and 4 GB each.

How it fits

Streams in, a stateful job in the middle, results out.

YOUR SOURCESMANAGED FLINKSTATEDESTINATIONSKinesisData Streams shardsAmazon MSKKafka topicsOther Kafkaself-managed, ConfluentSource connectorsexactly-once readsKinesis, MSK, Kafkaoffsets in checkpointsStudio notebookinteractive, +2 KPUFlink jobJava, Python, SQLwindows, joinsenrichment, CEPKPUs = Par / PerKPUCheckpointsautomatic, 50 GB/KPUSnapshotsdurable backupsAmazon S3Iceberg tablesOpenSearchsearch indexDynamoDBkey-value sinkKinesis, MSKdownstream streamsYour consumersdashboards, appsmetrics + logs to CloudWatch · each application adds 1 KPU for orchestration · runs in your VPC

Source connectors track read positions (Kinesis sequence numbers, Kafka offsets) inside each checkpoint, so a restart replays from the last committed point and state stays consistent. Checkpoints run automatically to the 50 GB of running storage allocated per KPU; snapshots are optional durable backups you take on stop or on a schedule for point-in-time restore. Sinks reach exactly-once only where the destination supports a transactional or idempotent write (for example the S3/Iceberg and Kafka connectors).

Three ways to write the job

Same engine, three front ends.

Code

Java (DataStream API)

The full Flink API: custom operators, timers, broadcast state, complex event processing. Packaged as a JAR on S3. The most control, the most code.

Python

PyFlink

The Table and DataStream APIs in Python, with user-defined functions. Fits teams whose logic and libraries already live in Python.

SQL

Flink SQL / Studio

Windows, joins and aggregations as SQL. A Studio notebook runs it interactively against a live stream, then deploys the same query as a streaming application.

The three choices that matter

A few decisions set the cost and whether it keeps up.

01

Do you actually need state

Flink earns its cost when the answer depends on past events: sliding windows, stream joins, enrichment against reference data, deduplication. For stateless per-record work, a Lambda consumer or a Firehose transform is simpler and cheaper.

02

Parallelism and ParallelismPerKPU

KPUs allocated = Parallelism / ParallelismPerKPU. Raising ParallelismPerKPU packs more tasks onto each KPU, which pays off when operators wait on I/O (lookups, sink writes). Default is 1, max 8; parallelism default max is 256.

03

Autoscaling or fixed capacity

Autoscaling adds KPUs on a throughput spike and removes them gradually. Fix the KPU count for steady load or tight cost control. If you plan to scale past parallelism 128, set maxParallelism up front: changing it later breaks restore from an existing snapshot.

Bar lengths are illustrative of task density per KPU, not measured throughput.

What you pay

A Managed Flink bill, in one line.

Compute
$0.11 per KPU-hour
Per-second billing; 1 KPU = 1 vCPU + 4 GB; each app adds 1 KPU for orchestration (US East N. Virginia)
+
Running storage
$0.10 per GB-month
50 GB of running application storage allocated per KPU for state (US East N. Virginia)
+
Durable backups
$0.023 per GB-month
Optional snapshots for point-in-time restore
+
Studio notebook
+2 KPUs
Interactive mode adds 2 KPUs: one for orchestration, one for the dev environment
+
Sources and sinks
their own bill
Kinesis shards, MSK brokers, S3, OpenSearch, DynamoDB billed on your account
What we see in production
Reaching for Flink for stateless record-by-record work.A Lambda consumer or a Firehose transform does map/filter without a cluster or state to operate.
State grows unbounded and checkpoints slow down.Set TTL on keyed state and bound window sizes; running storage is capped at 50 GB per KPU.
maxParallelism left at the default, then scaled past 128.It is fixed at the job's max; changing it later blocks restore from an existing snapshot, so set it before you grow.
Exactly-once assumed end to end.Flink gives exactly-once within the job; the sink delivers it only if it is transactional or idempotent. Check the connector.
Running on an old Flink version.1.20 is the latest supported, 1.15.2 the oldest still supported; use in-place version upgrades to move statefully.
No alarm on restarts or checkpoint failures.Watch fullRestarts and numberOfFailedCheckpoints in CloudWatch; repeated restarts mean the job is not keeping up or state is too large.
How we help

Industrial-grade data pipelines

Use cases in production, not POCs: tests, CI/CD, lineage, on-call that sleeps at night.

Read the approach →
PDF · coming soon

The one-page Managed Flink field guide.

Best practices, cost traps and performance checks, in the order you should run them. Printable, no fluff.

One email when the PDF is ready. Nothing else unless you tick the box.