How Flink Works Internally
Flink is not just a library that runs a loop over events. It turns a streaming program into a distributed dataflow, splits operators into parallel subtasks, places those subtasks into task slots, and keeps state recoverable with checkpoints.
The problem Flink is trying to solve
Imagine a stream of events that never really ends:
clicks
payments
orders
sensor readings
database changes
You want to transform those events, group them, join them, aggregate them, detect patterns, write clean output, and survive machine failures while the stream keeps moving.
That is the problem Flink is built for.
It is not just trying to run code on many machines. It is trying to run a continuous dataflow where:
- events keep arriving
- operators may need memory
- work must be split across machines
- failures should not corrupt results
- late events need a time model
- the job may need to scale up or down
Flink turns stream processing into a distributed dataflow with managed state and recovery.
That sentence is the useful mental model. The rest of the internals are mostly how Flink makes that model run.
A Flink job is a graph
When you write a Flink program, you usually write something that looks like a chain:
events
.map(parseEvent)
.keyBy(Event::userId)
.window(TumblingEventTimeWindows.of(Duration.ofMinutes(1)))
.aggregate(countEvents)
.sinkTo(output);
Flink does not think of this as one loop.
It turns the program into a graph:
source -> map -> keyBy/shuffle -> window aggregate -> sink
The boxes are operators. The arrows are streams.
An operator is a processing step. It might read from Kafka, parse JSON, filter bad events, repartition by key, update a window, call an async service, or write to a sink.
source operator reads records
map operator transforms one record at a time
filter operator keeps or drops records
keyBy exchange repartitions records by key
window operator keeps state until a window closes
sink operator writes results somewhere else
Operators become subtasks
One operator is the logical idea.
A subtask is one running parallel copy of that operator.
If a map operator has parallelism 4, Flink runs four map subtasks:
map subtask 0
map subtask 1
map subtask 2
map subtask 3
Each subtask handles part of the input.
That is the first scaling trick. Flink does not make one operator magically faster. It creates many parallel instances of that operator and divides the stream across them.
operator: parse event
parallelism: 4
runtime:
parse event [0]
parse event [1]
parse event [2]
parse event [3]
The number of running parallel instances is the operator's parallelism.
Tasks are not the same as operators
The names are easy to mix up.
An operator is a logical processing step.
A subtask is one parallel instance of that operator.
A task is the runtime unit Flink schedules and executes. A task may contain one subtask, or it may contain a chain of subtasks from multiple operators.
Flink often chains compatible operators together:
source -> map -> filter
can become one runtime task:
task:
source subtask -> map subtask -> filter subtask
This avoids extra thread handoffs, queues, serialization, and network buffers between cheap local steps. It is one reason Flink can keep latency low.
But Flink cannot chain through every edge.
A keyBy usually creates a network shuffle:
source -> map -> keyBy(user_id) -> window
After the keyBy, records with the same key must land on the same downstream parallel instance. That means Flink has to repartition the stream.
Tasks are what the runtime executes. Operators are what your program describes.
What parallelism really means
Parallelism is not "how many machines do I have?"
Parallelism is "how many parallel instances of this operator should run?"
For example:
Kafka source parallelism: 8
parse map parallelism: 8
window parallelism: 16
sink parallelism: 4
Different operators can have different parallelism.
Flink also lets you set a default parallelism for the job, and override it for specific operators when one stage needs more or less capacity.
env.setParallelism(8);
events
.map(parseEvent)
.setParallelism(16)
.keyBy(Event::userId)
.process(updateUserState)
.setParallelism(32);
That can be useful, but it introduces a practical question:
How does data move from 16 upstream subtasks to 32 downstream subtasks?
The answer depends on the edge.
One-to-one edges can keep records in the same partition. Repartitioning edges redistribute records across the network. A keyBy hashes each key so all records for the same key go to the same downstream keyed subtask.
user_id=17 -> window subtask 3
user_id=42 -> window subtask 9
user_id=17 -> window subtask 3 again
That is why keyed state works.
All events for a key arrive at the same parallel instance, so that instance can keep local state for that key.
Why keyBy matters so much
keyBy is one of the most important Flink operations because it decides where state lives.
Before keyBy, records are just flowing through partitions.
After keyBy(user_id), Flink guarantees that records for the same user_id reach the same keyed operator subtask.
events
.keyBy(Event::userId)
.process(new UserStateFunction())
Inside that process function, state is local to the current key:
state for user 17
state for user 42
state for user 91
The state is partitioned with the stream.
That is the reason Flink can handle things like deduplication, windows, joins, and per-user counters at high scale. It does not keep all state in one central database for every event. It routes the event to the worker responsible for that key, then updates local managed state.
Flink scales stateful work by partitioning both the events and the state by the same key.
The hard part is key distribution.
If one key receives half the traffic, one subtask receives half the work. Increasing parallelism will not fix a single hot key unless you change the keying strategy or split that workload differently.
JobManager and TaskManagers
A Flink cluster has two main kinds of runtime processes.
The JobManager coordinates the job.
It accepts the submitted job graph, schedules work, reacts to failures, coordinates checkpoints, and manages recovery.
TaskManagers are the workers.
They run the tasks, exchange data with each other, hold local state, and send heartbeats and metrics back to the coordinator.
client submits job
|
v
JobManager plans and coordinates
|
v
TaskManagers run the tasks
The client is usually not part of the running dataflow. It packages and submits the job. After that, the cluster runs it.
Task slots are the scheduling units
A TaskManager is a worker process. It has one or more task slots.
Task slots are Flink's unit of scheduling and resource sharing.
TaskManager A
slot 0
slot 1
TaskManager B
slot 0
slot 1
If a cluster has two TaskManagers with two slots each, it has four slots.
That does not automatically mean only four operators can run. Flink can place a chain of subtasks from the same job into one slot.
For example, with parallelism 4:
slot 0: source[0] -> map[0] -> filter[0] -> sink[0]
slot 1: source[1] -> map[1] -> filter[1] -> sink[1]
slot 2: source[2] -> map[2] -> filter[2] -> sink[2]
slot 3: source[3] -> map[3] -> filter[3] -> sink[3]
This is why a rough rule is:
available slots should be at least the highest parallelism you want to run
It is a simplification, but a useful one.
Slots are not perfect CPU cages. They are mainly a scheduling and managed-memory sharing mechanism. Multiple subtasks in the same TaskManager still share the same JVM process and machine resources.
How a job gets deployed
A normal deployment path looks like this:
1. You submit a JAR, Python job, or SQL job.
2. Flink builds a logical dataflow graph.
3. The JobManager creates an execution graph.
4. The scheduler asks for slots.
5. TaskManagers provide slots.
6. Subtasks are deployed into those slots.
7. Records start flowing between subtasks.
On Kubernetes or YARN, the resource manager can start containers for Flink. In standalone mode, the TaskManagers are already running and announce their slots to the cluster.
The important point is that Flink is not deploying "one copy of your app" per machine.
It is deploying many subtasks from the job graph across available slots.
Where state lives
Stateless operators can process a record and forget it.
Stateful operators remember something across records.
Examples:
dedup: seen event ids
window: partial aggregate until the window closes
join: records waiting for a match
pattern detection: previous events in the pattern
counter: count per user
Flink manages that state for the operator.
The state backend decides how state is stored locally. A heap backend keeps state as Java objects. A RocksDB-backed state backend stores state in an embedded RocksDB database on the TaskManager's local disk.
That is why Flink can often handle much larger state than an in-memory cache approach. The tradeoff is that disk-backed state adds serialization, IO, checkpoint, and recovery costs.
Checkpoints make state recoverable
If a TaskManager dies, the state on that machine may disappear.
Flink handles this with checkpoints.
A checkpoint is a consistent snapshot of the job's state. Flink periodically records enough information to restart the job from a known point instead of starting from scratch.
Conceptually:
source offsets
operator state
keyed state
timers
are captured together.
When a failure happens, Flink restarts affected tasks and restores their state from the latest successful checkpoint. Sources rewind to the recorded offsets, and the job continues from there.
This is why state, sources, sinks, and checkpoints have to be designed together. A checkpointed Flink job can recover its internal state, but the final end-to-end guarantee also depends on source and sink behavior.
Backpressure is how pain travels upstream
In a streaming job, one slow operator can slow the whole pipeline.
If the sink cannot keep up, its input buffers fill. The upstream operator cannot send as quickly. Then its buffers fill. Eventually the source slows down too.
source -> parse -> window -> sink
slow ^
That is backpressure.
Backpressure is not always bad. It is how Flink avoids endlessly buffering records in memory when the downstream side is saturated.
But it tells you where capacity is missing:
- a hot key is overloading one subtask
- a sink is too slow
- checkpointing is blocking progress
- RocksDB state is doing too much IO
- network shuffle is saturated
The runtime graph helps you see which subtasks are busy, backpressured, or idle.
How Flink scales
Flink scales by splitting the graph.
more source partitions
more operator subtasks
more task slots
more TaskManagers
partitioned keyed state
checkpointed recovery
For stateless work, scaling is mostly about adding more parallel subtasks and enough CPU/network.
For keyed stateful work, scaling also depends on whether keys are evenly distributed and whether state can be redistributed cleanly.
Flink uses key groups as the unit for redistributing keyed state. The maximum parallelism controls how many key groups exist. When you rescale a job from parallelism 8 to 16, Flink can move key groups to new subtasks during restore.
That is powerful, but not instant magic. Rescaling large state means reading, moving, and restoring a lot of data.
The whole picture
A Flink job starts as code, but the runtime sees a distributed graph:
logical code
-> operators
-> subtasks
-> chained tasks
-> task slots
-> TaskManagers
-> checkpointed state
If you remember only one thing, remember this:
Flink scales by partitioning the stream and the state, then running parallel subtasks across task slots while the JobManager coordinates scheduling and recovery.
That is why it shows up in systems that need more than a worker loop and a cache.
For small retry dedup, Redis may be simpler. For stream-native dedup, windows, joins, and event-time logic with replay and recovery, Flink gives you a runtime designed around those problems.
Sources
Field notes
Sign in to leave a comment.
Sign in