← CatZep/Engineering Guides/Reference/System Internals Guides

Apache Flink — Internal Architecture

A developer's guide to how Flink actually works under the hood: the cluster runtime, how a program becomes parallel subtasks, how state lives and survives failures, and how checkpoints, watermarks, and backpressure keep results correct on an unbounded stream.

Apache Flink is a distributed engine for stateful computation over data streams. Where a batch engine processes a finite dataset and stops, Flink treats data as an unbounded stream of events that may never end, and it keeps long-lived state as it goes. Its design rests on three reinforcing ideas: every operator can hold state that survives failures; results are computed in event time (when things happened) rather than the order events happen to arrive; and the combination of checkpoints and replayable sources gives exactly-once semantics for that state. Understanding Flink means understanding how those three pieces fit together on top of a fairly conventional master/worker runtime.

Concepts explained in this guide

Read these definitions first, or use the linked terms as you go. The examples and limitations are local to this guide; companion pages are optional.

1. Design Goals and Core Ideas

Every internal decision in Flink traces back to a small set of goals for processing unbounded streams correctly. Keeping them in mind makes the rest of the architecture predictable.

2. Cluster Architecture

A Flink cluster follows a master/worker pattern. The master is the JobManager and the workers are TaskManagers. Unlike Cassandra's peer-to-peer ring, there is a clear coordination role — but it is split into three distinct responsibilities so that a single failed job cannot take down the whole cluster.

The components are:

• Dispatcher: the cluster entry point. It receives submitted jobs over REST, starts a dedicated JobMaster for each one, and serves the web UI. It is the front door, not the scheduler.
• ResourceManager: manages task slots — the unit of resource a TaskManager offers. It hands free slots to JobMasters that ask for them and, in deployments like YARN or Kubernetes, can request or release TaskManagers dynamically.
• Per-job coordination reduces coupling, but shared TaskManagers/resources/control planes can still cause cross-job failures; isolate and measure under the deployment's fault model.
• TaskManager: a worker JVM (or process) that provides a fixed number of task slots. Each slot runs one parallel slice of the pipeline. TaskManagers register with the ResourceManager, heartbeat to it, and exchange data directly with each other over the network.

A slot allocates configured resources/managed memory and can share operators; it is not universal CPU/OS-process isolation.

Submitting a job proceeds roughly like this:

function submit(job_graph):
  dispatcher.receive(job_graph)            # REST entry point
  jm = dispatcher.start_job_master(job_graph)
  needed = jm.compute_required_slots()     # from operator parallelism
  slots = resource_manager.allocate(needed)
  exec_graph = jm.build_execution_graph(job_graph)
  for subtask in exec_graph.subtasks():
    slot = slots.pick_for(subtask)
    task_manager(slot).deploy(subtask)     # ship code + state handle
  jm.track_status()                        # reschedule on failure

3. The Dataflow Graph

A Flink program is a logical description of operators and the streams connecting them. Before it runs, that description passes through three representations, each lower-level than the last.

• StreamGraph: the direct, logical translation of your program on the client. One node per operator (source, map, keyBy/window, sink), with the edges describing how streams connect.
• JobGraph: the optimized graph the client ships to the JobManager. Its key optimization is operator chaining — fusing operators that have a one-to-one (forward) connection and the same parallelism into a single task, so records pass between them by direct method call instead of crossing a network or serialization boundary.
• ExecutionGraph: the parallel, physical graph the JobMaster builds. Each chained task becomes N parallel subtasks (one per unit of parallelism), each a separate instance running in its own slot. This is the graph that is actually scheduled and deployed.

Edges between operators come in two flavors. A forward edge keeps a record on the same subtask (and enables chaining). A redistributing edge — produced by keyBy, a rebalance, or a change in parallelism — sends records across the network to a different subtask. A keyBy in particular partitions the stream by a hash of the key so that all records for a given key always reach the same downstream subtask, which is what makes keyed state possible.

function chain(stream_graph):              # client side
  for op in stream_graph.operators():
    if op.input_is_forward() and op.parallelism == op.input.parallelism \
       and chaining_enabled(op):
      fuse(op, op.input)                   # one task, no network
  return job_graph                         # chained operators

function parallelize(job_graph):           # JobMaster side
  for task in job_graph.tasks():
    for i in range(task.parallelism):
      execution_graph.add_subtask(task, index=i)
  return execution_graph

4. State and State Backends

State is what separates Flink from a stateless stream filter. Anything an operator needs to remember between events — a running count, the contents of an open window, the last seen value per key — is held as Flink-managed state so the runtime can snapshot, restore, and repartition it. There are two kinds.

Where that state physically lives is the job of the state backend. The choice is purely about capacity and access cost; it does not change semantics.

Keyed state is the reason a keyBy matters so much: because every record for a key always lands on the same subtask, that subtask can keep the key's state locally with no coordination. When the job is rescaled, keys are reassigned to subtasks using key groups (fixed-size buckets of the key space), so state moves in whole buckets rather than key by key.

5. Checkpointing

Checkpoint barriers identify a consistent recovery cut of source positions and managed state. Aligned checkpoints coordinate input barriers; asynchronous snapshots can upload a captured state version while processing continues. Unaligned modes also capture required in-flight channel state. Exact mechanics depend on checkpoint mode and connectors.

Do not acknowledge a durable checkpoint handle before the required snapshot upload/metadata completion succeeds. Recovery restores the completed state/source positions, then replays; physical processing may happen again, while the logical recovered state reflects the correct prefix.

End-to-end effects need replayable retained sources and compatible idempotent/transactional sink commits. Checkpoints do not roll back arbitrary HTTP calls, emails, or uncoordinated database writes. They may align/stall inputs; 'without stopping any stream' is not an absolute performance guarantee.

6. Savepoints vs Checkpoints

Checkpoints are runtime recovery snapshots with a configured lifecycle; savepoints are user-managed snapshots for compatible planned changes. Formats can be canonical/native and depend on backend/release. Portable does not mean any serializer, backend, maximum parallelism, or application version can restore.

Use stable operator UIDs, compatible state schemas/serializers, supported key-group/max-parallelism settings, and an explicitly tested rescale/upgrade/rollback procedure. Stop/drain choices can affect timers, final windows, and side effects. Retain required state until restore is verified.

7. Time and Windows

Event timestamps assign records to windows under the chosen source clock/policy. Watermarks estimate event-time progress; they are not proof all prior events arrived. Multiple inputs, idle sources, skew, and out-of-order data need explicit watermark policy.

Whether a late record is still accepted depends on the target window's end, trigger, allowed lateness, and cleanup horizon. A timestamp below the watermark may still belong to a window that has not expired. Do not universally side-output every event_time≤watermark. Define half-open window boundaries and any correction/retraction behavior.

Keep retained source data, bounded state, and downstream correction semantics aligned. Missing/late telemetry is different from a zero count. Processing-time execution can be useful but may produce different replay results.

8. Network Shuffle and Backpressure

When subtasks are connected by a redistributing edge, records cross the network from a producer subtask to a consumer subtask. Flink moves them in buffers (batches of serialized records) rather than one record per packet, for efficiency. The interesting part is what happens when the consumer cannot keep up.

Flink uses credit-based flow control to handle this natively, with no explicit throttling logic in user code:

• Each consumer subtask has a pool of input buffers. It continuously tells the producer how many free buffers it has — these are its credits.
• The producer only sends as many buffers of data as the consumer has credits for. It never floods a channel the consumer cannot drain.
• If the consumer slows down, its free buffers fill and its credit drops toward zero. The producer, out of credits, stops sending on that channel and its own output buffers begin to fill.
• That pressure propagates upstream subtask by subtask — eventually reaching the source, which slows down its intake. The whole pipeline self-regulates to the speed of its slowest stage.

Credit-based channel buffers are bounded, but sustained pressure can grow source lag beyond retention or hit external limits. Backpressure does not guarantee no end-to-end loss during an unbounded outage.

function producer_send(channel, buffer):
  if channel.credits == 0:
    block()                              # backpressure: cannot send
  else:
    channel.transfer(buffer)
    channel.credits -= 1

function consumer_loop():
  while True:
    buf = input_gate.next_buffer()       # blocks if none ready
    process(buf)
    recycle(buf)
    announce_credit(buf.channel, +1)     # I have a free buffer again

Worked example: late does not always mean discard

Work through the following bounded example and its failure boundary.

Run and change the example

Download guide_labs.py. Python 3.10+, standard library only. In its directory, run:

python3 guide_labs.py windows
python3 guide_labs.py test

The model tests window-end cleanup versus raw timestamp and checkpoint/state replay.

The program tests its stated finite contracts. It is an educational model, not an implementation of the named production product or a proof of unmodeled distributed faults.

9. Summary

Flink's design is a small set of ideas that compose into correct, fault-tolerant stream processing: