← CatZep/Engineering Guides/Reference/System Internals Guides
A developer's guide to how Kafka actually works under the hood: the distributed commit log, brokers and partitions, on-disk segments, replication and the in-sync replica set, and how records flow from producer to consumer.
Student edition · understand, apply, verify
Understand a partition log, replica visibility, producer retries, and consumer recovery without conflating them.
What you should be able to do: Choose explicit acknowledgement/isolation/replay policies and advance positions correctly even when offsets have gaps.
Assumes basic programming and HTTP. Product defaults, formats, APIs, and guarantees depend on the deployed release/configuration. This is an architecture tutorial; pseudocode and local models are not production integrations.
Apache Kafka is a distributed, durable, append-only commit log built for high throughput and strict per-partition ordering. It is not a queue in the traditional sense and not a database — it is a replicated log that producers append to and consumers read from at their own pace. Almost every design decision follows from one core abstraction: the partition is an ordered, immutable sequence of records identified by monotonically increasing offsets. Understanding Kafka means understanding how that log is partitioned across brokers, persisted on disk, replicated for durability, and consumed in parallel.
Read these definitions first, or use the linked terms as you go. The examples and limitations are local to this guide; companion pages are optional.
An offset is a position in one partition log. Commit the next processed position, not simply a count of visible records. Filtering/compaction can leave gaps.
Example. A batch with offsets 100 and 105 resumes at 106, not previous+2.
Boundary. Kafka HW is an exclusive boundary; read_committed also uses LSO. Group ownership does not eliminate replay or transactionalize external effects.
Broker-recognized producer identity/epoch/partition sequences deduplicate supported Kafka retries. Transactions coordinate Kafka outputs and consumer offsets.
Example. Commit a transformed output and matching input positions together; read_committed filters uncommitted/aborted records.
Boundary. Fresh re-emission and arbitrary external writes are not covered automatically. Identity, source replay, isolation, and retention/failure policy matter.
Backward compatibility lets new readers read old data; forward lets old readers read new data. Field identity/default rules differ by format.
Example. Milliseconds changed to seconds under an unchanged integer field is a breaking semantic change.
Boundary. Optional is not universally safe. Registry checks are configured structural rules, not business semantics; schema-ID headers are format/vendor conventions, not Kafka itself.
An offset is a position in one log partition. A checkpoint is a recoverable, consistent record of input positions and processing state. On restart, state must correspond exactly to the input that is considered processed.
Example. If a batch increments a total and crashes before saving its offset, replay increments twice. Commit the total, dedup state, and partition position together, or use a compatible transactional/idempotent sink.
Boundary. Saving an offset after a write avoids that specific loss window but can still duplicate; framework checkpoints alone do not make arbitrary external effects transactional.
Backpressure tells an upstream producer that the downstream stage cannot accept more work. Buffering delays that signal; shedding deliberately rejects or drops lower-priority work when a bound is reached.
Example. A sink accepts 100 events/s while input is 150/s. A 1000-event queue fills in 20 seconds. The system must then block, reject, or apply its documented drop policy.
Boundary. A finite queue cannot absorb an infinite outage. Prioritized messages still need hard quotas, deadlines, and bounded client queues; otherwise an exemption can exhaust the system.
A WAL records updates before data pages are considered committed. Recovery replays it after a crash. fsync (or an equivalent storage contract) requests that buffered bytes reach durable storage rather than only process/OS memory.
Example. A server acknowledges after its WAL commit and required replica writes are durably accepted. A process crash can then reconstruct the committed update.
Boundary. Appending to a file alone does not guarantee power-loss durability. Device behavior, filesystem ordering, replicas, corruption, and the declared fault budget still matter.
Every internal decision in Kafka traces back to a small set of goals. Keeping them in mind makes the rest of the architecture predictable.
| Goal | How Kafka achieves it |
|---|---|
| Durability | Replica acknowledgement, OS cache, fsync, and crash model are separate. Configure storage/replication policy for required durability rather than assuming every record is fsynced before ACK. |
| Strict ordering | Within a single partition, records are stored and delivered in the exact order they were appended. Offsets never go backwards. |
| High throughput | Sequential disk writes, batching, compression, the OS page cache, and zero-copy reads. Disk is treated as a fast sequential medium, not random-access storage. |
| Horizontal scalability | A topic is split into partitions spread across brokers. Partitions are the unit of both parallelism and ordering. |
| Replay & multiple readers | Reading does not consume. The log is retained; each consumer tracks its own offset, so many independent readers can re-read the same data. |
The single most important mental model is the log itself: an append-only, ordered file where each record gets the next integer offset. Producers only ever write to the tail. Consumers read forward from a position they control. Because a read is just "give me records starting at offset N," the same record can be delivered to many consumers, re-read after a failure, or reprocessed days later — as long as it is still within the retention window.
A Kafka cluster is a set of brokers — servers that store partition data and serve produce and fetch requests. Unlike a fully peer-to-peer system, Kafka has explicit roles: one broker is the leader for a given partition while others are followers, and a small controller quorum manages cluster metadata and leader assignment.
The key components are:
Historically Kafka stored all cluster metadata (broker membership, topic configs, partition leadership, ISR state) in an external ZooKeeper ensemble, and a single elected controller broker watched ZooKeeper for changes. That added an operational dependency and made metadata changes a bottleneck at large partition counts.
Modern Kafka uses KRaft (Kafka Raft), which removes ZooKeeper entirely. A dedicated set of controller nodes runs a Raft consensus quorum and stores all metadata in an internal, replicated metadata log — the controllers literally use a Kafka-style log to record metadata changes. Brokers subscribe to that metadata log and apply updates as events. The result is faster failover, far higher partition scalability, and one fewer system to operate.
Each partition replica on a broker is a directory on disk. The log is not one giant file — it is split into segments, and each segment is accompanied by sparse index files that make random lookups by offset or timestamp cheap.
.log plus sparse .index and .timeindex files. Old data is reclaimed by retention (delete) or compaction (keep latest per key).The on-disk structures are:
.log file: the actual records, appended sequentially. The file name is the base offset of the first record it contains. Only the newest segment — the active segment — is written to; older segments are immutable, which makes them safe to read, cache, and delete wholesale..index (offset index): a sparse map from a relative offset to a byte position in the .log file. It holds one entry every few kilobytes of data, not one per record. To find offset N, Kafka binary-searches the index for the closest preceding entry, seeks to that byte position, then scans forward in the log. Because it is sparse it is tiny and can be memory-mapped..timeindex (time index): a sparse map from a timestamp to an offset, enabling "give me records from time T" lookups (used by time-based seeks and retention).Reclaiming space happens in one of two ways, chosen per topic by its cleanup.policy:
| Policy | What it does | Best for |
|---|---|---|
| delete (retention) | Deletes whole segments once they are older than retention.ms or the partition exceeds retention.bytes. Deletion is by segment, never by individual record, which keeps it cheap. | Event streams where only recent data matters. |
| compact (log compaction) | A background cleaner rewrites segments keeping only the latest value per key. Older values for the same key are discarded; a record with a null value (a tombstone) marks a key for deletion. | Compacted state and tombstone/offset retention have configured horizons. Reconstruct only from retained compatible history; deletion/retention can remove data needed by old consumers. |
A producer serializes keys/values, selects a partition, batches, and applies its acknowledgement/retry policy. Keyed Java-client default hashing is commonly Murmur2-based; other clients, custom partitioners, explicit partitions, and partition-count changes can differ. Per-key order needs stable mapping and appropriate producer retry/order settings, not just a hash formula.
acks=0 does not wait for broker acknowledgement; acks=1 waits for the leader's configured append response; acks=all uses the in-sync-replica acknowledgement contract. min.insync.replicas, placement, leader-election policy, storage, and producer settings define the declared fault model. Acknowledgement is not an unconditional disk-fsync promise.
Idempotent producer IDs/epochs/sequences remove supported retry duplicates, not fresh application re-emissions as a new logical event. Transactions can coordinate Kafka outputs and input offsets with read_committed consumers; an external database/API needs its own sink protocol.
Kafka records contain opaque bytes. Avro, Protobuf, JSON, and other encodings/registries are choices, not mandatory Kafka wire formats. A magic-byte/schema-ID header is a registry encoding convention. Reader/writer schema resolution, defaults, units, semantics, and transitive compatibility need explicit validation.
Each partition has a leader and configured replicas. Followers fetch data; an ISR tracks replicas meeting the applicable synchronization criteria. LEO is next append position. High watermark is an exclusive committed visibility boundary: records with offset below it are eligible under ordinary fetch isolation. read_committed also respects the last stable offset and filters aborted transactions.
acks=all uses the ISR contract and min.insync.replicas rather than requiring every configured replica. Replicated OS/file state is not universally per-record fsync; durability depends on supported settings and the failure model. Safe leader election and retained valid copies matter.
Consumers usually fetch leaders; supported follower fetching can serve eligible reads in some deployments. 'Followers exist only for redundancy' is not universal. Leader epochs and recovery truncate incompatible uncommitted state under protocol rules, not simply by choosing the highest timestamp.
Append a batch, replicate under the current ISR/epoch, and complete the requested acknowledgement policy. Because HW is the next-offset/exclusive boundary, record offset o is below it only when o<HW—not o≤HW. Keep transactional visibility separate from replicated append visibility.
A broker append may first reach OS cache; do not label every leader append durable to power loss. A timeout can leave a batch present/committed, so supported retry identity and the application event ID matter. Replicas report positions through their protocol; 'is_online' is not evidence a write completed safely.
# Illustrative position rule, not a broker implementation:
record_offset = 105
high_watermark = 106
visible = record_offset < high_watermark
assert visibleWithin a conventional group, each assigned partition has one active consumer owner, but redelivery/rebalance and asynchronous work can still repeat or complete out of order. More consumers than partitions do not add useful conventional partition readers. Protocol/assignor defaults vary by client and release; new group/shared-consumption features require their own contract.
Compacted state and tombstone/offset retention have configured horizons. Reconstruct only from retained compatible history; deletion/retention can remove data needed by old consumers.
For external effects, commit-after-write permits replay and needs idempotency/transactional state; commit-before-write permits loss. Kafka transactions can couple Kafka outputs and offsets, with read_committed downstream isolation, not arbitrary external systems. Retention can remove unread input; backpressure alone does not preserve unlimited history.
Transactions exist to make the consume-process-produce loop atomic: read from an input topic, do some work, write to output topics, and commit the input offsets — all as one unit, so a crash never produces duplicates or loses progress. They are what turns Kafka's at-least-once defaults into true exactly-once semantics (EOS) across topics.
Transactions build directly on the idempotent producer from §4. Idempotence (PID + per-partition sequence numbers) already removes duplicates from producer retries, but only within one session and one partition at a time. Transactions extend that guarantee to span multiple partitions and to survive a producer restart, by adding a stable identity and a coordinator that runs a two-phase commit.
| Piece | Role |
|---|---|
transactional.id | A stable, user-chosen id that outlives a process restart. It maps to a PID and an epoch, which is how zombies get fenced. |
| Transaction Coordinator | A broker-side module that drives each transaction. The coordinator for a given transactional.id is the leader of the __transaction_state partition the id hashes to. |
__transaction_state | An internal compacted topic — the durable transaction log recording each txn's state and the set of partitions it touched. |
| Control markers | Special commit/abort control records the coordinator writes into every data partition the txn wrote to. |
isolation.level=read_committed | Consumer setting: deliver only committed data, filtering aborted and still-open transactions. |
__transaction_state (the commit point), writes commit markers to every involved partition including __consumer_offsets, and finishes with CommitComplete.The commit is a genuine two-phase commit, with the transaction log as the decision record:
EndTxn(commit=true) to the coordinator.__consumer_offsets partition holding the input offsets.read_committed consumers exclude aborted/uncommitted records. Cross-partition observation is not automatically one globally simultaneous snapshot; use the supported transaction/isolation protocol.
Two mechanisms make it correct:
initTransactions() bumps the epoch for the transactional.id. If an old, hung, or partitioned producer with the same id later tries to write, the broker sees the stale epoch and rejects it with ProducerFenced — so a zombie cannot corrupt a transaction after a failover has taken over its id.read_committed consumer can only read up to the LSO — the offset below which every transaction is decided. The broker will not advance the LSO past an open transaction, so a long-running transaction stalls downstream readers. Keep transactions short; aborted records still occupy log space (filtered on read via an aborted-transaction index) until retention or compaction reclaims them.processing.guarantee=exactly_once_v2 and manages the transactional.id lifecycle for you. EOS is opt-in because it costs real overhead — coordinator RPCs, the two-phase commit, a marker per partition, and added read latency from the LSO — so you pay for it only where duplicates are unacceptable.Sparse indexes find a candidate log byte position, then fetch logic scans/filters valid batches under offset, isolation, size, and protocol rules. OS page cache often makes tail reads efficient. sendfile/zero-copy is a supported optimization on suitable paths, not a universal guarantee under TLS, transformations, platform differences, or every client protocol.
Compaction and transaction filtering create offset gaps without changing offset identities. Do not assume an immutable segment is never rewritten by a cleaner or that a null-value tombstone persists forever. Define retention/compaction/replay horizons and recovery policy.
Work through the following bounded example and its failure boundary.
| Step | Evidence, decision, and expected result |
|---|---|
| Read visible offsets 100 and 105 | Compaction, transaction/control records, or filtering can leave gaps. Offset identifies a log position, not dense record count. |
| Process both records | Durably couple external effect/contribution and the recovered input position, or use a compatible idempotent/transactional sink. |
| Commit next offset 106 | Do not calculate old_offset+len(records). Preserve per-partition order of completed work and avoid committing past unfinished records. |
| Replay after a lost response | Supported producer retry idempotence differs from application re-emission. A fresh logical attempt may duplicate unless its stable event identity is deduplicated. |
It waits under the in-sync-replica acknowledgement contract, whose minimum/configuration matters. That is not a universal per-record fsync or survival guarantee for every fault.
Download guide_labs.py. Python 3.10+, standard library only. In its directory, run:
python3 guide_labs.py log
python3 guide_labs.py testThe model creates offset gaps and checks next-position commits, replica visibility, and transactional local contributions.
The program tests its stated finite contracts. It is an educational model, not an implementation of the named production product or a proof of unmodeled distributed faults.
The whole system is built from a few reinforcing ideas:
| Concern | Mechanism |
|---|---|
| What is the core abstraction? | An append-only, ordered, offset-indexed commit log, split into partitions. |
| How does it scale and order? | Partitions: the unit of parallelism and the unit of ordering. Order holds within a partition, never across. |
| How is data stored and reclaimed? | Immutable segments with sparse offset/time indexes; reclaimed by retention (delete) or compaction (latest per key). |
| How do producers control durability? | acks 0/1/all, batching with linger.ms, idempotent IDs, and optional transactions. |
| How is data made durable? | Leader/follower replication; the ISR plus the high watermark define what is committed and readable. |
| How do many readers share work? | Consumer groups assign one partition to one consumer; offsets are committed to __consumer_offsets. |
| How is it so fast? | Sequential I/O, the OS page cache, and zero-copy sendfile from cache straight to the socket. |
| How do brokers coordinate? | A KRaft (Raft) controller quorum holds metadata and assigns leaders — no more ZooKeeper. |