← CatZep/Engineering Guides/Reference/System Internals Guides

← Back to index

Apache ZooKeeper — Internal Architecture

A developer's guide to how ZooKeeper actually works under the hood: the ensemble and quorums, the znode data model, the ZAB consensus protocol, the write and read paths, sessions and watches, and how it persists and recovers state.

Apache ZooKeeper is a small, strongly-consistent coordination service — the kernel that other distributed systems build on for the hard problems they would rather not solve themselves: leader election, configuration, distributed locking, group membership, and service discovery. It deliberately does very little. It exposes a tiny filesystem-like API over a tree of small data nodes, guarantees that every server in the cluster sees updates in the same total order, and lets clients put watches on nodes to be notified of change. Everything else — locks, queues, barriers, leader election — is a recipe built on those primitives. Understanding ZooKeeper means understanding two things: the data model clients see, and the ZAB protocol that keeps every replica's copy of that data in agreement.

Concepts explained in this guide

Read these definitions first, or use the linked terms as you go. The examples and limitations are local to this guide; companion pages are optional.

1. Design Goals and Core Ideas

ZooKeeper's design is shaped by a refusal to be a general-purpose database. It is a coordination kernel, and every decision serves that narrow goal. Keeping these in mind makes the rest predictable.

ZooKeeper is a CP system in CAP terms: it favors consistency and partition tolerance. If a server cannot reach a quorum, it stops serving writes rather than risk divergence. It is built for small amounts of critical metadata read and watched by many clients — not for bulk data.

2. Architecture: The Ensemble

ZooKeeper uses leader/follower/observer roles under its protocol. Elections can temporarily leave no serving leader; stale prior leaders may remain unaware. Current authority and valid quorum/recovery rules govern commits.

The pieces that make this work:

• A serving leader orders writes under the ensemble's valid epoch/quorum protocol. Do not confuse this with every process always having one identical current view.
• Followers: replicate the leader's transaction stream, vote in elections and on proposals, and answer client reads from their local in-memory tree. A follower forwards any write it receives to the leader.
• Observers: like followers but non-voting. They hear committed transactions and serve reads, so adding them increases read throughput without slowing down the quorum (a larger voting set means more acks per write).
• Sessions: a client connects to one server and establishes a stateful session over TCP, kept alive by periodic heartbeats. The session — not the connection — is the unit of identity; it can survive reconnection to a different server.

3. Quorums and Why Odd Numbers

Majorities of one fixed voting membership intersect. With 2f+1 voters, f may be unavailable while a quorum remains. This arithmetic supports ZAB's durable election/recovery rules; it does not alone prove no conflicting decision or stop an unaware old leader.

Odd membership is an efficiency choice, not a way to avoid every partition or guarantee an active leader at every instant. Observers add read capacity without votes. Safe membership changes and surviving capacity are part of the operational contract.

4. Data Model: znodes and the Namespace

ZooKeeper presents data as a hierarchical namespace very much like a filesystem. Every node in the tree is called a znode and is addressed by an absolute, slash-separated path such as /app/servers/host-A. The crucial difference from a real filesystem is that every znode can hold both data and children — there is no distinction between files and directories.

Two properties of the model matter for how you use it:

• Small data. Each znode stores a small byte payload — by default capped at roughly 1 MB, but in practice you store a few hundred bytes (a hostname, a config value, a lock holder's id). ZooKeeper is metadata, not bulk storage; the entire tree lives in memory on every server.
• Rich stat metadata. Every znode carries a stat structure: data and child version numbers, ACLs, timestamps, and the zxids of its creation (czxid) and last modification (mzxid). The version number powers optimistic concurrency — a setData can require an expected version and fail if another client changed the node first.

5. znode Types and Watches

Persistent nodes last until deletion; ephemeral nodes are tied to sessions. Sequential creation appends a scoped sequence component, but counters/namespace lifetime are finite and parent recreation/wrap/recovery matter. Do not claim eternal global nonreuse from a sequence suffix alone.

Classic watches are one-shot notifications that tell a client to reread; supported persistent/recursive watches have different behavior. A watch is not a durable lossless event queue: reconnect, missed intermediate changes, and session expiry need a resync policy. Check state and install/re-arm the watch through supported atomic API semantics, handling the node disappearing between steps.

6. The ZAB Protocol

The engine that keeps every replica's copy of the tree identical is ZAB — the ZooKeeper Atomic Broadcast protocol. ZAB is not Paxos, though it solves a related problem; it is a primary-backup atomic broadcast designed specifically so that a recovered ensemble re-applies exactly the same prefix of transactions in exactly the same order. It runs in two modes that alternate: leader election / recovery, and broadcast.

The zxid

Everything in ZAB is ordered by the zxid (ZooKeeper transaction id), a 64-bit number split into two halves. The high 32 bits are the epoch — incremented every time a new leader is elected. The low 32 bits are a counter that the leader increments for each transaction it proposes within its epoch. Comparing zxids is just comparing 64-bit integers: a higher epoch always wins, and within an epoch the higher counter wins. This single number is how servers decide who has the most recent state and how they re-apply history deterministically.

Leader election (Fast Leader Election)

When a server starts, or loses contact with the current leader, the ensemble runs Fast Leader Election. Each server votes for a candidate and broadcasts that vote; servers update their own vote toward whoever has the most up-to-date history. The comparison key is (epoch, last-zxid, server-id) — the server with the highest last-committed zxid wins, because choosing the most current server guarantees no committed transaction is lost. Once a server sees a quorum agree on the same candidate, that candidate becomes leader.

Recovery and atomic broadcast

Before serving writes, a new leader synchronizes followers up to its own history (the recovery phase), establishing a clean starting point in a new epoch. Then it enters broadcast mode, where every write is a two-phase commit:

1. The leader assigns the write the next zxid and sends a PROPOSE to all followers.
2. Each follower writes the proposal to its transaction log (with an fsync) and replies ACK.
3. Once a quorum has acked, the leader broadcasts COMMIT, and every server applies the change to its in-memory tree in zxid order.

# Leader broadcast loop (simplified)
function broadcast(txn):
  zxid = (current_epoch << 32) | (++counter)   # total order
  proposal = (zxid, txn)
  log.append(proposal); fsync()                # leader logs first
  for f in followers:
    send(f, PROPOSE, proposal)
  wait until acks >= quorum                     # majority of voters
  for f in followers:
    send(f, COMMIT, zxid)
  apply_to_memory(txn)                          # in zxid order, on every server

7. Write Path

Because there is exactly one leader, the write path always converges on it. A client may be connected to any server, but only the leader can originate a transaction, so a write received by a follower is forwarded.

The end-to-end sequence for a create or setData:

1. The client sends the write to its connected server.
2. If that server is a follower, it forwards the request to the leader.
3. The leader assigns the next zxid and sends a PROPOSE to all followers, logging it durably itself.
4. Each follower fsyncs the proposal to its transaction log and returns an ACK.
5. On a quorum of acks, the leader sends COMMIT; every server applies the change to its in-memory tree, and the originating server replies success to the client.

# From the client's connected server's perspective
function handle_write(req):
  if not is_leader:
    forward_to_leader(req)            # followers cannot originate writes
  zxid, ok = leader.propose_and_commit(req)   # PROPOSE -> quorum ACK -> COMMIT
  if ok:
    apply_to_memory(req)              # applied in zxid order, everywhere
    return SUCCESS(zxid)
  return FAILED                       # e.g. no quorum reachable

The key durability point: a follower must persist a proposal to disk (fsync) before acking it. Combined with the quorum rule, this guarantees that any committed transaction survives the failure of a minority of servers — it is already on stable storage on a majority.

8. Read Path and Consistency

Ordinary reads use the connected server's local state and can lag other clients' committed writes. ZooKeeper's documented write ordering/session guarantees are not an unconditional linearizable read service. A client's own ordered write/read behavior is distinct from a globally latest read.

sync requests a supported synchronization barrier before a subsequent read, but do not label it 'now guaranteed current' under all leader/failure/timing conditions. Use the actual release/client contract and coordination protocol for cross-client freshness. Critical external effects need version/ownership conditions and fences, not a stale local read as authorization.

9. Sessions and Ephemeral Nodes

A negotiated session is renewed through supported client traffic/heartbeats. Expiry is an authority decision committed under the ensemble's protocol; it is not proof a client process died. Disconnected/paused clients may still try old effects after a new holder proceeds.

Reconnect can preserve a still-valid session, but watch/data state must follow client resynchronization rules. Expiry removes ephemeral nodes through committed changes when the ensemble can progress. A permanent loss of quorum can delay cleanup. Namespace payloads are small coordination data, not a bulk store; packet/data limits and ACL/version fields must be checked for the deployed release.

10. Persistence and Recovery

Protocol logging must preserve proposal/election state under its configured durable-storage contract. Append without fsync/equivalent does not prove power-loss safety; settings and devices matter. ZAB recovery synchronizes the agreed history before serving, not arbitrary replay of every locally present uncommitted proposal.

Snapshots may be fuzzy while writes continue. Supported snapshot/log recovery uses version/order bookkeeping to reconstruct valid state; application create/delete operations are not universally idempotent by themselves. Retain compatible snapshot/log coverage and test restore; do not purge merely because any newer file exists.

11. Classic Recipes

Use reviewed ephemeral/sequential lock/election recipes with session/connection error handling and bounded deadlines/cancellation. On uncertain node creation, discover your own uniquely identified contender rather than creating unbounded duplicates. Compare child basenames with the created basename, not a full path versus a name.

Watch the relevant predecessor, then reread/recheck after notification or disappearance. Do not infer ownership just from one watch event; other contenders, deletion, reconnection, and session loss change state. Release only your own node. Install a durable nonreused fencing identity at external resources before trusting the new holder's effects.

Configuration watches need a current state reload and version guard. They reduce polling, not every possible missed intermediate update. Sequential naming and zxids have scope/lifecycle constraints; avoid promising globally eternal IDs.

Worked example: a session expires while its holder is paused

Use this example to check which promise each mechanism actually provides.

Run and change the example

Download guide_labs.py. Python 3.10+, standard library only. In its directory, run:

python3 guide_labs.py coordination
python3 guide_labs.py test

The model demonstrates scoped version conditions, stale read snapshots, and installed-token rejection.

The program tests its stated finite contracts. It is an educational model, not an implementation of the named production product or a proof of unmodeled distributed faults.

12. Summary

ZooKeeper is small on purpose. A few reinforcing ideas account for the whole system: