A developer's guide to how Cassandra actually works under the hood: cluster architecture, the storage engine, replication, the Log-Structured Merge tree, and how data moves when nodes join or leave.
Student edition · understand, apply, verify
Separate Cassandra's partition placement, timestamp reconciliation, consistency levels, and crash durability.
What you should be able to do: Reason about a key's replicas and conditional updates without treating quorum overlap or a commit-log append as universal linearizability/power-loss safety.
Assumes basic programming and HTTP. This is an architecture-level tutorial, not a release-specific implementation contract. Defaults, file formats, APIs, and guarantees depend on the deployed release and configuration; use the linked product documentation before production changes. Snippets marked pseudocode are not runnable integrations.
Apache Cassandra is a distributed, wide-column NoSQL database designed for high write throughput, linear horizontal scalability, and no single point of failure. Its design borrows two big ideas: the data distribution and replication model from Amazon Dynamo (consistent hashing, tunable consistency, gossip, hinted handoff) and the on-disk storage model from Google Bigtable (the Log-Structured Merge tree, memtables, and SSTables). Understanding Cassandra means understanding how those two halves fit together.
Read these definitions first, or use the linked terms as you go. The examples and limitations are local to this guide; companion pages are optional.
Consistent hashing
A placement algorithm that maps both keys and server tokens into a shared, ordered hash space. A key belongs to the first token clockwise at or after its hash; the end wraps to the start. Adding a token changes only its newly owned interval, rather than most keys.
Example. Tokens at 10, 40, 80 own positions (80,10], (10,40], and (40,80]. Adding token 25 moves positions (10,25] from token 40 to token 25.
Boundary. It stabilizes placement, not replica consistency. Expected primary remapping when adding an equally weighted node is 1/(n+1); balance depends on token allocation and workload.
Quorum
A set of replicas whose responses are sufficient for an operation. In a fixed set of N replicas, read size R and write size W overlap whenever R+W>N. A majority is one particular quorum, not the definition of every quorum.
Example. With N=3, writes to {A,B} and reads from {B,C} share B. If the write completed durably and replicas preserve a comparable version, B carries that version forward.
Boundary. Overlap alone does not provide linearizability: concurrent or failed writes, sloppy replica sets, and version selection matter. Consensus adds protocol rules to preserve decisions.
LSM trees, memtables, and SSTables
An LSM design buffers updates in a memtable, logs them under durability policy, flushes sorted data files, and compacts versions later.
Example. A key has versions in two SSTables and a memtable. Reconcile versions/deletes instead of returning the first physical match.
Boundary. Efficiency trades against read/write/space amplification. Not all I/O is sequential, and immutable files need safe publication/reader lifetime.
Tombstones
A durable delete marker that carries a version. Replicas merge it like a write so an old live value is not treated as a new record during repair.
Example. A deletes key k while B is offline. A later repair sends the versioned delete to B, removing B's old value.
Boundary. Collecting the marker too early can resurrect data. Safe retention depends on repair completion, replica return policies, and the maximum tolerated offline period.
Schema compatibility and semantic contracts
Backward compatibility lets new readers read old data; forward lets old readers read new data. Field identity/default rules differ by format.
Example. Milliseconds changed to seconds under an unchanged integer field is a breaking semantic change.
Boundary. Optional is not universally safe. Registry checks are configured structural rules, not business semantics; schema-ID headers are format/vendor conventions, not Kafka itself.
Linearizability
Each completed operation appears to occur at one instant between its invocation and response, respecting real-time order. A read that begins after a write finishes cannot return a value older than that write, unless a later write superseded it.
Example. A write x=7 returns success; a later read must return 7 or a value from a later ordered write. A read overlapping that write may return the old or new value.
Boundary. Not the same as durability or transaction serializability. A leader may be stale; linearizable leader reads need current authority and an applied-state barrier.
1. Design Goals and Core Ideas
Every internal decision in Cassandra traces back to a small set of goals. Keeping them in mind makes the rest of the architecture predictable.
Goal
How Cassandra achieves it
No single point of failure
Fully peer-to-peer. Every node is identical; there is no master, no name node, no config server.
Linear scalability
Partitioning can add capacity; actual growth depends on skew, hot partitions, replication, compaction/repair, and the workload.
High write throughput
Writes are append-only (commit log + in-memory memtable). No read-before-write, no in-place updates.
Tunable consistency
The client chooses, per query, how many replicas must respond. Trade latency against consistency.
Availability under the requested consistency policy
A reachable sufficient replica set is required. Quorum/conditional operations can be unavailable during partitions.
2. Cluster Architecture
A Cassandra cluster is a ring of identical, equal peers. There is no coordinator process and no leader election for data ownership. Any node can accept any request; the node that receives a client request becomes the coordinator for that single request and routes it to the nodes that own the data.
Every node is identical and gossips with every other node. The node a client hits becomes the coordinator for that request.
The key components running on every node are:
Coordinator role: a transient, per-request role. The coordinator forwards reads and writes to the correct replicas, gathers responses, and applies the consistency level before replying to the client.
Gossip disseminates observations when communication permits, but a partition can prevent convergence; timeout suspicion is not proof of death.
Snitch: the component that tells Cassandra the network topology — which rack and which datacenter each node lives in. The replica-placement strategy uses this to spread replicas across failure domains.
Partitioner: the hash function (Murmur3Partitioner by default) that maps a partition key to a token, deciding which node owns the row.
3. Data Distribution: Partitioning and the Token Ring
Cassandra distributes data using consistent hashing. The entire output space of the hash function (a 64-bit signed integer for Murmur3) is treated as a ring. Each node is assigned one or more tokens — positions on the ring. A node owns the range of tokens from the previous node's token (exclusive) up to its own token (inclusive).
The partition key is hashed to a token; the token determines which node owns the row.
To locate a row, Cassandra hashes its partition key to a token, then walks clockwise around the ring to find the first node whose range contains that token. That node is the first replica; the next N-1 nodes clockwise hold the additional copies.
A concrete walk-clockwise example
Picture the ring as the numbers 0 to 100 wrapping back to 0, with four single-token nodes: A at token 10, B at 40, C at 70, and D at 100. Each node owns the range from its predecessor's token (exclusive) up to its own token (inclusive), so A owns (100,10], B owns (10,40], C owns (40,70], and D owns (70,100]. If hash('user:42') resolves to 55, you start at 55 and move in the direction of increasing tokens; the first token you reach is 70, so Node C owns the row. For RF=3 you keep walking clockwise to the next distinct physical nodes, which gives the replica set { C, D, A }.
hash('user:42') = 55 lands in C's range (40,70]; the next two distinct nodes clockwise complete the RF=3 replica set.
Virtual nodes (vnodes): naming and mapping
Virtual-token count is configurable and release-specific. More tokens can balance placement but add repair/streaming coordination; a hot key remains indivisible.
A virtual node is not a named process — it is simply one token, identified by its token value (a 64-bit integer for Murmur3). Each physical node is configured with num_tokens and on join is assigned that many tokens. Each assigned token defines one vnode range, so "owning 16 vnodes" means owning the 16 ranges whose upper-bound tokens belong to that node. The cluster maintains a single sorted token-to-endpoint map — each physical node has a host-ID (a UUID) and an address — and gossip distributes this map so any node can resolve any token to its owning machine. The mapping is many-to-one and interleaved: consecutive tokens on the ring usually belong to different physical nodes, which is exactly what spreads data evenly and lets bootstrap and decommission stream from many peers in parallel.
Three physical nodes, each owning several interleaved tokens. Each dot is one vnode (one token); colors show the owning machine.
The routing logic on the coordinator is straightforward:
function route(partition_key):
token = murmur3(partition_key) # 64-bit signed int
primary = ring.find_owner_clockwise(token)
replicas = []
node = primary
while len(replicas) < replication_factor:
if placement_strategy.accepts(node, replicas): # rack/DC aware
replicas.append(node)
node = ring.next_clockwise(node)
return replicas
4. Replication
Replication factor counts distinct configured replicas, usually per datacenter under NetworkTopologyStrategy. Placement uses topology/rack constraints; 'next n nodes' describes only a simplified strategy. Not every configuration can satisfy requested replicas in every failure domain.
For a fixed replica set, R+W>RF proves read/write overlap. It does not automatically promise every read sees the latest logical request under concurrent/partial writes, clock timestamps, topology scopes, and repair policy. Ordinary writes reconcile by timestamp; concurrent valid updates can be discarded. Use supported lightweight/conditional transactions and serial consistency when a business invariant needs linearizable compare-and-set, with its partition/operation scope explicit.
ANY, ONE, QUORUM, LOCAL_QUORUM, and serial settings have different response and topology semantics. A hint can satisfy particular write policies, not every quorum. Minority failures can make the requested CL unavailable; labeling Cassandra simply 'always available AP' is misleading.
Hints, read repair, and anti-entropy repair serve different coverage/horizon roles. Repairs need matching ranges/version rules, retained tombstones, and a documented schedule. Defaults/available consistency and compaction options vary by release.
5. The Storage Engine: Log-Structured Merge Tree
Writes use commit-log and memtable state, then sorted immutable files, and background compaction. This often improves write batching/locality but does not mean every I/O is sequential or every immutable-file read needs no coordination. Safe publication, deletion, and reader/cache lifetimes still matter.
Reads reconcile versions across files and memory. Bloom filters can rule out absence, but a positive result still needs lookup. Compaction merges compatible sorted state and may collect tombstones only when implementation safety conditions, grace/repair policy, and overlapping older data permit it. Blind age-based collection can resurrect deleted values.
STCS, LCS, TWCS, and newer strategies have workload/release-specific costs; do not assert one default across all versions. Measure read/write/space amplification, hot partitions, compaction backlog, and failure headroom.
6. Write Path
A replica's commit-log append, memtable visibility, durable flush completion, and coordinator success are distinct boundaries. Commit-log synchronization mode controls whether that replica acknowledgement includes the promised fsync/equivalent durability; periodic policies can have a loss window. Validate your actual release/settings.
The coordinator replies after the configured consistency-level requirement is met, using valid replica responses rather than merely checking 'is_alive'. Timeouts mean the write outcome may be unknown; retry semantics must account for timestamps, counters, and logical operation identity. Counters and conditional writes have special protocols and cannot be treated as ordinary idempotent upserts.
Flush/compaction publication must precede discarding the covered log state. Cache validity and repair are part of the correct read path; a fast hit is not automatically the latest value under every consistency contract.
7. Read Path
A read is more involved than a write because the current value of a row may be spread across the memtable and several SSTables (each holding a different version or column). The coordinator gathers candidate data from all sources and merges by timestamp — last write wins at the cell level.
Caches and bloom filters cut work; the memtable and surviving SSTables are merged by timestamp into the result.
On a single replica, the local read works like this:
function read(key):
if row_cache.contains(key): return row_cache.get(key) # fast path
candidates = []
candidates += memtable.get(key) # newest, in memory
for sstable in sstables:
if not sstable.bloom_filter.might_contain(key):
continue # skip files that cannot hold the key
offset = sstable.index.lookup(key) # partition index / summary
candidates += sstable.read_at(offset)
row = merge_by_timestamp(candidates) # last write wins; drop tombstoned cells
row_cache.put(key, row)
return row
The bloom filter is the key optimization: it lets a replica skip SSTables that definitely do not contain the requested key, avoiding pointless disk reads. A delete does not erase data; it writes a tombstone that shadows older versions until compaction physically removes them.
8. Membership and Failure Detection
Separate gossip/suspicion from authoritative topology transition. Nodes exchanging status do not prove a common view during a partition. Membership/metadata mechanisms differ across Cassandra releases, including newer coordination/metadata services; do not teach every version as 'gossip alone, no coordination'.
Bootstrap, removal, replacement, and decommission follow supported procedures with streaming, replica coverage, and safe transition gates. A suspected offline node can return; prevent stale ownership/data from bypassing current topology. Benchmark stream/repair load and verify ring/metadata health before declaring a change complete.
9. Adding a Node (Scaling Out)
Adding a node is called bootstrapping. The new node is assigned tokens (with vnodes, many small ranges spread around the ring). It now becomes a replica for those ranges, so it must receive the corresponding data from the nodes that currently hold it before it can serve reads.
The new node takes over token ranges; current owners stream the relevant data to it before it serves traffic.
The bootstrap sequence is:
The new node starts, contacts a seed node, and learns the ring topology through gossip.
It calculates the token ranges it will own and announces a JOINING state.
Neighbors that currently own those ranges stream the relevant SSTable data to the new node, in parallel across many vnode ranges.
Once streaming completes, the node transitions to NORMAL and begins serving reads and writes for its ranges.
The previous owners now hold redundant copies. An operator runs nodetool cleanup on them to delete data they no longer own and reclaim disk.
The old owners keep serving the data until streaming finishes, so there is no availability gap and no data loss during scale-out.
on new_node bootstrap:
contact_seed(); learn_ring_via_gossip()
my_ranges = compute_owned_ranges(my_tokens)
announce(state = JOINING)
for range in my_ranges:
source = current_owner(range)
stream_sstables_from(source, range) # parallel
announce(state = NORMAL) # now serves traffic# operator later: nodetool cleanup on old owners
10. Removing a Node (Scaling In or Replacing Failure)
Removing a node must preserve the replication factor: every range the departing node held must still have RF copies afterward. The procedure depends on whether the node is alive.
A live node decommissions by streaming its data out; a dead node is removed and survivors rebuild the missing copies.
Scenario
Command
What happens
Node is alive (graceful scale-in)
nodetool decommission
The leaving node itself streams each of its ranges to the new owners (the next replica clockwise), then leaves the ring. RF is preserved throughout.
Node is dead (hardware failure)
nodetool removenode <id>
The dead node cannot stream anything. The surviving replicas that still hold copies of its ranges rebuild the now-missing third copy by streaming among themselves.
Replace in place
replace_address flag
A fresh node takes over the dead node's exact tokens and bootstraps the data from the remaining replicas.
Topology changes require supported membership/metadata and streaming procedures. Verify replica coverage and transition completion rather than assuming gossip automatically restores every guarantee.
decommission (node alive):
announce(state = LEAVING)
for range in my_ranges:
new_owner = next_replica_clockwise(range)
stream_sstables_to(new_owner, range)
announce(state = LEFT); shut_down()
removenode (node dead):
for range owned by dead_node:
survivors = live_replicas(range)
new_owner = next_replica_clockwise(range)
stream_sstables(from = survivors, to = new_owner)
Worked example: a later request loses to an older timestamp
A caller writes A with timestamp 200; another later logical operation writes B with timestamp 150.
Sequence for this example. The table explains the evidence and decision at each step.
Step
Evidence, decision, and expected result
Store the two versions
Cassandra's ordinary timestamp reconciliation can prefer timestamp 200 even if B was attempted later in real time.
Read an overlapping quorum
Overlap carries replicas' chosen versions; it does not redefine timestamp order into wall-clock request order or general transactions.
Require a business condition
Use a supported lightweight/conditional transaction and serial consistency where the invariant requires it, following the release's scope and failure semantics.
Repair a delete safely
Retain delete metadata until repair/offline and overlapping-data conditions permit removal; gc_grace age alone is not proof every stale copy is gone.
Check your understanding: Does QUORUM plus QUORUM make every operation linearizable?
No. Specify the replica set, timestamp/concurrency behavior, read-repair policy, failure paths, and conditional transaction semantics. Ordinary overlap is not a general transaction protocol.
Run and change the example
Download guide_labs.py. Python 3.10+, standard library only. In its directory, run:
python3 guide_labs.py replication
python3 guide_labs.py test
The model distinguishes timestamp winner, real-time request order, and replica-set overlap.
The program tests its stated finite contracts. It is an educational model, not an implementation of the named production product or a proof of unmodeled distributed faults.
The whole system is built from a few reinforcing ideas:
Concern
Mechanism
Where does a row live?
Consistent hashing of the partition key onto a token ring; vnodes for even spread.
How many copies?
Replication factor, placed by a topology-aware strategy.
How consistent?
Tunable consistency level per query; R + W > RF gives strong reads.
How are writes so fast?
Append-only commit log + in-memory memtable; no in-place updates.
How is data stored and reclaimed?
Immutable SSTables (LSM), reconciled and compacted in the background.
How do nodes coordinate?
Gossip + Phi Accrual failure detection; no master.
How does it scale?
Bootstrap streams ranges in; decommission/removenode streams them out. RF is preserved throughout.
The recurring theme: every operation is decentralized, append-oriented, and eventually reconciled. There is no master to lose, no in-place mutation to lock, and no rebalancing step that takes the cluster offline.
