← CatZep/Engineering Guides/Reference/System Internals Guides

Elasticsearch — Internal Architecture

A developer's guide to how Elasticsearch actually works under the hood: the distributed layer over Lucene, node roles and cluster state, shards and routing, the inverted index, the indexing and query paths, and how near-real-time search is achieved.

Elasticsearch is a distributed search and analytics engine built on top of Apache Lucene. Lucene is a single-machine, full-text search library: it stores an inverted index and answers term queries fast, but it knows nothing about clusters, networks, or replication. Elasticsearch wraps many Lucene indexes and adds the distributed half: it shards data across nodes, replicates each shard for safety, elects a master to manage cluster state, and exposes everything through a JSON-over-HTTP API. Understanding Elasticsearch means understanding how that distributed layer coordinates a fleet of independent Lucene engines.

Concepts explained in this guide

Read these definitions first, or use the linked terms as you go. The examples and limitations are local to this guide; companion pages are optional.

1. Design Goals and Core Ideas

Every design choice in Elasticsearch follows from one ambition: take Lucene's fast single-machine full-text search and make it distributed, horizontally scalable, and near-real-time, without giving up the rich query and aggregation capabilities developers expect.

2. Cluster Architecture and Node Roles

An Elasticsearch cluster is a set of nodes that share a cluster name. Unlike a fully peer-to-peer system, Elasticsearch has an elected master that owns the cluster's metadata. The master does not sit in the data path — it manages the cluster state: the list of nodes, index mappings and settings, and which shard lives on which node. Every node holds a copy of the cluster state, and the master publishes updates to all of them.

Nodes are configured with one or more roles:

• Master-eligible: can be elected master. The cluster elects exactly one active master from this pool. The master applies cluster-state changes — creating indices, allocating shards, processing node joins and departures — and publishes the new state. Master-eligible nodes form a voting configuration; a quorum of them must agree to elect a master and to commit each state change.
• Data: holds shards (primaries and replicas) and does the actual indexing and searching. This is where Lucene runs and where disk and memory pressure live.
• Coordinating: a role every node can play. The node that receives a client request becomes the coordinating node for it: it routes sub-requests to the right shards, gathers responses, merges them, and replies. A node with no other roles is a dedicated coordinating-only node — a smart load balancer.
• Ingest: runs ingest pipelines that transform or enrich documents before they are indexed (parsing, field renaming, lookups), offloading that work from data nodes.

Discovery is how nodes find each other and form the cluster. On startup a node contacts a configured list of seed hosts, learns about other master-eligible nodes, and participates in master election. Once a master is elected, it admits new nodes by adding them to the cluster state and publishing it. Failure detection runs continuously: the master pings the other nodes and they ping the master; a node that stops responding is removed from the cluster state, and the master reallocates its shards.

3. Indices, Shards and Routing

An index is a logical collection of documents (for example, all your logs). Physically, an index is split into primary shards, and each primary can have one or more replica shards. A shard is a complete, independent Lucene index. Splitting an index into shards is what lets a single index outgrow one machine and lets queries run in parallel.

Which shard does a document go to? Elasticsearch routes it deterministically:

function route(document):
  routing = document._routing or document._id   # default routing key is the doc id
  shard   = hash(routing) % num_primary_shards  # pick exactly one primary
  return shard

Stable routing under the configured shard/routing-partition scheme narrows reads. Changing schemes and routed-partition options require supported migration semantics.

A replica maintains compatible logical shard operations/state; segment file layouts may differ under operation-based replication. Its visibility and recovery depend on checkpoints and reader refresh.

4. Lucene Internals: The Inverted Index

At the bottom of every shard is Lucene, and at the heart of Lucene is the inverted index. A forward index maps a document to its words; an inverted index maps each term to the list of documents that contain it (the postings list). That inversion is what makes "find every document containing fox" a single dictionary lookup instead of a scan.

Lucene stores this efficiently:

• Term dictionary: the sorted set of all unique terms in a field. Because it is sorted, Lucene can binary-search it and supports range and prefix queries cheaply.
• Postings: for each term, the list of document ids that contain it, often with the term frequency and the positions of each occurrence. Frequencies feed relevance scoring; positions enable phrase and proximity queries.
• Segments: a Lucene index is not one big file. It is a set of immutable segments, each a small, self-contained inverted index. New documents go into new segments; existing segments are never modified. Immutability is the key trick — a segment needs no locking, can be cached aggressively, and is safe to read concurrently.

Because segments are immutable, a delete cannot remove a document in place. Instead Lucene records the doc id in a per-segment deleted-documents bitset and filters it out of results. An update is just a delete plus a fresh insert. The space is reclaimed later, during merge.

5. Indexing Path

The primary validates/applies a document and records recoverable operation state under configured translog durability and replica acknowledgement. An append alone is not power-loss durability. Request versus asynchronous translog modes and in-sync replica changes affect the fault contract.

Refresh opens a new search view; an old reader does not change automatically. Flush performs a durable Lucene commit and safely advances recovery-log cleanup under engine rules; it is not a blind truncate of all concurrent operations. Merge rewrites segments and preserves readers' old file lifetimes.

Successful indexing does not universally mean immediate search visibility. GET can use a different real-time path. Use refresh=wait_for or another supported policy when search visibility after a write is required, and state the latency/load cost.

6. Near-Real-Time Search

Search readers are snapshots. New indexed state becomes searchable through the supported refresh/reopen path, while configured translog/replica durability can be established separately. An old reader can still return prior state; GET-by-ID may use a different real-time path.

refresh_interval and explicit/wait-for refresh policies trade freshness against segment/merge costs, and defaults can depend on deployment/activity/release. Do not promise exactly one second or immediate power-loss durability for every write. Asynchronous translog policies can have a loss window.

Use the actual durability, replica, search visibility, and concurrent-version conditions required by the API. Retained reader lifetimes and partial shard responses remain part of the query contract.

7. Replication Model

Elasticsearch replicates each shard with a primary-backup model. All writes for a shard go to its primary first; the primary then forwards them to the replicas. This is different from Cassandra's quorum model — there is a designated primary that orders the writes, which gives Elasticsearch a clear, single source of truth for each shard.

The write sequence for a single document is:

function replicate_write(doc):
  primary.validate(doc)
  primary.index(doc)                       # 1. apply locally, assign a seq#
  for replica in in_sync_replicas:
    send(replica, op = doc, seq = seq_no)   # 2. forward the same operation
  wait_for_replica_acks()                   # 3. replicas apply and ack
  return ack_to_client()                    # 4. ack once replicas confirm

Two bookkeeping concepts keep replicas in step and make recovery cheap:

Recovery can replay retained compatible operation history when identity/checkpoints/history retention allow it; otherwise the supported protocol may use file-based recovery. A brief outage alone does not guarantee one fixed translog-only path.

8. Query Path

A coordinator targets relevant shard copies, pruning by routing, indices, and other supported constraints. Query phase returns hit identifiers/scores; fetch phase hydrates selected hits. Distributed top-k is meaningful under compatible scoring/statistics; shard-local statistics can differ from globally scored retrieval.

Check failed/partial shard responses and timeout flags before reporting totals as exact. Aggregations have their own merge/approximation rules. Deep from/size pages create large candidate work; use supported search_after/point-in-time semantics and deterministic ordering for stable paging, with current authorization.

Query-time analyzers can intentionally differ from index-time analysis, for example synonyms, if they produce compatible matching terms. Keyword fields may have configured normalizers; 'keyword always stores verbatim' is too broad.

9. Analysis: Turning Text Into Terms

Before text can go into an inverted index, it must be broken into terms. That job belongs to an analyzer, a pipeline of three stages applied in order.

The subtle but crucial point is that analysis happens at two times, and they must agree:

• Index time: when a document is indexed, each text field is analyzed and the resulting terms populate that field's inverted index.
• Index/query analysis must be compatible, not necessarily identical. Query-time synonym and other intentional transformations can use a different configured analyzer.

Index/query analysis must be compatible, not necessarily identical. Query-time synonym and other intentional transformations can use a different configured analyzer.

Worked example: an acknowledged write is not in an old search

Work through the following bounded example and its failure boundary.

Run and change the example

Download guide_labs.py. Python 3.10+, standard library only. In its directory, run:

python3 guide_labs.py index
python3 guide_labs.py test

The model holds separate durable and searchable snapshots, then tests conditional version updates.

The program tests its stated finite contracts. It is an educational model, not an implementation of the named production product or a proof of unmodeled distributed faults.

10. Summary

Elasticsearch is best understood as a thin-but-clever distributed layer over many Lucene engines. The pieces reinforce each other: