← CatZep/Engineering Guides/Reference/System Internals Guides

Apache Lucene — Internal Architecture

A developer's guide to how Lucene actually works under the hood: the inverted index, immutable segments, the analyzer pipeline, the indexing and search paths, BM25 scoring, deletes and merging, near-real-time reopen, and the on-disk codec files that engines like Elasticsearch and Solr are built on.

Apache Lucene is a single-machine, embeddable full-text search library written in Java. It is not a server: it has no cluster, no network layer, and no query language of its own — just an API for building an inverted index on local disk and answering ranked queries against it fast. Engines like Elasticsearch and Solr are the distributed half; Lucene is the search engine inside each shard. Almost everything about Lucene follows from two decisions: the core data structure is an inverted index, and that index is stored as a set of immutable segments. Once you understand those two ideas, the indexing path, deletes, merging, and near-real-time search all fall out as consequences.

Concepts explained in this guide

Read these definitions first, or use the linked terms as you go. The examples and limitations are local to this guide; companion pages are optional.

1. Design Goals and Core Ideas

Every internal choice in Lucene serves fast, ranked retrieval over large text collections from a single process. Keeping the goals in mind makes the rest of the design predictable.

2. The Index Model: Documents & Fields

Documents may omit fields, but field indexing/doc-values/point semantics have compatibility constraints. 'No fixed document schema' does not mean arbitrary type changes are always allowed.

A single field can contribute to up to four different structures, each answering a different kind of question:

• Indexed (inverted) — the field's text is analyzed into terms and added to the inverted index. This is what makes it searchable.
• Stored — the original field value is kept verbatim so it can be returned in a search hit. Storage is for display, not search; stored values are never consulted to match a query.
• Doc values — a per-field column store (document → value) used for sorting, faceting, and aggregations, where you need the value for many documents rather than the documents for a value.
• Points — numeric, date, and geo fields are indexed in a BKD tree (a disk-friendly k-d tree) for efficient range and nearest-neighbor lookups, which an inverted index handles poorly.

The same logical field, say price, is often configured for several of these at once — a point for range filters, a doc value for sorting, and stored for display — because each backs a different access pattern.

3. Analysis: Text to Terms

Text is not indexed as-is. Before it reaches the inverted index it passes through an analyzer, a pipeline that turns a raw string into a stream of normalized terms. The analyzer has three stages.

• Character filters preprocess the raw character stream — for example stripping HTML tags or mapping characters.
• The tokenizer splits the stream into tokens, typically on whitespace and punctuation. Each token also carries a position and an offset back into the original text.
• Token filters transform the token stream: lowercasing, removing stop words, folding accents, and stemming words to a root form (running → run).

function analyze(text, analyzer):
  text   = analyzer.char_filters.apply(text)   # e.g. strip HTML
  tokens = analyzer.tokenizer.split(text)       # "Quick-Brown" -> Quick, Brown
  for f in analyzer.token_filters:
    tokens = f.apply(tokens)                     # lowercase, stop, stem
  return tokens                                  # -> [quick, brown, fox]

4. The Inverted Index

The inverted index is the heart of Lucene. Instead of mapping documents to the terms they contain (a "forward" index), it maps each term to the list of documents that contain it — its postings list. Answering a query then means reading a few short postings lists rather than scanning every document.

Lucene codecs use compact term dictionaries/indexes, historically including BlockTree/FST structures. Exact formats and acceleration structures depend on codec/version.

Because the term dictionary is sorted and postings are stored as ascending document IDs, Boolean queries become efficient list operations. An AND of two terms is a merge-intersection that skips through the shorter list:

function intersect(postingsA, postingsB):     # docs containing BOTH terms
  result = []
  a = postingsA.first();  b = postingsB.first()
  while a != END and b != END:
    if a.doc == b.doc:
      result.add(a.doc);  a = postingsA.next();  b = postingsB.next()
    elif a.doc < b.doc:
      a = postingsA.advance(b.doc)            # skip-list jump, not linear scan
    else:
      b = postingsB.advance(a.doc)
  return result

Postings carry skip lists, so advance(target) can jump ahead instead of stepping one document at a time — which is what keeps multi-term queries fast even on long lists.

5. Segments & Commits

Segments hold mostly immutable indexed data; live-doc/deletion and some update state use versioned files/metadata. Immutability simplifies concurrency, but writers, publication, references, and reader lifetimes still require synchronization. Existing readers retain their snapshot.

A commit safely records the live segment set under the Directory/IndexWriter storage protocol. OS cache/flush is not a durable commit, and arbitrary file writes are not equivalent to the API's crash-safe publication. Keep referenced old files until readers/commits release them.

6. The Indexing Path

The class that writes to an index is the IndexWriter. Only one may have a given index open for writing at a time (enforced by a write lock). Adding documents flows through an in-memory buffer and out to a new segment.

Step by step:

• Buffer. Each indexing thread fills its own DocumentsWriterPerThread (DWPT) — an in-memory mini-index. Per-thread buffers avoid contention so many threads can index in parallel.
• Flush. When a buffer fills (by RAM budget or document count), it is flushed: its in-memory inverted index is serialized to disk as a brand-new segment. After a flush the segment exists, but a crash could still lose it.
• Commit. A commit fsyncs the flushed segments and writes a new segments_N, making everything since the last commit durable.

writer = IndexWriter(directory, config)
for doc in source:
  writer.addDocument(analyze(doc))   # into this thread's DWPT (RAM)
  if dwpt.ram_used > flush_threshold:
    segment = dwpt.flush_to_disk()   # new immutable segment, not yet durable
writer.commit()                      # fsync + write segments_N (durable)
writer.close()

7. The Search Path

Reading is done through a DirectoryReader, which opens one leaf reader per segment, wrapped by an IndexSearcher. A query is not executed directly; it is compiled into objects that know how to iterate postings and produce scores.

The pipeline is:

• Query → Weight. The Query is rewritten (e.g. a wildcard expands to the terms it matches) and turned into a Weight that captures collection-level statistics for scoring.
• Weight → Scorer, per segment. For each segment the Weight produces a Scorer that walks the relevant postings, yielding matching document IDs and their scores. Segments are searched independently — and can be searched concurrently.
• Collect. A Collector (typically a bounded min-heap / priority queue) keeps only the top-K highest-scoring hits, so the engine never sorts the full match set.
• Fetch. Only for the K winners does Lucene read stored fields to hydrate the result. The expensive disk reads happen for a handful of documents, not the whole match set.

searcher = IndexSearcher(DirectoryReader.open(directory))
weight   = searcher.createWeight(query)        # stats for scoring
topk     = PriorityQueue(maxsize = K)          # keep best K by score

for leaf in searcher.leaves():                 # one per segment
  scorer = weight.scorer(leaf)
  for doc in scorer:                           # skip-list iteration over postings
    if leaf.live_docs.get(doc):                # skip deleted docs
      topk.insert(doc, scorer.score())

return [fetch_stored_fields(hit) for hit in topk.sorted()]   # only the K winners

8. Relevance Scoring (BM25)

Boolean matching decides which documents qualify; a similarity decides their order. Lucene's default is BM25, which scores a document for a query term from three intuitive signals.

score(doc, query) = Σ over query terms t:
    idf(t) · ( tf(t,doc) · (k1 + 1) )
            / ( tf(t,doc) + k1 · (1 - b + b · docLen / avgDocLen) )

  idf(t) = ln( 1 + (N - n_t + 0.5) / (n_t + 0.5) )   # N = #docs, n_t = #docs with t
  k1 ≈ 1.2   # tf saturation: higher = tf matters longer
  b  ≈ 0.75  # length normalization: 0 = ignore length, 1 = full

The two tunables are k1 (how quickly extra term occurrences stop helping) and b (how strongly long fields are penalized). The idf values come from the collection statistics captured in the Weight, which is why scoring needs that rewrite step before iteration.

9. Deletes & Updates

Hard deletion marks documents nonlive in the reader's versioned state; physical bytes may remain until merge/cleanup. Soft-delete policies retain selected old versions. Delete visibility follows reader refresh, not necessarily the instant of the write call.

IndexWriter.updateDocument provides supported atomic delete-and-add behavior with respect to readers. Describing it as two independent public calls loses that guarantee. Updates/deletes that match several documents follow their API contract; a field's logical meaning and uniqueness constraints are the application responsibility.

Physical reclamation is affected by merges, retained commits, reader references, and supported cleanup. Do not call merge the only possible route for every obsolete file to leave storage.

10. Segment Merging

Flushing constantly produces new, small segments, and deletes leave dead documents behind. Left alone, an index would drift toward thousands of tiny segments — and every query has to visit every segment. Merging is the background process that keeps this in check by combining several segments into one larger segment.

Merging does two things at once:

• Bounds the segment count. Fewer segments means fewer postings lists to open and intersect per query, so search stays fast as the index grows.
• Reclaims deleted space. A merge reads only the live documents from the input segments, so tombstoned documents are physically dropped — this is the only time deleted data actually leaves the disk.

A MergePolicy decides which segments to merge and when. The default, TieredMergePolicy, groups segments into size tiers and merges within a tier, preferring segments with many deletes. Merges run on background threads, reading the inputs and writing one new segment before the old ones are dropped at the next commit — so search continues uninterrupted while a merge is in flight.

function maybeMerge(segments, policy):       # runs continuously in background
  candidates = policy.findMerges(segments)   # by size tier + delete ratio
  for group in candidates:
    new_seg = merge(group.live_documents())  # dead docs excluded here
    atomically_replace(group -> new_seg)     # visible at next commit

11. Near-Real-Time Search

A new document is searchable as soon as it is in a segment — and a flush creates a segment without a durable commit. Near-real-time (NRT) search exploits that gap: instead of committing (an fsync, which is slow), you open a reader over the just-flushed segments straight from the OS file cache.

The mechanism is reader reopen. Rather than building a fresh reader from scratch, DirectoryReader.openIfChanged(oldReader) returns a new reader that reuses the leaf readers for unchanged segments and only opens leaves for the new ones. Because segments are immutable, this sharing is safe and cheap.

reader = DirectoryReader.open(writer)        # NRT reader tied to the writer
... index more documents ...
newReader = DirectoryReader.openIfChanged(reader)   # picks up flushed segments
if newReader != null:
  reader.close()                             # old leaves not reused are released
  reader = newReader
searcher = IndexSearcher(reader)             # now sees the new documents

12. Segment Files & Codecs

A single segment is physically a group of files that share one base name (_0.tim, _0.doc, …), each holding one part of the inverted index. The component that reads and writes these files is the Codec, and it is pluggable.

Because the codec is recorded per segment, an index can hold segments written by different codec versions at once. When Lucene upgrades its default format, existing segments keep the codec they were written with until a merge rewrites them in the current one — which is why merging is also how a format upgrade physically happens.

Worked example: two reader snapshots

Work through the following bounded example and its failure boundary.

Run and change the example

Download guide_labs.py. Python 3.10+, standard library only. In its directory, run:

python3 guide_labs.py index
python3 guide_labs.py test

The model demonstrates stable reader snapshots and mutable writer state; it is not a Lucene codec.

The program tests its stated finite contracts. It is an educational model, not an implementation of the named production product or a proof of unmodeled distributed faults.

13. Summary

Lucene is a small set of ideas that compose into fast, ranked, durable search on a single machine: