← CatZep/Engineering Guides/Reference/System Internals Guides

← Back to index

MinIO — Internal Architecture

A developer's guide to how MinIO actually works under the hood: the S3-compatible API surface, the server/drive/erasure-set/pool layout, Reed-Solomon erasure coding, the write and read paths, healing, and how a deployment scales and replicates.

MinIO distributes request coordination across storage servers. Those servers are not stateless: persistent object shards, metadata, runtime coordination, and recovery state still matter. Coding and valid-version commit rules provide recoverability only within the stated fault model.

Concepts explained in this guide

Read these definitions first, or use the linked terms as you go. The examples and limitations are local to this guide; companion pages are optional.

1. Design Goals and Core Ideas

Every internal decision in MinIO traces back to a small set of goals. Keeping them in mind makes the rest of the architecture predictable.

2. Architecture and Deployment Layout

MinIO distributes object operations across a storage deployment; a request-serving node coordinates operations, but that does not make storage nodes stateless. Persistent shards, metadata, configuration, and runtime ownership/recovery are state.

Define total erasure-set drives n, parity m, and data-equivalent shards k=n−m consistently. Sets span configured drives/servers; one server can own several drives in a set. Evaluate failures by the number of affected shards and independent domains, not 'one server always means one shard'. Pool and layout support are release-specific.

Within a pool, set placement can be determined from object identity under its configured mapping. Pool choice and cross-pool discovery are separate: free-space-aware placement of new objects cannot universally be recomputed from name alone. Expansion/decommissioning require supported coordinated procedures, capacity, and observed health.

3. Erasure Coding

Reed–Solomon turns k data units into n=k+m coded units. Any k valid shards with distinct compatible identities can reconstruct under the code's assumptions. Storage multiplier is n/k; overhead relative to original data is m/k, not m/n. For n=16,m=4: k=12, multiplier 1.333, overhead 33.3%; parity is 25% of stored shards.

A decoding threshold is not automatically a commit protocol. A common MinIO write-quorum formulation requires k responses, and k+1 when data and parity counts are equal, with valid matching metadata; check the precise release/configuration. The unconditional (n/2)+1 rule is wrong for EC:4 on sixteen drives: nine responses cannot even reconstruct twelve data-equivalent units.

Recoverability assumes the same object version, known shard index, compatible coding parameters, intact length/checksum data, and enough valid survivors. A checksum detects corruption under its error model; it does not create extra parity. Loss of more than m shards can be unrecoverable, and correlated server/zone faults may remove multiple shards at once.

4. Object Layout and xl.meta

When an object is stored, MinIO does not write one file. It writes one shard part plus one metadata file on each drive of the erasure set. The shards together reconstruct the object; the metadata file describes how.

On each drive, the object occupies a directory named after its key, holding:

• part.1, part.2, ...: the actual shard data for this drive (one part per object part for multipart uploads).
• The metadata records coding parameters using a consistent convention: k data-equivalent shards, m parity, n=k+m total, plus shard index, part lengths, checksums, and version identity.

Keeping a copy of xl.meta on every drive means the metadata is itself protected by the same quorum as the data — there is no separate, single-point metadata store to lose. Very small objects are written inline inside xl.meta to avoid a second file open. Object versions are recorded as multiple entries inside the same xl.meta, each with its own version id and checksums.

How an object maps to drives

Placement is fully computed from the object name — there is no lookup table. A deterministic hash of bucket/object, modulo the number of erasure sets, picks the one set that holds the object, so the same name always resolves to the same set. The object is then Reed-Solomon split into the set's data + parity shards, and a second name-derived permutation lays exactly one shard per drive, shuffling which drives hold data versus parity so no drive becomes a permanent parity hotspot. Because both steps are pure functions of the name, any server can locate every shard with arithmetic alone.

5. Write Path (PUT)

A PUT coordinates version-safe data and metadata publication. Stream bounded chunks through the configured encoder, write shards with stable indices/checksums/version identity, and count only successfully completed compatible writes. An online drive is not proof its append/fsync succeeded.

Use the product's layout-specific write and metadata quorum/recovery rules. If completion is uncertain, a client retry may create/reuse a version according to the API; do not infer that a timeout rolled back all remote writes. Partial artifacts need recovery/cleanup under the implementation protocol.

Data and xl.meta publication is not safely reproduced by a loop that appends bytes then independently counts metadata writes. Concurrent object mutation, checksums, atomic local rename/write ordering, failure-domain placement, and stale coordinators all matter. Use reviewed server behavior rather than treating this tutorial as its implementation.

6. Read Path (GET)

A read first selects a valid committed object version using supported metadata rules. It then retrieves enough checksum-validated shards with distinct indices from that version, reconstructs missing data if necessary, and verifies expected length/integrity. Mixing newer and older shards can corrupt the result even when their count reaches k.

At least k=n−m compatible shards are the coding requirement, not 'any n data-equivalent shards'. Unreachable or corrupt shards reduce available survivors; metadata uncertainty can make a read fail despite adequate physical bytes. Repair restores redundancy only while enough valid versioned data remains.

7. Healing

Drives fail, bits rot, and servers come and go. Healing is the process that restores the full set of shards for an object by reconstructing the missing or corrupt ones from the survivors. It is the erasure-coding analogue of repair in a replicated system.

Healing is detected and driven three ways:

Reconstruction always uses the same Reed-Solomon decode as a read: as long as at least N−M shards survive, the missing shard's data can be regenerated and re-encoded. The healed shard is written back with a freshly computed checksum, and xl.meta is updated to mark the object whole again.

function heal(set, object):
  meta = read_quorum_xl_meta(set, object)
  N, M = meta.scheme
  good = [drive.read(object) for drive in set.drives
          if drive.is_online() and highwayhash(...) == meta.checksum[drive]]
  if len(good) < N - M:
    log_unrecoverable(object); return            # lost too many shards
  full = reed_solomon_reconstruct(good, N - M, M) # regenerate every shard
  for drive in set.drives:
    if drive_shard_missing_or_bad(drive, object):
      drive.write(object, full[drive.index])      # rewrite + checksum
      drive.update_xl_meta(object)

8. Scaling and Replication

Use supported pool expansion and decommissioning procedures. Pool placement of new objects may be free-space-aware while set placement within a pool follows a stable mapping. Existing versions can require cross-pool lookup; do not claim the whole location is always a pure hash of the object name.

Expansion, rebalance, and decommissioning consume capacity and network I/O and can fail. Plan survivor space, reader/version lifetime, repair, and rollback; 'no downtime/no loss' is an objective under those prerequisites, not a universal result of adding a pool.

Remote bucket/site replication has its own lag, version/delete, conflict, and failover contract. Active-active availability is not global linearizability. Read/write coordination and distributed lock rules belong to the actual product implementation; quorum arithmetic alone does not protect every stale coordinator or partially written version.

Worked example: 16 shards, four lost drives

Use n=16 total shards with m=4 parity, so k=12 valid compatible shards are needed.

Run and change the example

Download guide_labs.py. Python 3.10+, standard library only. In its directory, run:

python3 guide_labs.py erasure
python3 guide_labs.py test

The program implements a small non-systematic Reed–Solomon code over GF(256), reconstructs from compatible survivors, and rejects an insufficient set. It does not implement MinIO. This toy code is non-systematic: its encoded shards are polynomial evaluations, not literal copies of the original data bytes. Shard IDs select equations; GF(256) arithmetic makes the chosen system invertible. Production metadata/version/integrity protocols are separate.

The program tests its stated finite contracts. It is an educational model, not an implementation of the named production product or a proof of unmodeled distributed faults.

9. Summary

The whole system is built from a few reinforcing ideas: