← CatZep/Engineering Guides/Reference/System Internals Guides

← Back to index

OpenObserve — Internal Architecture

A developer's guide to how OpenObserve actually works under the hood: the node roles, the schema-on-ingest pipeline, columnar Parquet on object storage, and how a query prunes its way down to the few files it actually needs.

OpenObserve is an observability platform — logs, metrics, and traces in one engine. Its design is built around a single, simple bet: object storage is the only durable store you need. There is no separate time-series database cluster, no dedicated search cluster, and no per-node attached disk that holds the source of truth. Data lands as compressed columnar Parquet files on an object store (S3, GCS, Azure Blob, MinIO, or even a local filesystem), and every other component is effectively a stateless worker that reads and writes those files. That choice — borrowing the data-lake storage model and pairing it with an Apache Arrow / DataFusion query engine — is what makes it cheap to run and is the thread that ties the rest of the architecture together.

Concepts explained in this guide

Read these definitions first, or use the linked terms as you go. The examples and limitations are local to this guide; companion pages are optional.

1. Design Goals and Core Ideas

Every internal decision in OpenObserve traces back to a small set of goals. Keeping them in mind makes the rest of the architecture predictable.

2. Architecture

OpenObserve combines ingestion, query, routing, compaction, and alerting roles with catalog/object storage; exact supported role/backend combinations vary by release. Use current deployment documentation rather than asserting one SQLite/Postgres/MySQL/etcd matrix universally.

WAL/buffer-owning ingesters have state that must survive or replay on replacement. Query workers may be more disposable, but caches, pinned snapshots, tenant state, and control-plane dependencies still matter. Metadata can be a substantial scaling/availability concern, not automatically tiny.

State acknowledgement storage, volume lifecycle, replication/replay, and warm query visibility explicitly. Adding replicas does not automatically recover unflushed local data or preserve the same catalog/authorization guarantees.

3. Data Model

The data model has three levels. An organization is the top-level tenant boundary — it isolates data, users, and quotas. Inside an organization you have streams, which are the equivalent of tables. Each stream has a type: logs, metrics, or traces. A stream is a sequence of records sharing a schema and a retention policy.

Schema inference is what makes ingestion frictionless. You do not run a CREATE TABLE. The first records to arrive on a stream define its columns and their types; when a later record carries a field the schema has not seen, the column is added and old rows simply have a null for it. This is schema-on-ingest — the schema is a living description maintained in the metadata store, not a fixed contract.

Partitioning is what makes querying fast. Every stream is partitioned by time (records are grouped into time buckets, typically hourly) and optionally by one or more partition keys (for example, a service or k8s_namespace column). Partitioning is expressed in the object-store key layout — each partition is its own prefix:

# object key layout for a partitioned stream
org/acme/logs/app_logs/
  2026/06/25/10/service=api/  # time bucket + partition key
    7f3a...parquet
    9b21...parquet
  2026/06/25/10/service=web/
    a44c...parquet
  2026/06/25/11/service=api/
    ...

Because partitions are encoded in the key prefix, a query with a time range and a service filter can be narrowed to a handful of prefixes before any file is opened. That is the first and cheapest layer of pruning.

4. Storage Format

Parquet groups column chunks into row groups and encodes/compresses them. Readers can project columns and use available statistics/bloom/index metadata to prune. Fetching metadata/footers itself can require I/O; 'before any bytes' is not literal unless that metadata was already cached.

Arrow is a memory layout; Parquet is an encoded persistent format. Decoding/decompression/conversion has cost, so do not describe it as a zero-conversion mapping. Compression ratios and min/max selectivity are workload-dependent. Keep immutable file metadata/length/checksum/version and the catalog publication boundary coherent.

5. Ingestion Path

Validate tenant, envelope, timestamps, schema, and sensitive-data policy. Acknowledgement follows the configured durable ingestion boundary; append in memory/OS cache alone is not fsync or host-loss protection. Retain recoverable input until its required object/catalog publication completes.

Convert/validate bounded batches to immutable Parquet objects; upload with idempotent version/run identity; then register a committed file manifest/catalog entry through a supported transaction. Blob upload plus DB update is not one atomic external action. Crash gaps need retry/reconciliation/orphan cleanup and no premature WAL truncation.

Handle schema drift/type conflicts under an explicit policy, not always by silently adding nullable columns. In-memory/WAL/remote query paths and latency depend on release/settings. Pseudocode is an architecture sketch, not a guaranteed implementation of all versions.

6. Indexing & Pruning

OpenObserve does not scan everything. The whole performance story is about reading as few bytes from the object store as possible, because object-store reads are the slow and metered part of a query. It does this with layered pruning, from coarse to fine.

The layers, applied in order:

Cached catalog statistics can prune before column-data fetch; footers/indexes may still require metadata I/O. Measure actual bytes and selectivity.

7. Query Path

A query is SQL (or PromQL for metrics). It is parsed into a logical plan, pruned down to a file set, then executed in parallel by queriers that fetch only the needed Parquet from the object store and run the plan with DataFusion over Arrow record batches.

The end-to-end flow is:

function query(sql):
  plan  = planner.build(sql)              # logical plan + filters
  files = prune(plan, metadata)           # partitions + min/max + bloom
  shards = split(files, num_queriers)     # scatter the file set
  partials = []
  for q in queriers:                      # run in parallel
    partials += q.execute(plan, shards[q])
  return merge(plan, partials)            # gather + final aggregation

# inside a single querier:
function execute(plan, files):
  for f in files:
    cols  = f.read_columns(plan.projected_columns)  # only needed columns
    batch = arrow.decode(cols)            # Parquet -> Arrow
    emit(datafusion.run(plan, batch))     # filter / aggregate

Two things keep this fast. First, scatter-gather: in HA mode the pruned file set is split across many queriers, so a large scan parallelizes across machines and the partial results are merged at the end. Second, columnar projection: each querier reads only the columns the query references out of each Parquet file, so a SELECT count(*) WHERE service='api' never pays to read the log body. DataFusion does the actual relational work — filtering, joining, aggregating — on Arrow batches, which are a columnar in-memory format that maps almost directly onto Parquet.

8. Compaction & Retention

Merge/retention uses validated immutable output plus atomic catalog replacement and safe old-file lifetime. A query planned against old files must not lose them mid-read. Use the supported snapshot/pinning/grace and bounded-reader protocol, not immediate delete after two separate catalog calls.

Deduplication needs a stated stable identity/window policy; merging is not inherently dedup. Metrics downsampling must preserve reset-aware counters, weighted means, and compatible distributions for quantiles. Retention applies to actual timestamp/content rules and file boundaries; partial expired rows may require rewrite rather than a universal prefix delete.

Capacity, recovery, authorization, and tenant quotas apply during backfills/compaction too. Isolate expensive queries and monitor freshness, lag, publication errors, and retained reader age.

Worked example: compaction meets an in-flight query

Use this example to check which promise each mechanism actually provides.

Run and change the example

Download guide_labs.py. Python 3.10+, standard library only. In its directory, run:

python3 guide_labs.py publication
python3 guide_labs.py test

The model separates object existence, catalog publication, reader pins, and safe cleanup.

The program tests its stated finite contracts. It is an educational model, not an implementation of the named production product or a proof of unmodeled distributed faults.

9. Summary

The whole system is built from a few reinforcing ideas: