← CatZep/Engineering Guides/System Design/Counting & Analytics

Designing a Logging / Analytics Pipeline

A system design interview guide to building a pipeline that collects logs and events from thousands of services and lands them, without loss or duplication, in the stores that engineers and analysts actually query.

Almost every company eventually arrives at the same realization: the data describing how the system behaves — request logs, error traces, business events — is at least as valuable as the data the system was built to serve. A logging or analytics pipeline is the plumbing that captures that exhaust from every running process and delivers it to the places people query it: a search store for debugging an incident at 3 a.m., a warehouse for the analyst building a dashboard, and cheap archival storage for the auditor who needs last year's records. The naive approach — have each service write straight to the destination — collapses the moment you have many producers, bursty traffic, or more than one consumer. This guide builds the pipeline up the way it actually has to be built: collect at the edge, buffer in the middle, process the stream, and fan out to several sinks, with delivery guarantees and ordering handled deliberately at each step.

Concepts to understand first

Each definition includes a concrete example and the condition under which the mechanism is useful. You can read this guide without opening another guide.

1. Requirements and Clarifying Questions

• How many hosts and how much data? (Assume 20,000 hosts, about 100 lines per second each.)
• Who reads it? Engineers searching recent logs, analysts querying events, auditors needing old data. (Assume all three.)
• How fresh? (Assume logs searchable within 30 seconds; analytics within minutes.)
• Can we lose data? Debug logs: a little under extreme load. Billing and audit events: never.
• How long to keep it? (Assume 7 days hot search, 1 year warehouse, 7 years archive for compliance.)

2. Back-of-the-Envelope Estimation

Assume 20,000 hosts × 100 lines/s, 300 bytes per line, 5× compression, Kafka replication factor 3 with 3 days of retention, and a search store that indexes 20% of the data (warnings and above, plus sampled info).

Two things stand out. The search store is the expensive sink, which is why routing and sampling matter. And a 6-hour sink outage builds a 13 TB backlog; with consumers able to run at 2× the input rate, the extra capacity (600 MB/s) drains it in about 6 more hours. That fits easily inside 3 days of retention.

3. Sources of Data

The first thing to be precise about is what is actually flowing into the pipeline, because the two main kinds of data have different shapes and different downstream uses. Lumping them together as "logs" hides design decisions you will later wish you had made on purpose.

Application logs are what a human reads when something breaks; they are forgiving in format but enormous in volume. Structured events are what a query engine aggregates; they need consistent field names and types or every downstream analysis becomes a guessing game. A good pipeline accepts both but nudges producers toward structure: even a log line is far more useful when it carries a service name, a timestamp, a severity, and a trace id as proper fields rather than buried in a string. The earlier you impose a little structure, the less parsing and guesswork everything downstream has to do.

4. Record Format and Interfaces

A pipeline's "API" is mostly its record envelope. Agree on it early; every stage depends on it.

{
  "event_id":   "01J8Z3K4Q6…",          # unique, set once at the source (dedup key)
  "ts":         "2026-09-25T14:03:07.412Z",  # event time
  "service":    "checkout",             # also the partition key
  "host":       "web-4411", "region": "us-east-1",
  "severity":   "ERROR",
  "trace_id":   "4bf92f35…",            # links logs to traces
  "schema":     "checkout_completed/v3",# structured events only
  "body":       { … }                   # text line or typed fields
}

5. The Collection Agent

Once you accept that data is generated on thousands of hosts, the question becomes how to get it off those hosts reliably without coupling every application to the pipeline. The standard answer is a collection agent: a small, separate process that runs on each host (or as a sidecar next to each container), tails the application's log files or receives events over a local socket, and forwards them onward.

Running collection as a separate agent rather than inside the application buys a clean separation of concerns. The application's only job is to emit a line or an event locally and move on; it never blocks on the network and never needs to know where the data eventually goes. The agent handles the messy realities of shipping data: batching many small records into efficient payloads, compressing them, retrying when the next hop is unavailable, and buffering to local disk so that a brief outage downstream does not lose data or stall the application. Because the agent is uniform across the fleet, you can also use it to attach common metadata — host name, region, environment, service version — to every record without touching application code.

# agent runs on every host, tails sources and forwards in batches
function agent_loop():
  buffer = []
  for record in tail(log_files) + recv(local_socket):
    record = enrich(record, host=hostname(), region=REGION)
    buffer.append(record)
    if buffer.full() or timer.expired():
      try:
        broker.send(compress(buffer))   # one batched, compressed call
        buffer.clear()
      except Unavailable:
        spill_to_disk(buffer)           # survive a downstream outage
        buffer.clear()                  # spilled batches are resent by replay_spill()

function replay_spill():                # background thread
  for batch in oldest_first(spill_dir):
    broker.send(batch)                  # retry with backoff until it succeeds
    delete(batch)                       # only after the broker acknowledged it

6. End-to-End Architecture

With sources and agents established, the central design is the path a record takes from the host where it is born to the store where it is finally queried. The defining move is to insert a durable buffer between the producers and the consumers, so that the side generating data and the side reading it are completely decoupled and can fail, scale, or slow down independently.

Reading the flow left to right:

• Sources and agents. Every host produces logs and events; its local agent batches, compresses, and ships them, so applications are never coupled to the rest of the pipeline.
• Durable broker. A partitioned, replicated log such as Kafka sits in the middle. Producers append to it and consumers read from it independently. It is the shock absorber: it persists records for a retention window, absorbs traffic spikes, and lets slow or restarting consumers catch up without dropping anything.
• Stream processors. A pool of consumers reads from the broker and does the real work — parsing raw lines into fields, enriching records with extra context, filtering noise, and routing each record to the sinks that should receive it.
• Sinks. The processed stream lands in several destinations at once, each tuned to a different access pattern: a search store for interactive debugging, a warehouse or lake for analytics, and cold object storage for long-term archive.

The single most important property of this shape is that the broker breaks the system into two halves that can be reasoned about — and operated — separately. A spike in producers, a slow sink, or a redeploy of the processors no longer ripples across the whole pipeline.

7. The Durable Buffer

The broker deserves its own section because it is what turns a fragile chain into a resilient system. Without it, every producer is directly coupled to every consumer: if the warehouse is slow, the applications writing to it slow down too, and a traffic spike that the sinks cannot absorb is simply lost. Putting a durable, append-only log in the middle changes all of that.

• Decoupling. Producers only need to know about the broker, and consumers only need to know about the broker. Either side can be redeployed, scaled, or fail without the other noticing. You can even add a brand-new consumer later and have it read the existing data — but only data still inside the broker's retention window (often 3–7 days). Anything older must be backfilled from the archive.
• Spike absorption. When events arrive faster than the sinks can accept them, the excess simply accumulates in the broker for its retention period and drains once the burst passes. The pipeline degrades into latency, not loss — as long as the backlog is drained before retention deletes it.
• Durability and replay. Because the broker persists and replicates records, a consumer that crashes can resume from its last committed position, and you can replay history — for example to backfill a new sink or to recover from a bug in a processor — by rewinding the read position.

8. Stream Processing

Raw records as they leave the agent are rarely in the form the sinks want. The stream processing layer is where they are transformed on the fly, record by record, as they flow from the broker to the sinks. Three jobs dominate this stage.

• Parse. Turn a raw log line into structured fields — extract the timestamp, severity, message, and any embedded key-value pairs — so that downstream stores can index and query them rather than treating each entry as opaque text.
• Enrich. Add context the source could not include: resolve a user id to a region, attach a deployment version, tag records with the data center, or join against a lookup table. Enrichment is what makes later analysis possible without re-deriving the same facts everywhere.
• Route and filter. Decide where each record goes. Debug-level chatter might go only to the search store with a short retention; billing events must go to the warehouse and the archive. Pure noise can be dropped here so it never costs storage downstream.

function process(record):
  record = parse(record)                  # text -> typed fields
  record = enrich(record, lookup_region)  # add context
  if is_noise(record):
    return                                 # drop, don't store
  for sink in route(record):              # fan out to chosen sinks
    sink.write(record)

Because processing runs as a horizontally scalable pool of consumers reading from the broker's partitions, it scales with traffic simply by adding more instances — up to the number of partitions, which is why partition count, discussed below, is a capacity decision and not just a correctness one.

9. Sinks and Fan-Out

The same processed record usually needs to live in more than one place, because no single store is good at everything. This is the fan-out: one logical stream is written to several sinks, each chosen for a different access pattern, cost profile, and retention.

Treating these as separate sinks rather than trying to force one store to do all three is what keeps each one affordable and fast at its job. The search store stays small and fast because it only holds recent hot data; the warehouse holds the structured events analysts care about; the archive holds everything cheaply for as long as policy requires. Routing decisions made in the processing layer determine which records reach which sink, so you are not paying search-store prices to store a year of debug logs.

10. Delivery Guarantees

At-least-once processing advances a sink's input position only after the required write is durably accepted. It permits redelivery after uncertain acknowledgements or crashes. It does not promise survival of arbitrary outages, retention expiry, source loss, or loss of all durable copies. At-most-once may be a deliberate policy for low-value debug telemetry, not for accepted audit events.

Kafka idempotent producers deduplicate supported producer retries, not arbitrary re-emission as a fresh logical event. Kafka transactions can atomically publish output records and input offsets inside Kafka; consumers must use the appropriate committed isolation. An external database/object/search sink needs its own transactional or idempotent publication contract.

Use stable source event IDs, explicit per-sink consumer groups/checkpoints, and a raw archive for longer replay. Approximate/windowed dedup is acceptable only when the consumer permits its undercount/duplicate horizon. The collection agent must persist source position and spill state in crash-safe order; a partially sent unacknowledged batch needs idempotent replay, not silent deletion.

11. Partitioning and Backpressure

Two operational mechanisms make the pipeline both fast and stable: partitioning, which buys ordering and parallelism, and backpressure, which keeps a fast producer from drowning a slow consumer.

Partitioning. The broker splits each topic into partitions. Records are assigned to a partition by hashing a key — for example the service name or the user id. This gives two properties at once: records that share a key always land in the same partition and are therefore consumed in order, while different keys spread across partitions and are consumed in parallel. The number of partitions is the ceiling on how many consumers can work a topic simultaneously, so it is the main lever for throughput.

Backpressure. Because the broker decouples the two sides, a slow sink does not stall producers; instead the consumer's lag — how far its committed position trails the newest record — grows. That lag is the system's primary health signal. Rising lag tells you a sink or the processor pool is the bottleneck, and it gives you a graceful place to react: add consumers, shed or sample low-value records, or let the broker's retention hold the backlog until the burst passes. The pipeline turns overload into measurable latency rather than silent data loss — until lag grows past retention, at which point Kafka deletes records nobody has read yet (see the diagram above). So alert on lag measured in time ("the consumer is 9 hours behind") and compare it with retention.

Strictly speaking, this is buffering. True backpressure means the slow side pushes back on the fast side: if the brokers themselves are overloaded, producer sends block or time out, the agents' buffers fill, and the agents must spill, sample, or drop. A good design says which of those happens, where, and how it is measured.

12. Schema Management

A schema is the producer/consumer contract for field names, types, defaults, and meaning. Backward compatibility means a new reader can consume old data; forward compatibility means an old reader can consume new data. Full compatibility requires both under the selected serialization rules and version scope.

An optional field is safe only if defaults, unknown-field handling, and reader expectations allow it. Avro, Protobuf, and JSON Schema rules differ; a registry is not automatically a semantic validator. Version meanings, units, and enum behavior as well as shapes. Edge checks quarantine invalid records with access control, bounded storage, and a replay owner rather than poisoning aggregates.

13. Failure Modes

14. Tradeoffs and Alternatives

Worked example: draining a six-hour outage

Assume raw ingress 600 MB/s, a sink outage of six hours, and recovered service capacity of 1200 MB/s.

Run a small, testable model

Download concepts.py. It uses only the Python 3.10+ standard library. From the directory where you saved it, run:

python3 concepts.py dedup
python3 concepts.py test

Use the transactional contribution example to see why replay can be safe even when delivery repeats.

The program contains deterministic assertions and failure traces. It is a learning model, not a production service: it does not implement network consensus, distributed durability, authentication, or third-party delivery. Read its contracts and try changing an input before extending it.

15. A 45-Minute Interview Plan

16. Common Mistakes

• Saying Kafka "never loses data". Retention deletes unread data. Lag must stay under retention.
• Claiming end-to-end exactly-once. Kafka transactions cover Kafka-to-Kafka. External sinks need idempotent writes.
• Committing the offset before the write. A crash then loses the record.
• Letting the app call the network to log. A slow pipeline then slows the app.
• One consumer group for all sinks. A slow sink then stalls fast ones. Give each sink its own group.
• Indexing everything in the search store. It is the most expensive sink. Route and sample.

17. Summary

A logging and analytics pipeline is a decoupled transport that collects data at the edge, buffers it durably, transforms it in flight, and fans it out to purpose-built stores. Its design is a handful of reinforcing decisions:

18. Check Yourself

1. A consumer writes a batch to the warehouse and crashes before committing its offset. What happens after restart, and why is that acceptable?
   Answer

   It re-reads and re-writes the batch. Because writes are idempotent upserts keyed by event_id, the replayed records overwrite themselves and nothing is double counted.

2. Kafka retention is 3 days. The search sink has been down for 4 days. What happened to the data?
   Answer

   The oldest day of unread records was deleted by retention before the search consumer read it. Recover it from the archive, and alert on lag-in-time well before it reaches retention.

3. Your topic has 24 partitions. You add 40 consumers in one group. How many do useful work?
   Answer

   24. Each partition is read by at most one consumer in a group, so 16 sit idle. Partition count caps parallelism.

4. Why do the search store and the warehouse use separate consumer groups?
   Answer

   Each group tracks its own offsets. A slow search store builds its own lag without holding back the warehouse.

5. A producer renames amount to total. What stops this from breaking dashboards?
   Answer

   The schema registry's compatibility check rejects the new schema at produce time. The safe path is to add total as a new optional field, migrate readers, then retire amount.

6. At 600 MB/s, how much backlog does a 6-hour sink outage create, and how long to drain it at 2× capacity?
   Answer

   600 MB/s × 21,600 s ≈ 13 TB. The spare capacity is 600 MB/s, so it takes about 6 more hours to catch up.

19. Key Terms

Collection agent

Per-host process that tails logs and ships batches (Fluent Bit, Vector, Filebeat).

Broker

Durable, partitioned, replicated log between producers and consumers (Kafka).

Partition

An ordered shard of a topic; the unit of parallelism.

Offset

An offset is a position in one log partition. A checkpoint is a recoverable, consistent record of input positions and processing state. On restart, state must correspond exactly to the input that is considered processed. Example and limits above.

Consumer group

Consumers sharing one set of offsets; each partition goes to one member.

Consumer lag

How far a consumer trails the newest record, in records or time.

Retention

How long the broker keeps records before deleting them, read or not.

At-least-once

No loss, possible duplicates.

Effectively-once

At-least-once delivery plus idempotent writes, so duplicates have no effect.

Dead-letter topic

Where records that fail processing are parked for inspection.

Schema registry

Central store of event schemas and compatibility rules.

Further reading

• Jay Kreps, "The Log: What every software engineer should know about real-time data's unifying abstraction" (2013).
• Apache Kafka documentation: design, message delivery semantics, and transactions.
• Martin Kleppmann, Designing Data-Intensive Applications, chapter 4 (encoding and evolution) and chapter 11 (stream processing).
• Confluent Schema Registry documentation: schema evolution and compatibility types.