← CatZep/Engineering Guides/System Design/Design Scenarios

Designing a High-Volume Telemetry Ingestion Pipeline

A system design interview guide to the pipeline that swallows a firehose of telemetry from heterogeneous sources and lands it, reliably and losslessly, into stores that serve both real-time analytics and LLM/RAG querying.

Behind every observability platform is an ingestion pipeline doing the unglamorous, load-bearing work: accepting telemetry from tens of thousands of sources in a dozen formats, buffering it durably so nothing is lost when a downstream store hiccups, transforming and validating it in flight, and routing it to whichever stores need it. The modern twist is that there are now two very different consumers of that data — the traditional real-time analytics path (dashboards, alerts, ad-hoc queries) and a newer semantic path that embeds telemetry into a vector store so an LLM can answer questions over it via retrieval-augmented generation. A good design serves both from one pipeline while keeping the operational and analytical stores cleanly separated. This guide builds it: the durable buffer at the center, stream processing for parse/enrich/route, the delivery guarantees (exactly-once, dead-letter, PII scrubbing), and the dual-sink fan-out.

Concepts to understand first

Each definition includes a concrete example and the condition under which the mechanism is useful. You can read this guide without opening another guide.

1. Requirements & Scope

The pipeline sits between producers of telemetry and the stores that consume it. Pin down what it must guarantee.

• Heterogeneous ingest. Accept logs, metrics, traces, and events from many source types (apps, hosts, network devices, third parties) in many formats, and normalize them.
• Durability & no loss. Once accepted, a record must not be lost even if every downstream store is temporarily unavailable. The buffer decouples producers from consumers.
• Throughput & backpressure. Sustain millions of records per second and degrade gracefully under spikes rather than dropping data or stalling producers.
• Transform in flight. Parse, validate against a schema, enrich with context, scrub PII, and route each record to the right sink(s).
• Dual consumers. Feed a real-time analytical path and a semantic (embedding/vector) path for LLM/RAG querying, keeping operational and analytical stores separated.
• Delivery semantics. At-least-once by default, exactly-once where correctness demands it, with a dead-letter path for records that cannot be processed.

2. Back-of-the-Envelope Estimates

Assume 10M records/second at peak, averaging 1 KB each.

Worked estimate, step by step

The numbers force two conclusions: the buffer must be partitioned into the low thousands to get the parallelism to keep up, and you cannot embed everything — the semantic path is fed a deliberately routed, filtered subset, which is why routing at processing time matters.

3. Data Model, Schema & Validation

Heterogeneous sources produce a mess of formats, so the first job is to normalize everything into a common envelope while validating structure. A typical normalized record carries a timestamp, a source/type, a set of resource attributes, the payload, and correlation IDs.

# normalized telemetry record (envelope + payload)
{
  "ts":        1751779200000,        # event time, ms
  "source":    "svc-checkout",       # who produced it
  "type":      "log" | "metric" | "trace" | "event",
  "resource":  { "host": ..., "region": ..., "env": ... },
  "trace_id":  "4bf92f...",          # correlation, if in a request
  "body":      { ... },              # the actual payload
  "schema_ver": 3                    # for evolution
}

Schema & validation. Use a schema registry (Avro/Protobuf/JSON Schema) so producers and consumers agree on structure and it can evolve compatibly. Validate at the edge: a record that fails schema validation is not silently dropped and not allowed to poison the pipeline — it is routed to a dead-letter queue for later inspection. Schema-on-write (validate at ingest) gives clean downstream data; schema-on-read (store raw, interpret later) gives flexibility. High-volume telemetry usually does light validation on write plus schema-on-read for the flexible payload, so a new field from a source does not break ingestion.

4. API

# edge ingest (agents batch + compress)
POST /v1/ingest            Content-Encoding: zstd
     body: [ Record, Record, ... ]          # up to ~1 MB per batch
  202 Accepted             # only after Kafka acks=all
  429 Too Many Requests    Retry-After: 2   # per-tenant quota: back-pressure
  400 Bad Request          # fails schema, with per-record errors

# topics
telemetry.raw.{logs|metrics|traces|events}   key = trace_id or source, RF=3, min.insync.replicas=2
telemetry.dlq                                 original bytes + error + stage
telemetry.clean.*                             scrubbed, enriched; fed to sinks

Return 202 only after Kafka acknowledges. That is the durability promise to the producer.

5. High-Level Architecture

The pipeline is a fan-in to a durable buffer, then stream processing, then a fan-out to separated sinks.

The shape is deliberate. Agents at the edge batch and compress so producers are cheap and the network is efficient. Kafka in the middle decouples everything: producers write as fast as they like, consumers read at their own pace, and a slow or dead sink never backs pressure onto the producers. Stream processing is where all the transformation and routing happens. And the fan-out separates concerns: the operational store is tuned for low-latency reads, the analytical store for cheap batch scans, and the vector store for semantic retrieval — three different access patterns, three different stores, one pipeline.

6. The Durable Buffer: Kafka

Kafka partitions provide append order, not necessarily event-time/causal order across concurrent producers. One active group member owns a partition at a time; asynchronous processing still needs ordered completion/checkpointing. Stable related keys help routing, but trace spans from different hosts can arrive in any order.

For a common durable broker-failure contract, use suitable replication placement, acks=all, min.insync.replicas, producer idempotence, and safe leader election settings. These are configuration guarantees under a fault model, not universal power-loss protection for every storage mode. acks=all waits for the current in-sync set, whose minimum matters.

Retention can be age or size constrained; downstream delay is safe only while unread input remains available and recovery throughput drains it. Agents/SDKs/brokers have finite buffers. Define explicit per-signal drop/backpressure and archive/replay policy.

7. Stream Processing & Guarantees

At-most-once may be acceptable for sampled debug data, not for accepted audit/billing records. At-least-once allows retries; exactly-once logical effects require stable identities plus transactional/idempotent sink state under a specified horizon. Kafka transactions cover supported Kafka records/offsets, not arbitrary external stores.

Use independent sink consumer groups or durable per-sink status to isolate failures. Required durable sink/quarantine writes precede input-position advancement; committing after one of several required writes can lose the rest. Replays reuse IDs and immutable payloads, and mismatch is an error. Checkpoints and contributions must agree.

Redact/minimize before any broker/archive/DLQ whose policy forbids retaining raw sensitive data. Hashing alone is not anonymity. Embedding/RAG is optional asynchronous derived work with tenant filters and source citations; approximate retrieval and an LLM response are not proof of a causal incident explanation. The compact pipeline snippets are pseudocode, not runnable OTel configuration.

8. Dual Sinks: Operational, Analytical, Semantic

The routing step fans each record to the sinks that need it, and the sinks are kept cleanly separated because they serve fundamentally different access patterns.

Operational vs analytical separation is the classic move: the operational store is optimized for freshness and low-latency reads and kept small, while the analytical store holds cheap, long-retained columnar data for scans and ML. Keeping them apart means a heavy analytical scan never contends with the low-latency operational path.

The semantic / RAG path

The newer sink supports natural-language querying of telemetry (“why did checkout error rates spike last night?”). A routed subset of records — typically errors, anomalies, and notable events, not the full firehose — is passed through an embedding model that turns each into a vector capturing its meaning, and the vectors are stored in a vector store (with an ANN index like HNSW). At query time, the user’s question is embedded, the vector store returns the nearest telemetry, and that context is fed to an LLM to generate a grounded answer — retrieval-augmented generation. Crucially this is a separate sink: embedding is expensive, so only a filtered subset flows here, and it never sits on the operational hot path.

9. Bottlenecks & Tradeoffs

• Partition key skew. The most common real bottleneck — a poor key funnels a hot tenant or hot service onto one partition, capping throughput. Key for even spread while preserving needed ordering.
• Exactly-once vs throughput. Transactional exactly-once adds coordination overhead and latency. Use it only where duplicates are a correctness bug; prefer at-least-once + idempotent sinks elsewhere.
• Buffer retention vs cost. Longer Kafka retention means surviving longer outages and more replay flexibility, but more storage. Size it to your worst tolerable downstream outage.
• Schema-on-write vs on-read. Strict validation gives clean data but rejects anything unexpected and couples producers tightly; loose validation is flexible but pushes correctness downstream. Most pipelines do light write validation + schema-on-read.
• Backpressure vs loss. Under sustained overload beyond buffer capacity you must eventually choose: shed low-value telemetry (sampling) or risk falling permanently behind. Decide the shedding policy deliberately, by signal priority.

10. Failure Modes

Worked example: one sink fails after another succeeds

One record is routed to both search and archive.

Run a small, testable model

Download concepts.py. It uses only the Python 3.10+ standard library. From the directory where you saved it, run:

python3 concepts.py dedup
python3 concepts.py test

The local contribution model demonstrates transactional replay; each external sink needs its own supported durability/idempotency contract.

The program contains deterministic assertions and failure traces. It is a learning model, not a production service: it does not implement network consensus, distributed durability, authentication, or third-party delivery. Read its contracts and try changing an input before extending it.

11. 45-Minute Interview Walkthrough

12. Common Mistakes

13. Summary

A high-volume telemetry pipeline is organized around a durable, replayable buffer that decouples producers from consumers, with in-stream processing that validates, enriches, scrubs, and routes to cleanly separated sinks — including a semantic path for LLM/RAG.

14. Check Yourself

1. Why put Kafka between producers and stores?
   Answer

   It decouples them. A slow or dead store only makes consumers fall behind. Data waits in Kafka and is replayed later.

2. How many partitions for 10 GB/s at 10 MB/s each?
   Answer

   10 GB/s ÷ 10 MB/s = 1,000 partitions, plus headroom.

3. Do Kafka transactions give exactly-once writes to Postgres?
   Answer

   No. They are atomic only across Kafka topics and offsets. For Postgres use an idempotent upsert, or store the offset in the same DB transaction.

4. What delivery guarantee do you get if you commit the offset before writing the sink?
   Answer

   At-most-once. A crash after the commit loses that batch.

5. Where should PII scrubbing happen, and why?
   Answer

   In the stream processor, before any queryable store. Doing it once in-stream beats redacting every store later.

6. Why embed only ~1% of records?
   Answer

   Embedding 10M/s costs too much and adds little. RAG needs the relevant few, such as errors and anomalies.

15. Key Terms

Topic / partition

Named log, split into ordered shards.

Consumer group

Consumers sharing a topic’s partitions.

Offset

An offset is a position in one log partition. A checkpoint is a recoverable, consistent record of input positions and processing state. On restart, state must correspond exactly to the input that is considered processed. Example and limits above.

acks=all

Ack only after in-sync replicas write.

Idempotent sink

Writing twice has the same effect as once.

DLQ

Dead-letter queue for bad records.

Schema registry

Shared, versioned record schemas.

Back-pressure

Slowing producers when consumers fall behind.

16. Further Reading

• Apache Kafka documentation (replication, delivery semantics)
• Confluent: Transactions in Apache Kafka
• Confluent: Exactly-once semantics in Kafka
• Apache Flink documentation
• OpenTelemetry Collector