A system design interview walkthrough for a YouTube-style service: how a raw upload becomes a watchable video, why transcoding is modeled as a directed acyclic graph, how the orchestration and worker pools are organized, and how a CDN serves the result to a global audience.
Student edition · start with the guarantee
Turn a validated upload into compatible streaming renditions through a recoverable dependency graph.
Correctness contract: Only durable validated artifacts from current task attempts are published. A manifest becomes visible after required aligned renditions exist; private content remains authorized.
Basic HTTP, databases, and programming are assumed. Specialized terms are explained below. Named products motivate the problem; this is an educational design, not a claim about their current internals. Capacity numbers are illustrative decimal payload estimates, not benchmarks. Add indexes, protocol overhead, replicas, headroom, and failure capacity before provisioning.
A video platform looks deceptively simple from the outside: someone uploads a file, other people press play. Underneath, two very different flows have to work well at scale. The upload and processing flow takes a large, arbitrarily-formatted source file and turns it into many playable renditions; it is write-heavy, bursty, and compute-intensive. The streaming flow delivers those renditions to viewers; it is overwhelmingly read-heavy, latency-sensitive, and global. Almost every interesting decision in the design comes from treating these two flows separately and connecting them with durable storage and a CDN. This guide builds the system up flow by flow, leaning on the same handful of ideas — decoupling, parallelism, and pushing bytes close to the viewer.
Each definition includes a concrete example and the condition under which the mechanism is useful. You can read this guide without opening another guide.
Resumable and multipart uploads
A persisted upload session tracks committed ranges or parts. After interruption, the client queries server progress and retries only missing bytes with checksums and an idempotent part identity.
Example. A 4 GB upload stops after three 100 MB parts. The server reports those durable parts, and the client resumes at the first missing part rather than trusting its last local send.
Limit. Validate ownership, total size, part checksums, and expiry. A completed blob becomes visible only after a conditional metadata commit and all referenced objects are durable.
DAG scheduling
A directed acyclic graph represents task dependencies. A node becomes runnable only when required predecessors have successfully produced their durable outputs. Acyclic means no dependency cycle, not automatic task termination.
Example. Inspect source video, encode several renditions in parallel, package aligned segments, then conditionally publish a manifest once the required renditions exist.
Limit. Tasks can hang or fail despite an acyclic graph. Use timeouts, bounded retries, attempt identities, cancellation, and atomic completion checks so stale attempts cannot publish bad artifacts.
Adaptive bitrate streaming
A player chooses among compatible encoded renditions segment by segment, using measured download throughput and buffered playback time. HLS/DASH manifests describe the renditions and segment locations.
Example. With only 2 seconds buffered, drop to a lower bitrate even if the last download was fast. With 30 seconds buffered, cautiously raise quality.
Limit. Renditions need compatible codecs and aligned keyframes/segment boundaries for seamless switching. CDN edges are not always the geographically nearest route, and private video still needs authorization.
Content addressing and garbage collection
An object is named by a cryptographic hash of its canonical bytes. Immutable versions reference object hashes. Garbage collection deletes only objects that no retained version or active upload can still use.
Example. Two file revisions reference one unchanged chunk. Deleting one revision must not delete that chunk while the other still references it.
Limit. Hash matches need an explicit collision model; cross-tenant existence checks can leak content. A grace period alone is insufficient for unbounded uploads: pin active sessions or bound their lifetime.
Transactional outbox
A business transaction writes its state change and an event row to the same database. A relay publishes committed event rows later, retrying until the broker durably accepts them.
Example. An order and its OrderCreated outbox row commit together. A relay crash after publish but before marking sent publishes the same event ID again on restart.
Limit. The relay is at-least-once, not magically exactly-once. Consumers dedup transactionally; preserve per-entity sequence when order matters. CDC is change data capture: reading committed database changes.
Backpressure and load shedding
Backpressure tells an upstream producer that the downstream stage cannot accept more work. Buffering delays that signal; shedding deliberately rejects or drops lower-priority work when a bound is reached.
Example. A sink accepts 100 events/s while input is 150/s. A 1000-event queue fills in 20 seconds. The system must then block, reject, or apply its documented drop policy.
Limit. A finite queue cannot absorb an infinite outage. Prioritized messages still need hard quotas, deadlines, and bounded client queues; otherwise an exemption can exhaust the system.
1. Clarifying Questions and Requirements
Spend the first few minutes asking questions. Each answer removes a branch of the design. Good questions for a video platform:
Which features? Upload and watch are core. Comments, likes, search, recommendations and live streaming are usually out of scope. Say so out loud.
How big? Daily active users, views per user, and the share of users who upload.
How long and how large are videos? This sets upload size, transcode time and storage.
Which devices and regions? This sets the rendition ladder, the codecs, and how global the CDN must be.
How fast must a new video be watchable? Minutes is normal. Seconds would need a very different pipeline.
Acknowledged completed uploads have verified durable objects under the stated fault/retention model; processing/publication is separately recoverable.
Owner can delete or make a video private
Low cost per view, since egress dominates the bill
Intuition
Think of two businesses sharing one warehouse. One is a factory that turns raw uploads into products. The other is a shop that hands products to millions of buyers. The factory cares about throughput. The shop cares about speed and cost. Design them apart.
2. Requirements and Scale
Before drawing boxes, pin down what the system must do and what shape the load takes. A video platform is defined far more by its access pattern than by its feature list, and that pattern drives nearly every later choice.
Requirement
What it implies for the design
Read-dominated
Views vastly outnumber uploads. The streaming path must be cheap and fast per request, which points straight at caching and a CDN rather than serving bytes from origin storage.
Global audience
Viewers are everywhere, so content must be served from edge locations near them. Round-trips to a single region would dominate startup latency.
Very large files
Source uploads can be gigabytes. Uploads need resumability, direct-to-storage paths, and processing that does not load whole files into a single machine's memory.
Many devices and networks
Phones, TVs, browsers, and flaky connections all need different resolutions, bitrates, and codecs — the core reason transcoding exists.
Two distinct flows
Upload/processing (write-heavy, compute-heavy) and streaming (read-heavy, latency-sensitive) are designed and scaled independently.
A good interview framing: state early that you are optimizing the read path for latency and cost (CDN, caching, adaptive bitrate) and the write path for throughput and resilience (durable original storage, parallel transcoding, retryable pipeline). Keeping the two flows mentally separate prevents most muddled designs.
3. Back-of-Envelope Estimation
Assumptions: 100M daily active users, 5 views each per day, videos 5 minutes long, and 1% of users upload one video a day. Playback averages 2.5 Mbps. Traffic peaks at 3× the average.
Quantity
Math
Result
Views per day
100M × 5
500M
View starts per second
500M / 86,400; ×3 at peak
5,787 avg / 17,361 peak
Bytes per view
300 s × 2.5 Mbps / 8
93.75 MB
Egress per day
500M × 93.75 MB
46.9 PB
Egress bandwidth
46.9 PB × 8 / 86,400; ×3 at peak
4.34 Tbps avg / 13.0 Tbps peak
Uploads
1M per day / 86,400
11.6 per second
Raw storage
1M × 300 MB (8 Mbps source)
300 TB/day
Rendition storage
ladder 0.4+0.8+1.4+2.8+5.0 = 10.4 Mbps × 300 s = 390 MB each
390 TB/day
Transcode compute
1M × 300 s × 5 renditions × 1 core-s per s / 86,400
17,361 cores busy
Total stored with 3 replicas
(300 + 390) TB × 3
2.07 PB/day ≈ 756 PB/yr
What the numbers tell you
Uploads are tiny (12 per second). Egress is huge (13 Tbps at peak). At $0.01 per GB, 46.9 PB a day costs about $469,000 a day. So the CDN hit ratio and the bitrate ladder matter more than any database choice. Storage grows by petabytes a day, so cold renditions must move to cheaper tiers.
4. API Design and Data Model
Keep the API small. Heavy bytes never pass through the API servers.
POST /v1/videos # {title, size, content_type} -> {video_id, upload_id}
PUT {presigned_url}?part=N # client uploads 8-64 MB parts straight to blob storage
POST /v1/videos/{id}/complete # {upload_id, part_etags[]} -> state=uploaded, enqueue transcode
GET /v1/videos/{id} # metadata + state (uploaded | processing | ready | failed)
GET /v1/videos/{id}/manifest # -> CDN URL of master.m3u8 / manifest.mpd (signed if private)
DELETE /v1/videos/{id} # soft delete; purge CDN and blobs asynchronously
Resumable upload means the client can ask which parts arrived and resend only the missing ones. A lost connection at 90% then costs one part, not the whole file.
A viral video can get thousands of views per second. Updating one row that often creates a hot spot. Count views in a stream, add them up in memory, and write totals every few seconds. Viewers accept a count that lags a little.
5. The Upload Flow and Architecture
Create a durable upload session bound to owner, object key, size/type policy, and expiry; issue scoped upload permission. On completion, verify durable object/checksum and atomically transition metadata with transcode intent/outbox. Direct blob upload does not make DB→queue publication atomic.
Workers validate untrusted media under resource/sandbox limits, persist task state, and write attempt-specific immutable outputs. Completion events are idempotent and match current job/attempt version. Publish ready manifests only after all required artifacts exist; clients may see uploaded/processing/failed states meanwhile. CDN cache misses do reach origin, and edge proximity/routing is not a guarantee of nearest geography.
6. Why Transcoding Is Necessary
You cannot simply store the uploaded file and stream it back. Source videos arrive in a huge variety of containers (MP4, MOV, MKV, WebM), codecs (H.264, H.265, VP9, AV1), resolutions, and bitrates — whatever the creator's camera or editor happened to produce. The viewer's side is just as varied: an old phone on a 3G connection and a 4K television on fiber cannot consume the same stream.
Transcoding bridges that gap by producing, from one source, a family of renditions:
Multiple resolutions — e.g. 240p, 480p, 720p, 1080p, 4K — so each device gets a size it can decode and display.
Multiple bitrates per resolution, so a player can drop to a lower-bandwidth version when the network degrades and climb back when it recovers.
Multiple codecs, because device support varies and newer codecs (AV1, H.265) cut bandwidth at the cost of more encoding work.
The renditions are not delivered as monolithic files. Each is cut into short segments (a few seconds each) and described by a manifest. This is the basis of adaptive bitrate streaming (ABR), standardized as DASH and HLS: the player reads the manifest, then chooses, segment by segment, the highest-quality rendition the current bandwidth can sustain. Without transcoding there is nothing for the player to adapt between.
Dimension
Why it varies
Tradeoff
Resolution
Screen size and decode capability differ across devices.
Higher resolution = better picture, more bytes and more encode cost.
Bitrate
Available bandwidth fluctuates mid-playback.
Lower bitrate = smoother on bad networks, softer image.
Codec
Device support and compression efficiency differ.
Modern codecs save bandwidth but cost far more CPU to encode.
7. Transcoding as a DAG
A directed acyclic graph has no dependency cycle. It permits topological scheduling, but does not prove tasks terminate or succeed. Inspect/validate the source before choosing rendition parameters; thumbnail and audio branches may run independently after their actual prerequisites.
Encoders usually apply visual watermarks while re-encoding decoded frames; a post-encode pixel overlay may require another decode/encode pass. Distinguish visual overlay from metadata/provenance watermarks. Packaging waits for required compatible audio/video artifacts, aligned keyframes/segments, and checksums before publishing the manifest.
Each task has a durable identity, attempt epoch, bounded lease/deadline/retries, and immutable output. A late timed-out attempt cannot mark a replacement job complete. Conditional artifact/manifest publication and garbage collection prevent partially ready or superseded outputs becoming visible.
8. Orchestrating the DAG
A DAG is just a description; something has to turn it into running work, in the right order, on a finite pool of machines. That orchestration splits into two collaborating pieces: a scheduler that knows the shape of the graph, and a resource manager that knows the state of the worker pool.
A preprocessor turns the request into a DAG; the DAG scheduler orders the stages; the resource manager assigns work to task workers; intermediate artifacts pass through temporary storage between stages, and the final output is the encoded video.
The flow through the orchestration layer is:
Preprocessor. Takes the incoming video and configuration and builds the concrete DAG — which renditions, which tasks, what dependencies. It also splits the source if needed and stages inputs.
DAG scheduler. Walks the graph in dependency order, releasing a task for execution only once all of its upstream tasks have completed. It is the component that understands "what can run now."
Resource manager. Takes runnable tasks from the scheduler and matches them to available workers. It is the component that understands "where can this run."
Task workers. A pool of (often specialized) workers that actually execute tasks — encoding, thumbnailing, watermarking, merging.
Temporary storage. Tasks rarely hand artifacts directly to one another. Instead each writes its output to temporary storage and the next task reads from it. This keeps workers stateless and decoupled, and lets a rerun pick up the previous stage's output without recomputing it.
The resource manager deserves its own look, because matching tasks to workers efficiently is where throughput is won or lost.
The resource manager keeps a task queue, a worker queue, and a running queue. A task scheduler pairs the head of the task queue with a free worker and dispatches to specialized task workers — watermark, encoder, thumbnail, merger.
Internally the resource manager coordinates three queues:
Queue
Holds
Purpose
Task queue
Tasks the scheduler has marked runnable (dependencies satisfied).
The backlog of work waiting for a worker.
Worker queue
Workers currently idle and available.
Capacity waiting for work.
Running queue
Task–worker pairs currently executing.
Lets the system track progress, detect stalls, and reclaim workers on failure.
A task scheduler inside the resource manager repeatedly pulls a runnable task and a free worker, assigns them, and moves the pair into the running queue. When a worker finishes, it returns to the worker queue and its task is reported complete so the DAG scheduler can release downstream tasks. Specializing workers by task type — a dedicated encoder pool, a thumbnail pool, a merger pool — lets each be sized and tuned independently, since encoding is far more expensive than thumbnailing.
function resource_manager_loop():
while True:
if task_queue.empty() or worker_queue.empty():
wait() # nothing to pair yet
continue
task = task_queue.pop() # dependencies already met
worker = worker_queue.pop(matching=task.type)
running_queue.add(worker.run(task)) # dispatch
function on_worker_done(worker, task, output):
temp_storage.put(task.id, output) # stage for next stage
running_queue.remove(worker, task)
worker_queue.push(worker) # worker is free again
dag_scheduler.mark_complete(task) # may release dependents
9. The Decoupled Pipeline
Even within a single worker's job, the work has natural stages: fetch the input, do the heavy compute, write the result back. Coupling those stages in one process means a slow upload to storage stalls a CPU that should be encoding, and any failure throws away the whole job. The fix is to split the work into modules separated by message queues.
Original storage feeds a download module, then a message queue, then an encoding module, then another queue, then an upload module, which writes to encoded storage that the CDN serves. The queues decouple the I/O-bound stages from the CPU-bound encoder.
The pipeline reads left to right, with a queue as the seam between every module:
Download module. Pulls the source segment from original storage and places it where the encoder can reach it. I/O-bound work.
Encoding module. The CPU-heavy stage that actually transcodes the segment. Compute-bound work.
Upload module. Writes the encoded output to encoded storage. I/O-bound again.
Message queues between every stage. Each module consumes from its inbound queue and produces to its outbound queue.
This decoupling pays off in three ways:
Independent scaling. Encoding is the bottleneck, so you run many encoders and only a handful of download/upload workers. Because the stages communicate through queues, you scale each pool to its own load without touching the others.
Failure isolation and retry. If an encoder crashes mid-task, its message stays on the queue (unacknowledged) and another worker picks it up. A failure in one stage does not corrupt or block the others. The catch: this is at-least-once delivery. A worker can finish, write its output, and crash before it acks, so the same task runs twice. Make every stage idempotent: write to a deterministic key such as video_id/rendition/segment_0042.m4s, so a rerun overwrites the same object instead of creating a duplicate.
Smoothing bursts. Uploads arrive in spikes; queues absorb the spike and let the worker pools drain it at a steady rate, so a flood of uploads becomes a deeper queue rather than a pile of dropped jobs.
The queue-between-stages pattern is the same idea as the DAG's temporary storage, applied at finer granularity: stages never call each other directly, they hand off through a durable buffer. That single rule is what makes the whole processing system scalable, resilient, and retryable.
10. Streaming to Viewers
Once renditions exist behind the CDN, the read path is comparatively simple — but it is where the bulk of the traffic lives, so getting it right matters most for cost and experience.
Serve from the edge. The CDN caches segments at edge locations close to viewers. A play request is satisfied from the nearest edge; origin (transcoded storage) is only hit on a cache miss. This is what keeps a global, read-dominated workload affordable and fast.
Adaptive bitrate. The player fetches the manifest, then requests segments from whichever rendition the measured bandwidth can sustain, switching up or down between segments. Startup is fast (begin on a low rendition) and quality climbs as the buffer fills.
Segments and seeking. In the common HLS/DASH setup each segment is its own small file with its own URL, so seeking just means fetching a different segment. Some packagers instead store a whole rendition as one fragmented MP4 file and list byte ranges in the manifest (HLS EXT-X-BYTERANGE, DASH SegmentBase). Then the player uses HTTP Range requests. Both work well with CDN caching. Plain progressive MP4 download also relies on Range for seeking.
# player-side adaptive bitrate loop
manifest = cdn.get(video_id + "/manifest") # lists renditions + segments
bandwidth = estimate_initial()
for segment in manifest.segments:
rendition = highest_rendition_under(bandwidth)
bytes = cdn.get(rendition.segment_url(segment)) # nearest edge; a byte-range GET in single-file mode
play(bytes)
bandwidth = measure(bytes) # adapt for next segment
Beyond throughput
This loop is throughput-based ABR: it trusts the last download speed. Speed estimates are noisy, so real players also look at the buffer. With 30 seconds buffered the player can risk a higher rendition. With 2 seconds left it drops fast to avoid a stall. Buffer-based schemes (BBA) and hybrids (MPC, and the BOLA algorithm used in dash.js) do this. Rebuffering hurts viewers far more than a short dip in quality.
11. Optimizations and Safety
With the core flows in place, a strong answer rounds out the design with the optimizations and safeguards a real platform needs.
Area
Technique
Parallelism
Split long videos into chunks and transcode the chunks concurrently across the worker pool, then stitch them back together. Turns a long serial encode into many short parallel ones.
Presigned upload URLs
Let clients upload directly to blob storage with short-lived, scoped credentials. Keeps multi-gigabyte transfers off the API tier and supports resumable, multipart uploads.
CDN cost and placement
Cache only popular content at the edge; let the long tail fall back to origin or regional caches. Place renditions where the audience is to cut both latency and egress cost.
Deduplication
Hash uploads (for example SHA-256) to detect byte-identical re-uploads and skip redundant storage and transcoding. A hash only matches exact bytes. Most re-shares are re-encoded, trimmed, or cropped, so catching those needs perceptual fingerprinting (the Content ID idea). That is used more for copyright and policy than for saving storage.
Content safety and inspection
The inspection task can also screen for malformed, malicious, or policy-violating content before a video is published, gating the completion handler on the result.
Two of these deserve emphasis in an interview. Chunked parallel encoding is the single biggest lever on processing latency, and it composes cleanly with the DAG and queue-based pipeline already described — each chunk is just more independent tasks. Content safety is easy to forget but expected at scale: the same inspection stage that validates format is the natural place to enforce policy before anything reaches the CDN.
12. Failure Modes
What fails
What the viewer or creator sees
Mitigation
Upload connection drops
Upload stalls at 60%
Multipart upload; resume only missing parts; abort stale uploads after a day
Transcode worker crashes
Video stuck in "processing"
Unacked message is redelivered; idempotent outputs; retry limit, then a dead-letter queue and state=failed
Poison video (corrupt file)
Crashes every worker it touches
Inspect first; cap retries; quarantine to the dead-letter queue
CDN edge or PoP down
Brief stall
DNS or anycast steers to the next PoP; the player retries the segment
Viral video misses the cache everywhere
Origin overload (thundering herd)
Tiered caching (edge → regional shield → origin); request coalescing at the shield
Metadata DB shard down
Video page will not load
Replicas with failover; the cache serves reads for popular videos
Region loss
Uploads fail in that region
Originals replicated across regions; playback still served by the CDN
13. Tradeoffs
Decision
Option A
Option B
Usual pick
When to transcode
Every rendition up front
On demand for rare ones
Up front for popular formats; lazy or skipped for the long tail
Codec
H.264: plays everywhere, cheap to encode
AV1/VP9: 30–50% fewer bytes, far more encode CPU
H.264 for all; add AV1 only for videos that get many views
Segment length
2 s: fast start and quick switching
6–10 s: fewer requests, better compression
About 4–6 s for on-demand video
CDN
Buy a commercial CDN
Build your own edge (Google GGC, Netflix Open Connect)
Buy first; build only at huge scale
View count
Exact and strongly consistent
Approximate and delayed
Approximate; fix up in batch
Worked example: a stale encoder finishes after retry
Rendition task T has an old attempt A1 and replacement A2.
Read top to bottom. The table below explains the state and guarantee at each step.
Step
State, calculation, and reasoning
1. Validate the original
A scoped upload session verifies ownership, size, checksum, and source metadata before scheduling encode parameters.
2. Lease attempts separately
A1 times out; A2 receives a newer attempt epoch. Encode to attempt-specific immutable output paths.
3. A1 returns late
Conditional completion rejects A1's stale epoch. Do not let it replace A2's manifest or mark an obsolete job ready.
4. Publish the ready manifest
Commit required rendition/artifact references and publish intent atomically. Align segment/keyframe boundaries for adaptation; retries and CDN authorization follow separate policies.
Check your understanding: Does an acyclic DAG guarantee every job terminates?
No. It excludes dependency cycles, but tasks can hang, fail, or retry forever. Bound attempts/deadlines and persist failure/cancellation states.
Run a small, testable model
Download concepts.py. It uses only the Python 3.10+ standard library. From the directory where you saved it, run:
python3 concepts.py workflow
python3 concepts.py test
The model keeps unknown outcomes explicit; adapt its operation/attempt identity to encoder completion and publication.
The program contains deterministic assertions and failure traces. It is a learning model, not a production service: it does not implement network consensus, distributed durability, authentication, or third-party delivery. Read its contracts and try changing an input before extending it.
Deep dives the interviewer picks. Usually the DAG and chunked parallel encoding, ABR with HLS/DASH, or CDN caching and cost.
38–45 min
Failure modes, tradeoffs, and what you would build next (live streaming, recommendations, regional compliance).
Interview angle
Say the key sentence early: "The write path is optimized for throughput and retry. The read path is optimized for latency and cost." Then every later choice can point back to it.
15. Summary
A video platform is two flows joined by storage and a CDN, each built from a few repeating ideas:
Concern
Mechanism
How does a huge upload reach storage?
Pre-signed URL straight to blob (original) storage; the API tier never handles the bytes.
How is upload progress tracked?
Metadata DB (with a cache) records state; a completion queue and handler flip state to ready when processing finishes.
Why transcode at all?
Sources vary in format; viewers vary in device and bandwidth. Multiple resolutions/bitrates/codecs enable adaptive bitrate (DASH/HLS).
How is the processing structured?
A DAG: split into video/audio/metadata, fan out parallel tasks (inspect, transcode, thumbnail, watermark), then assemble.
What runs the DAG?
A preprocessor builds it, a DAG scheduler orders it, a resource manager with task/worker/running queues assigns it to specialized workers; temp storage passes artifacts between stages.
How does processing scale and survive failure?
Modules (download → encode → upload) separated by message queues: each scales independently, failures are isolated, work is retryable.
How are videos delivered?
CDN edge caching close to viewers, adaptive bitrate, and HTTP range requests for seeking.
What rounds out a real design?
Chunked parallel encoding, presigned uploads, CDN cost/placement, dedup, and content safety via the inspection stage.
The recurring theme: separate the read-heavy streaming path from the write-heavy processing path, express transcoding as a DAG of parallelizable tasks, and decouple every stage with a durable buffer — temporary storage between DAG stages, message queues between pipeline modules — so the system scales, isolates failure, and retries cleanly. The CDN does the rest by putting bytes next to the viewer.
16. Common Mistakes
Mistakes students make
These cost points in interviews.
Streaming bytes through the API servers. Use presigned URLs, so the API tier handles only metadata.
Calling every DAG task independent. Draw the real edges. Watermark and packaging wait for the encode.
Forgetting at-least-once delivery. Queue retries run tasks twice. Outputs need deterministic keys.
Treating a file hash as copy detection. A hash catches exact copies only. Re-encoded copies need fingerprints.
Skipping the numbers. Without the egress estimate, you cannot explain why the CDN matters most.
17. Check Yourself
Why should the client upload straight to blob storage instead of through the API servers?
Answer
Uploads are gigabytes. Pushing them through the API tier ties up its bandwidth and memory for no benefit. A presigned URL lets storage take the bytes directly. The API servers only sign a short-lived, scoped permission and record metadata.
With 500M views a day at 93.75 MB each, what is the average egress in Tbps?
Answer
500M × 93.75 MB = 46.875 PB a day. Times 8 bits gives 375 Pb. Divided by 86,400 s that is about 4.34 Tbps on average, and 13 Tbps at a 3× peak.
An encoder finishes a segment, writes it to storage, and crashes before acking. What happens, and how do you keep it safe?
Answer
The queue redelivers the message, so another worker encodes the same segment again. That is safe if the output key is deterministic, for example video/rendition/segment_N. The rerun overwrites the same object with the same bytes.
Why does the watermark task wait for transcode, while thumbnail does not?
Answer
The watermark is stamped onto each output rendition, so that rendition must exist first. A thumbnail only needs frames from the source video, so it can start right after the split.
The player has 25 s of video buffered and just measured a slow download. Should it drop quality at once?
Answer
Not necessarily. With a large buffer it can ride out a short dip. Buffer-aware ABR only drops quality when the buffer runs low. Pure throughput-based ABR would drop at once and switch quality too often.
Why would you not encode every video in AV1?
Answer
AV1 saves bandwidth but costs many times more CPU to encode. Most videos get few views, so the savings never pay back the encode cost. Encode AV1 only for videos with many views, where saved egress is large.
18. Key Terms
Rendition
One encoded version of a video at a given resolution, bitrate and codec.
Segment
A short chunk of a rendition, usually 2–10 s, fetched as one unit.
Manifest
A playlist (HLS .m3u8 or DASH .mpd) listing renditions and segments.
ABR
A player chooses among compatible encoded renditions segment by segment, using measured download throughput and buffered playback time. HLS/DASH manifests describe the renditions and segment locations. Example and limits above.
Presigned URL
A short-lived URL that grants one specific storage action.
DAG
Directed acyclic graph of tasks and their dependencies.
Idempotent
Running it twice has the same effect as running it once.
Origin shield
A mid-tier cache that shields origin from edge misses.
19. Further Reading
Martin Kleppmann, Designing Data-Intensive Applications (O'Reilly), ch. 11 on stream processing and message brokers.
Alex Xu, System Design Interview – An Insider's Guide, the chapter "Design YouTube".
Apple, HTTP Live Streaming, RFC 8216 (IETF).
MPEG-DASH, ISO/IEC 23009-1, and the open-source reference player dash.js.
Netflix Technology Blog posts on per-title encoding and Open Connect.
