← CatZep/Engineering Guides/System Design/Media & Collaboration

Designing a Cloud File Storage / Sync Service

A developer's guide to designing a Google Drive or Dropbox style service: how files move between a user's devices, how only the changed parts of a file travel over the wire, where bytes and metadata actually live, and how concurrent edits and offline clients are reconciled without losing work.

A cloud file storage service lets a user keep a folder of files in sync across every device they own, with the cloud as the authoritative copy. The surface looks simple — drop a file in a folder and it appears everywhere — but underneath there are two largely separate problems. One is storing bytes: taking possibly enormous files, breaking them into pieces, deduplicating and compressing them, and parking them durably in a blob store. The other is tracking truth: a metadata layer that records which files exist, who owns them, who they are shared with, what their revision history is, and which blocks make up each version. The interesting design work is in keeping those two layers consistent and in pushing changes out to other devices quickly. This guide walks through the requirements, then the upload path, the block and metadata layers, storage tiering, deduplication, sync and conflict handling, the notification path, and the consistency model that ties it together.

Concepts to understand first

Each definition includes a concrete example and the condition under which the mechanism is useful. You can read this guide without opening another guide.

1. Clarifying Questions

Ask before you design. Each answer changes a box on the diagram.

• Sync client or web only? A desktop sync client means delta sync, offline edits and conflicts.
• How big? Users, daily active users, storage quota, and a typical file size.
• Max file size? This decides resumable uploads and chunk size.
• Collaborative editing? Real-time co-editing (Google Docs) is a different system. Usually out of scope here.
• How long are revisions kept? This drives storage cost and garbage collection.

2. Requirements

Before drawing boxes it helps to pin down what the service actually promises. The functional requirements are modest in number but each one shapes the design.

Uploaded versions must remain durable under a defined failure/retention model. Define separate guarantees for atomic version commits, read-your-writes synchronization, bounded-stale metadata reads, and fresh authorization. A stale immutable version can be coherent without being current; keep its chunks pinned while it is readable rather than calling this 'strong-ish'.

3. Back-of-Envelope Estimation

Assumptions: 50M users with 10 GB each, 10M daily active users, 2 file uploads each per day, 500 KB average file, 30% saved by dedup, and 3 replicas. Peak is 2× average.

4. API Design and Data Model

POST /v1/uploads                        # {path, size} -> {session_id}
PUT  /v1/uploads/{sid}?offset=N         # chunk body; returns committed offset
POST /v1/blocks/query                   # {hashes[]} -> which ones the server lacks
PUT  /v1/blocks/{hash}                  # upload one missing block
POST /v1/files/{id}/versions            # {base_version, block_hashes[]} -> new version or CONFLICT
GET  /v1/files/{id}?version=V           # metadata + ordered block list
GET  /v1/changes?cursor=C               # long poll; returns changes since cursor + new cursor
POST /v1/shares                         # {file_id, grantee, role}

The changes cursor matters. A device that was offline sends its last cursor and gets every change since. This is the durable form of the "offline backup queue".

Shard metadata by owner or by shared namespace, so a folder tree and its change journal live on one shard. Then a commit is a single-shard transaction.

5. High-Level Architecture

A client talks to the service through a load balancer that fans requests out across a fleet of stateless API servers. Those servers own the metadata layer; the heavy lifting of file bytes is delegated to a separate tier of block servers writing to cloud storage. A notification service runs alongside to push changes back out to clients.

The components and their jobs:

• Load balancer: spreads client connections across the API fleet and removes any single server as a bottleneck or a point of failure.
• API servers: stateless front-end servers that authenticate requests, enforce sharing permissions, and orchestrate the metadata reads and writes. Being stateless, they scale horizontally — any server can handle any request.
• Metadata DB: the source of truth for files, folders, users, versions, sharing, and the file→block mapping. Transactional, because correctness here matters more than raw throughput.
• Metadata cache: an in-memory cache (for example, a key-value store) that absorbs the heavy read traffic on metadata so the database is not hammered for every directory listing.
• Block servers: the tier that turns a file into blocks (and back), compresses and encrypts them, and reads/writes them to the blob store. They carry the bandwidth, not the API servers.
• Cloud storage: the durable blob store that holds the actual file blocks, replicated for durability.
• Cold storage: a cheaper, slower tier that rarely-accessed blocks are migrated to in order to cut cost.
• Notification service: watches for committed changes and pushes them to subscribed devices over long-lived connections.
• Offline backup queue: holds the change events destined for devices that are currently offline, so nothing is missed and the events are delivered on reconnect.

6. Upload Strategies

Not all uploads are the same. A 20 KB note and a 4 GB video have completely different failure profiles, and the protocol should reflect that. The service supports two upload modes.

Resumable upload is what makes large-file sync feel robust. Because the session tracks progress on the server side, the client can crash, the laptop can sleep, the network can flap, and the upload still picks up where it left off rather than starting over.

# resumable upload: client side
function upload(file):
  if file.size <= SIMPLE_UPLOAD_LIMIT:
    POST /upload  body=file              # one shot, retry whole on fail
    return

  session = POST /uploads               # open session, get session_id
  offset = 0
  while offset < file.size:
    chunk = file.read(offset, CHUNK_SIZE)
    try:
      PUT /uploads/{session.id}?offset={offset}  body=chunk
      offset += len(chunk)
    except network_error:
      offset = GET /uploads/{session.id}/offset  # resume point
  POST /uploads/{session.id}/complete          # commit the file

7. Block Servers and Delta Sync

This is the central idea of the whole design. Files are not stored as single opaque blobs; each file is split into fixed-size blocks (a few megabytes each). The list of blocks, in order, is the file. Two things fall out of this that make sync efficient.

An in-place edit changes each block it overlaps. It may cross a boundary and affect multiple blocks; compressed/encrypted file formats can propagate a small logical edit into many changed bytes.

The second is the processing pipeline each block goes through before it lands in storage. On the way down, a block server takes a raw block and:

1. Hashes it, producing a content fingerprint used for deduplication and as the block's identity.
2. Compresses it, so the bytes stored and transferred are smaller.
3. Encrypts it, so what lands in the blob store is unreadable without the key.
4. Writes the resulting object to cloud storage, then records the block in the file→block mapping in metadata.

Download is the same steps in reverse: fetch the blocks for the requested version, decrypt, decompress, and reassemble them in order. Splitting files this way is what lets delta sync, deduplication, and tiering all operate at the granularity of a block rather than a whole file.

# client computes the delta, block servers process each block
function sync_changed_file(file, known_blocks):
  local_blocks = split_into_blocks(file)       # fixed-size or content-defined
  for block in local_blocks:
    h = hash(block)
    if h not in known_blocks:                  # only changed blocks
      payload = encrypt(compress(block))
      block_server.put(h, payload)             # write to blob store
  metadata.commit_version(file, [hash(b) for b in local_blocks])

Inserts break fixed-size blocks: content-defined chunking

Fixed-size blocks have a trap. Insert one byte near the start of a file and every later byte shifts right. Every block boundary now cuts different bytes, so every block after the edit gets a new hash. The "one small edit" re-uploads almost the whole file.

Content-defined boundaries usually resynchronize after local edits, allowing unchanged chunks to retain identities. The number/bytes of changed chunks is workload-dependent, not guaranteed one; bound minimum/maximum sizes and verify reconstruction.

A small runnable demo. It inserts 12 bytes near the start of a 64 KB file and counts how many chunks must be uploaded again:

import hashlib, random

def fixed_chunks(data, size=1024):
    if type(size) is not int or size <= 0:
        raise ValueError("positive integer chunk size required")
    return [data[i:i+size] for i in range(0, len(data), size)]

def cdc_chunks(data, gear, mask=1023, minimum=256, maximum=4096):
    """Bounded educational gear-hash chunking, not a production CDC standard.
    Invariant: emitted nonempty chunks partition data[:start], each <= maximum.
    Bound: remaining input bytes. The last chunk may be shorter than minimum."""
    if len(gear) != 256 or minimum <= 0 or maximum < minimum or mask <= 0:
        raise ValueError("valid gear table and positive chunk bounds required")
    chunks, start, fingerprint = [], 0, 0
    for i, byte in enumerate(data):
        fingerprint = ((fingerprint << 1) + gear[byte]) & 0xffffffff
        size = i+1-start
        if size >= maximum or (size >= minimum and fingerprint & mask == 0):
            chunks.append(data[start:i+1])
            start, fingerprint = i+1, 0
    if start < len(data):
        chunks.append(data[start:])
    return chunks

def chunks():
    rng = random.Random(7)
    gear = [rng.getrandbits(32) for _ in range(256)]
    before = bytes(rng.getrandbits(8) for _ in range(64000))
    after = before[:1000] + b"hello, world" + before[1000:]
    for label, split in (("fixed", fixed_chunks), ("content-defined", lambda data: cdc_chunks(data, gear))):
        a, b = split(before), split(after)
        assert b"".join(a) == before and b"".join(b) == after
        assert all(0 < len(chunk) <= 4096 for chunk in b)
        known = {hashlib.sha256(chunk).digest() for chunk in a}
        changed = [chunk for chunk in b if hashlib.sha256(chunk).digest() not in known]
        print(label, "chunks", len(b), "; changed chunks", len(changed), "; changed bytes", sum(map(len, changed)))
    assert cdc_chunks(b"", gear) == []

chunks()

Output:

fixed 1 KB       chunks= 63  must upload=63
content-defined  chunks= 61  must upload=1

For this seeded 64 KB input, the bounded model reports 63 changed fixed-size chunks (64,012 bytes) versus one changed content-defined chunk (1,741 bytes). That is an observed example, not a general one-chunk guarantee.

8. Metadata and Caching

If the blocks are the bytes, the metadata is the meaning. The metadata DB records everything the byte layer does not understand:

• Files and folders: names, paths, sizes, timestamps, and the tree structure they form.
• Users and ownership: who owns what, and their account-level state.
• Versions: the revision history of each file, each version pointing at the ordered list of blocks that compose it.
• Sharing: which files and folders are shared with which users, at what permission level.
• File→block mapping: the heart of reassembly — for a given version, the ordered block hashes needed to rebuild the file.

Metadata reads vastly outnumber writes. Every directory listing, every sync check, every "has this changed?" poll touches metadata. To keep the database from being the bottleneck, hot metadata is served from a metadata cache sitting in front of it. A read first checks the cache; on a miss it falls back to the database and populates the cache. The catch is that a cache introduces the possibility of staleness, which is exactly the problem the consistency model (section 10) has to solve.

# read-through metadata cache
function get_metadata(key):
  value = cache.get(key)
  if value is not None:
    return value                  # cache hit
  value = metadata_db.get(key)    # miss: fall back to source of truth
  cache.set(key, value)
  return value

9. Storage Tiering

Most files are touched heavily right after they are created and then sit untouched for months or years. Treating every block as equally hot wastes money, because the fast, replicated storage that serves active files costs far more per byte than archival storage. The service therefore tiers blocks by access pattern.

A background process watches access timestamps and moves blocks that have not been read in a long while down to cold storage, updating their location in metadata. If a user later opens an archived file, the block is retrieved from the cold tier — slower, but rare, and the cost savings on the long tail of dormant data are large.

10. Deduplication

Content-addressed chunks can share bytes within an authorized dedup domain under a cryptographic collision model. Use conditional/idempotent object writes and immutable metadata references. Bare exists→put→refcount.set races and can lose references.

Garbage collection derives liveness from retained manifests and active session/read pins, using an epoch/grace or equivalent safe sweep protocol. Bound session lifetime; a grace period shorter than an active upload is not enough. Verify block ownership rather than letting a client reference any known hash. Cross-tenant existence checks and convergent encryption leak content information; a hash is not authorization or general anonymization. Tenant-scoped dedup is often safer.

11. Sync and Conflict Handling

Commit latest-version changes through an atomic compare-and-set transaction: verify base_version and permissions, insert the immutable version manifest, advance latest, and insert notification/outbox intent. A read of latest followed by unconditional add_version is racy: two devices can both pass the check.

On conflict, retain both edits, create a conflict copy, or invoke a format-specific reviewed merge. Block-level changes do not necessarily merge opaque formats semantically. Replayed commits use a stable operation identity; notifications can duplicate and devices catch up through a durable change cursor. Access/ownership changes are checked when committing and downloading.

12. Notification Service

Once a change is committed, the other devices need to learn about it without constantly asking. The service uses long polling: a device opens a request to the notification service that the server holds open instead of answering immediately. The connection stays parked until either a change relevant to that device occurs — at which point the server responds and the device pulls the update — or a timeout fires, after which the device simply re-opens the connection. This gives near-real-time delivery for online devices without the cost of a flood of empty polls.

Long polling only works for devices that are currently connected. A device that is offline — laptop closed, phone out of signal — cannot hold an open connection, yet it must not miss the changes that happened while it was away. That is the job of the offline backup queue: change events destined for a disconnected device are durably queued, and when the device reconnects it drains the queue and applies everything it missed before resuming normal long polling.

# notification fan-out on a committed change
function publish(file, recipients):
  for device in devices_of(recipients):
    if device.online:
      long_poll(device).respond(change_event(file))  # push now
    else:
      offline_queue(device).enqueue(change_event(file)) # deliver later

function on_reconnect(device):
  for event in offline_queue(device).drain():
    device.deliver(event)                # catch up on missed changes
  open_long_poll(device)                  # resume real-time path

13. Consistency Model

Metadata cache entries have explicit freshness semantics; TTL/invalidation do not by themselves guarantee latest committed state. A delayed reader can fetch old DB state, a writer commits and invalidates, then the reader refills old state. Versioned entries only help if a durable invalidation/version floor prevents a late fill after the key was deleted. Fill leases invalidate stale tokens; simpler TTL schemes provide only the stated weaker policy.

Base-version checks and authorization use the authoritative transaction, not an untrusted stale cache. Version manifests are immutable and chunks remain pinned while retained/readable. Stale metadata can remain coherent with its version without being current; preserve that distinction instead of promising 'strong-ish'. Revocation policy may require a fresh authorization check even when immutable bytes are cached.

14. Failure Modes

15. Tradeoffs

Worked example: two devices commit from version 7

Device A and B both edit base version 7.

Run a small, testable model

Download concepts.py. It uses only the Python 3.10+ standard library. From the directory where you saved it, run:

python3 concepts.py chunks
python3 concepts.py test

The bounded chunking model reconstructs the exact input, avoids empty trailing chunks, and reports changed bytes as well as changed chunks. The version-CAS model is also available with python3 concepts.py versions.

The program contains deterministic assertions and failure traces. It is a learning model, not a production service: it does not implement network consensus, distributed durability, authentication, or third-party delivery. Read its contracts and try changing an input before extending it.

16. 45-Minute Interview Walkthrough

17. Summary

The design comes down to a handful of decisions that reinforce each other:

18. Common Mistakes

• Claiming fixed blocks make every edit cheap. Inserts shift all later blocks. Use content-defined chunking.
• Reference counts with check-then-act. They race. Record references in version commits and use mark-and-sweep GC with a grace period.
• Saying "invalidate after write" is always fresh. A slow reader can refill the cache with old data. Use leases, versions or TTLs.
• A notification queue with no cursor. If the queue is lost, devices never catch up. A change journal plus a cursor fixes this.
• Global dedup with per-user keys. The two do not mix. Pick one and state the tradeoff.

19. Check Yourself

1. A user adds one byte at the start of a 40 MB file split into 4 MB fixed blocks. How many blocks change?
   Answer

   All 10, plus a new small last block. Every byte after the insert shifts by one, so every block holds different bytes and gets a new hash. Content-defined chunking would change only about one chunk.

2. Why does content-defined chunking use a minimum and a maximum chunk size?
   Answer

   The cut rule is random-like. Without limits, some chunks would be a few bytes (too much metadata) and some would be huge (bad dedup and large re-uploads). The limits keep chunks near the target size.

3. Two uploads of the same new block run at once through store_block. What can go wrong?
   Answer

   Both see exists as false, both put the block, and both set the refcount to 1. The real count should be 2. Later one delete drops it to 0 and the block is removed while a file still uses it.

4. Describe the stale-cache race with "write DB, then invalidate".
   Answer

   A reader misses and reads the old value from the DB. The writer then updates the DB and invalidates. The slow reader then sets the old value in the cache. It stays stale until a TTL or the next write. Leases or versioned sets block the late set.

5. With 10M daily active users each uploading 2 files of 500 KB, 30% dedup and 3 replicas, how much storage is added per day?
   Answer

   20M × 500 KB = 10 TB. After dedup that is 7 TB. With 3 replicas it is 21 TB a day, about 7.7 PB a year.

6. A laptop was offline for a week. How does it catch up without missing anything?
   Answer

   It sends the last cursor it saw to /changes. The server returns every journal entry after that cursor. The laptop pulls the new block lists and fetches only blocks it lacks.

20. Key Terms

Block / chunk

A piece of a file, stored and named by its hash.

Delta sync

Sending only the chunks that changed.

Content-defined chunking

Splits a file at boundaries chosen from its bytes rather than fixed byte offsets, using a rolling/fingerprint scheme with minimum and maximum chunk sizes. Unchanged regions can recover the same boundaries after a local edit. Example and limits above.

Rolling hash

A hash over a sliding window, cheap to update per byte.

Content addressing

Naming data by the hash of its bytes.

Mark-and-sweep GC

Mark live blocks, then delete the unmarked ones.

Convergent encryption

Encryption keyed by content hash, so dedup still works.

Lease (cache)

A token that makes a late cache fill fail after an invalidate.

21. Further Reading

• Andrew Tridgell and Paul Mackerras, "The rsync algorithm" (1996 technical report).
• Wen Xia et al., "FastCDC: a Fast and Efficient Content-Defined Chunking Approach for Data Deduplication", USENIX ATC 2016.
• Rajesh Nishtala et al., "Scaling Memcache at Facebook", NSDI 2013 (leases and invalidation).
• Martin Kleppmann, Designing Data-Intensive Applications, ch. 5 (replication and conflicts).
• Alex Xu, System Design Interview – An Insider's Guide, the chapter "Design Google Drive".