Distributed Systems Patterns Cheat Sheet

In the interviews I run, most candidates know the concepts but struggle with the names, and some don't know a concept at all. In projects I've seen, most of the time goes into managing state and failures. I wanted a one-page reference I can check before a design review or a system design interview, with a link to the deep dive for every concept.

This cheat sheet groups the concepts by the failure they handle. It covers all Patterns of Distributed Systems catalog plus the service-level patterns you meet in microservices: retries, outbox, and sagas. Each entry says what the concept is, shows the smallest example, says when to use it, and names its cost. Where bool.dev already has a deep dive, the entry links to it.

Why distributed systems fail: partial failure

A remote call has three outcomes that look the same to the caller. The request got lost, the server is slow, or the server did the work, and the response got lost. In all three cases, the caller sees a timeout and doesn't know whether the work happened.

That's partial failure: one part of the system breaks while the rest keeps running. It breaks the first two fallacies of distributed computing: "the network is reliable" and "latency is zero". So treat every remote call as potentially slow, lost, or duplicated. Every concept below answers one of those outcomes.

CAP theorem, PACELC, and consistency models

CAP theorem

During a network partition, a system must choose: answer every request (availability) or return only up-to-date data (consistency). It can't do both. Gilbert and Lynch proved this in 2002.

Example: two replicas lose the link between them. A client writes to replica A. If replica B keeps serving reads, they're stale (AP). If B refuses reads, it's unavailable (CP).

Use it when: you decide what a service does while its replicas can't talk to each other.

Note: Read more in CAP Theorem.

PACELC

PACELC extends CAP to the normal case. If there's a Partition, choose Availability or Consistency. Else, choose Latency or Consistency. Daniel Abadi proposed it in 2012, because the latency trade-off shapes real databases more often than partitions do.

Example: a write that waits for a replica in another region before it returns is consistent, and it pays that region's round trip on every request.

Use it when: you compare databases or replication settings.

Note: one database can land in different PACELC classes depending on its configuration, so classify your setup, not the product.

Consistency models

A consistency model says which values a read may return. Weaker models trade freshness for latency and availability:

I pick the consistency level per operation, not per database: a payment needs a linearizable read, a like counter doesn't.

Note: a stronger model costs latency on every read, and it stops serving reads during a partition. Read more in Consistency Models for Distributed Systems.

ACID vs BASE

ACID describes a transaction: all or nothing, valid before and after, isolated from other transactions, durable once committed. BASE (basically available, soft state, eventually consistent) describes a system that stays up and lets replicas catch up later.

Example: a bank transfer between two accounts in one database is an ACID transaction. A product catalog replicated across regions is usually BASE.

Note: a system can use ACID inside each service and BASE between services. Read more in ACID in Simple Terms and BASE Model in NoSQL Databases.

Write-ahead log and replicated log

Most replicated systems, from PostgreSQL to Kafka to etcd, are built around one append-only log. The patterns in this group describe how a node writes that log, keeps it small, and copies it to other nodes.

Write-Ahead Log

Before a node changes its data, it appends the command to a log file on disk and flushes it. After a crash, the node replays the log and gets back every change it acknowledged. The data structures themselves can stay in memory and reach disk later.

Example: PostgreSQL writes every change to its WAL before it updates the table pages. A crash loses the dirty pages in memory, and recovery replays the WAL.

Use it when: a node must not lose an acknowledged write, and flushing the full data structure on every write is too slow.

Note: every write pays for an fsync. Systems batch several writes into one flush to amortize it (see Request Batch below).

Segmented Log

A single log file grows forever and gets hard to clean up. A segmented log splits it into fixed-size files, and the node starts a new segment when the current one is full. Old segments are separate files, so the node deletes or archives them as a whole.

Example: Kafka rolls a new segment for a partition when the active one reaches log.segment.bytes, 1 GiB by default.

Use it when: the node must trim a log while it keeps writing to it.

Note: a read by offset first has to find the right segment, so systems keep an index per segment.

Low-Water Mark

The low-water mark is the log index below which the node no longer needs the entries, because the state is in a snapshot or the retention period is over. The node deletes the segments below it.

Example: Kafka deletes segments older than retention.ms. A Raft node takes a snapshot and drops the log entries the snapshot covers.

Use it when: the log would otherwise fill the disk.

Note: a follower that falls behind the low-water mark can't catch up from the log, so the leader has to send it a full snapshot.

Replicated Log

A replicated log keeps several nodes in sync by giving all of them the same write-ahead log, in the same order. The leader appends each command, followers copy it, and every node applies the commands to its own state machine in log order. Raft and Multi-Paxos are protocols for exactly this.

Example: every etcd member holds the same Raft log, so every member ends up with the same key-value data.

Use it when: several nodes must hold identical state and survive the loss of a minority.

Note: an entry counts only after a majority stores it, so every write pays at least one round trip to the followers. Read more in Raft Consensus Algorithm Explained.

High-Water Mark

The high-water mark is the index of the last log entry that a majority of nodes has stored. The leader tracks it and sends it to followers. Entries above it may still disappear if the leader fails, so no node shows them to clients.

Example: a Kafka consumer reads only up to the partition's high watermark, the offset that every in-sync replica has copied.

Use it when: clients must never read a write that a leader failure could roll back.

Note: a consumer sees new data only after the slowest required replica copies it, so a slow follower adds read latency.

Singular Update Queue

A node puts every state-changing request into one queue, and a single thread applies them in order. Callers don't block: each gets a future that completes when the thread applies its request. This gives a strict order without locks.

Example: a Raft node applies committed log entries to its state machine one at a time, in log order, on one thread.

Use it when: the order of updates matters and lock contention on shared state is the bottleneck.

Note: one thread is one core, so the queue caps throughput. A slow update holds back every request behind it.

Replication, quorums, and consensus

Replication: Leader and Followers, multi-leader, leaderless

Replication keeps copies of the data on several nodes. The three topologies differ in who accepts writes. The catalog's Leader and Followers pattern is the first row: one node coordinates replication, and the others copy its log.

Use it when: you need reads closer to users or a copy that survives a node loss.

Note: asynchronous replication loses the writes the leader didn't ship before it crashed. Synchronous replication waits for a follower on every write.

Follower Reads

The leader handles every write, but reads can go to followers. That spreads the read load and lets a client read from the replica closest to it.

Example: an app sends reports to a PostgreSQL read replica. Kafka (since 2.4, KIP-392) lets a consumer fetch from the closest replica in its rack.

Use it when: reads outnumber writes, or clients sit far from the leader.

Note: a follower lags behind the leader, so a follower read can be stale. If the client needs read-your-writes, send it the version of its last write and make the follower wait until it has applied that version, or read from the leader.

Quorums: Majority Quorum and R + W > N

A majority quorum means a decision needs more than half of the nodes. Two groups can't both have a majority, so a network split can't produce two independent decisions.

In a leaderless store, the same idea becomes tunable: a write goes to W replicas and a read asks R replicas, out of N. If R + W > N, every read set overlaps the latest write set in at least one replica. Amazon's Dynamo paper describes (N, R, W) = (3, 2, 2) as a common configuration.

Use it when: you size a cluster (majority) or tune a leaderless store for read speed (small R) or write speed (small W).

Note: overlap isn't linearizability. Concurrent writes, sloppy quorums, and slow read repair can still return stale or conflicting values.

Consensus: Raft and Paxos

Consensus gets a group of nodes to agree on one value, or one ordered log, even when some nodes fail. Raft and Paxos need a majority, so 2f + 1 nodes survive f failures: 3 nodes survive one, 5 nodes survive two. Paxos reaches agreement in two phases: a proposer first gets a majority to promise not to accept older proposals, then asks the majority to accept its value. Consensus also needs timeouts to make progress, because no deterministic protocol can guarantee agreement in a fully asynchronous network with even one crash (the FLP result, 1985).

Example: etcd runs Raft and stores the state of a Kubernetes cluster.

I wouldn't implement Raft or Paxos myself in a business system. I'd use etcd, ZooKeeper, or the leader election my database or orchestrator already has.

Note: every write waits for a majority, so a consensus group is slower than a single node, and it stops accepting writes when it loses the majority. Read more in Raft Consensus Algorithm Explained.

Lease, Generation Clock, and fencing tokens

Leader election usually hands out a lease: the right to act as leader for a fixed time, renewed while the leader stays healthy. The trap: the leader pauses (a long GC, a VM migration), its lease expires, a new leader takes over, and the old one wakes up and keeps writing.

A generation clock fixes this. Every new leader gets a higher generation number (Raft calls it the term, Kafka's controller the epoch), and it sends that number with every request. A node that sees a request from a lower generation rejects it.

A fencing token is the same idea applied to a lock. The lock service hands out a number that grows with every grant, and the storage rejects any write with a lower number than the last one it saw.

Use it when: exactly one node may run a job or write a resource at a time.

Note: both only work when the receiver checks the number. A lock without a token is fine for avoiding duplicate work, not for correctness.

Cluster membership and failure detection

A cluster has to know which nodes are alive, who leads, and where the metadata lives. The patterns in this group answer those questions.

HeartBeat

Each node sends a small message to the others at a fixed interval. If a node stays silent longer than a timeout, the others mark it as failed and start a failover.

Example: a Raft leader sends empty append messages as heartbeats. A follower that hears nothing within its election timeout starts an election. In Kubernetes, each kubelet renews a Lease object, and the control plane marks the node not ready when the renewals stop.

Use it when: the cluster must detect a failed node without a human.

When a heartbeat times out, I don't assume the node is dead. It might just be slow, and from the outside you can't tell the difference.

Note: a short timeout detects failures fast, and also flags slow nodes as dead (a GC pause, a busy network). A long timeout avoids false alarms and leaves a dead leader in place longer.

Gossip Dissemination

In a large cluster, sending every update to every node costs n² messages. With gossip, each node passes what it knows to a few random peers every round, and they pass it on. The news reaches the whole cluster in about log(n) rounds, with no central node.

Example: Cassandra nodes gossip once per second with up to three other nodes to share membership and state.

Use it when: a cluster has hundreds of nodes, and membership or metadata changes must spread without a coordinator.

Note: gossip is eventually consistent. For a short time, nodes disagree about who is alive, so it isn't enough for decisions that need one answer, such as electing a leader.

Emergent Leader

Instead of an election, the nodes sort the members by the time they joined the cluster, and the oldest member acts as leader. Every node can compute the same answer from the membership list.

Example: in Hazelcast, every member returns the member list in the same order, and the first one, the oldest member, acts as the master.

Use it when: the cluster already shares a membership list (often by gossip) and needs a coordinator for cluster-level tasks, not for writes.

Note: during a network split, each side can pick its own leader. Use it only for work where a temporary second leader is harmless, or add a majority check.

Consistent Core

Running consensus across a large data cluster is slow. A consistent core is a small cluster, usually 3 or 5 nodes, that runs consensus and stores the metadata: membership, leases, partition assignments, configuration. The large cluster stores the data and asks the core for decisions.

Example: Kubernetes keeps its cluster state in etcd. Kafka keeps its metadata in a KRaft controller quorum (before that, in ZooKeeper).

Use it when: the data cluster is too large for every node to join consensus, and it still needs linearizable metadata.

Note: the core is a dependency of everything else. If it loses its majority, the data cluster can still serve, but it can't change leaders or membership.

State Watch

A client registers interest in a key, and the server pushes a notification when that key changes. The client doesn't have to poll.

Example: Kubernetes controllers watch etcd through the API server and react when a Pod or Deployment changes. ZooKeeper clients set watches on znodes.

Use it when: many clients must react quickly to changes in shared configuration or membership.

Note: a client can miss events while it reconnects. etcd watches resume from a revision for that reason, and a client that falls too far behind has to read the full state again.

Clocks and ordering in distributed systems

Wall-clock time

Each server's clock drifts, and NTP corrections can move it backward. So don't order events across nodes by their timestamps.

Example: two replicas accept writes to the same key 3 ms apart. With last-write-wins, the replica whose clock runs ahead wins, even if its write happened first. The other write disappears without an error.

Note: wall-clock time is fine for logs, TTLs, and dates people read. Time-sortable IDs (Snowflake, ULID, UUIDv7) sort by the clock of the node that made them, so the same skew applies. Read more in Unique ID Generation Cheat Sheet.

Lamport Clock and Version Vector

Leslie Lamport's 1978 clock is just a counter. Each node increments it on every event and sends it with every message, and a receiver jumps to max(local, received) + 1. If event A caused event B, A's counter is lower.

A version vector (often called a vector clock) keeps one counter per node. Compare two vectors: if neither is greater in every position, the events were concurrent, and you have a conflict to resolve.

Use it when: you need causal order (Lamport Clock) or need to detect concurrent writes (Version Vector), as in multi-leader and leaderless stores.

Note: a Lamport clock can't tell concurrent events from ordered ones. Version vectors grow with the number of nodes.

Versioned Value

The store keeps every update to a key as a new version instead of overwriting it. A read asks for the latest version or for the value at a given version or time.

Example: etcd gives every change a cluster-wide revision, and you can read a key at an old revision. CockroachDB stores multiple timestamped versions of each row (MVCC), so a read sees a consistent snapshot without blocking writes.

Use it when: readers need a consistent snapshot, history, or a watch that resumes from a version.

Note: old versions use disk until compaction removes them, and a read at a compacted version fails.

Hybrid Clock and Clock-Bound Wait

CockroachDB and YugabyteDB order transactions with a hybrid logical clock (HLC). It pairs physical time with a logical counter, so timestamps stay close to real time while still respecting causality.

Clock-bound wait handles the rest of the clock uncertainty: before a node uses a timestamp, it waits out the maximum clock offset, so no other node can still hand out an earlier one. Google Spanner does this with TrueTime, which returns an interval with a known uncertainty, and a commit waits until the uncertainty passes. The Spanner paper reports the uncertainty as a sawtooth from about 1 to 7 ms, "4 ms most of the time", using GPS receivers and atomic clocks.

Note: HLCs still assume bounded clock skew. Clock-bound wait adds the uncertainty to every write's latency, so it only pays off when the bound is tight.

Sharding, partitioning, and consistent hashing

Sharding: Key-Range Partitions vs hash

Sharding splits one dataset across nodes by a shard key. Range sharding (the catalog's Key-Range Partitions) keeps neighboring keys together, so range scans stay on one shard. Hash sharding spreads keys evenly, so writes don't pile up on one node.

Example: shard orders by created_at ranges, and today's shard takes every write. Shard by hash(customer_id), and writes spread out, but "all orders from last hour" hits every shard.

Use it when: one node can't hold the data or take the write load, after read replicas and caching.

Note: the shard key decides which queries stay on one shard, and it's expensive to change later. Key-range partitions also need splitting when one range grows hot. Read more in the sharding section of 6 Steps to Scale Your Application in the Cloud.

Fixed Partitions

Create many more partitions than nodes up front, and map keys to partitions with hash(key) % partitions. That number never changes. When a node joins, the cluster moves whole partitions to it, and no key changes its partition.

Example: Hazelcast splits data into 271 partitions by default. A Kafka topic has a fixed partition count, and the cluster reassigns partitions between brokers.

Use it when: nodes join and leave, and you want rebalancing to move whole partitions instead of single keys.

Note: pick the count with growth in mind, because changing it means rehashing every key. Kafka can add partitions, but that changes which partition a key goes to.

Consistent hashing

Plain hash(key) % N moves almost every key when N changes. Consistent hashing puts nodes and keys on a ring, and each key goes to the next node clockwise. Adding a node moves only about 1/N of the keys: the ones between the new node and its neighbor.

Use it when: nodes join and leave often, as in caches and leaderless stores.

Note: with few nodes, the ring is uneven. Virtual nodes (many points per server) smooth the load. A single hot key still lands on one node.

Request handling between nodes

Inside a cluster, nodes exchange many small messages, and network round trips dominate the cost. These patterns reduce the number of round trips and preserve request order.

Request Batch

Collect several requests and send them as one message. One network call and one disk flush then cover many requests.

Example: the Kafka producer groups records for the same partition into a batch, up to batch.size bytes, and waits up to linger.ms to fill it.

Use it when: many small requests go to the same node, and throughput matters more than the latency of a single request.

Note: waiting to fill a batch adds latency, so cap the wait as well as the size.

Request Pipeline

Send the next request on a connection without waiting for the previous response. The connection keeps carrying requests while the receiver still works on earlier ones.

Example: the Kafka producer sends up to max.in.flight.requests.per.connection requests per connection before it waits, 5 by default.

Use it when: the round trip, not the server, limits throughput.

Note: if a request fails and the client retries it, the retry can land after a later request. Kafka's idempotent producer keeps the order with up to 5 requests in flight because the broker checks sequence numbers.

Single-Socket Channel

Send all requests from one node to another over a single TCP connection. TCP delivers bytes in order, so the requests arrive in the order they were sent.

Example: In a Leader and Followers cluster, each follower gets the leader's log entries over one connection, so entry 42 never arrives before entry 41.

Use it when: the receiver must process requests in the order the sender issued them.

Note: one connection limits throughput, so pair it with a pipeline and batching, and with a Singular Update Queue on the receiving side.

Request Waiting List

A node can't answer some requests until something else happens, such as replication to other nodes. The node parks each such request in a waiting list with a condition and a timeout, and answers it when the condition becomes true.

Example: a Kafka producer request with acks=all waits on the leader until all in-sync replicas have the records, then the broker sends the response.

Use it when: a response depends on events from other nodes, and the node shouldn't block a thread per request.

Note: every parked request needs a timeout. Otherwise, a lost follower response keeps requests and memory waiting forever.

Retries, timeouts, idempotency, and circuit breakers

Timeouts and retries with exponential backoff and jitter

Give every remote call a timeout, then retry only the failures that can succeed on a second try (timeouts, 503, 429), never a 400. Wait longer after each attempt (exponential backoff), cap the wait, and randomize it (jitter), so clients don't retry in sync.

In .NET, the standard handler from Microsoft.Extensions.Http.Resilience already retries three times with exponential backoff and jitter. It also retries every HTTP method by default, so turn retries off for writes that aren't idempotent:

builder.Services.AddHttpClient("inventory")
    .AddStandardResilienceHandler(o =>
    {
        o.AttemptTimeout.Timeout = TimeSpan.FromSeconds(2);
        o.TotalRequestTimeout.Timeout = TimeSpan.FromSeconds(10);
        o.Retry.DisableForUnsafeHttpMethods(); // no retries for POST, PUT, PATCH, DELETE
    });

Retry in one layer only. The AWS Builders' Library describes a five-deep stack of service calls with three tries at each layer: "the load on the database will increase 243x".

Note: retries add load when the dependency is already overloaded. AWS limits them with a token bucket, so retries stop when the bucket runs dry.

Idempotent Receiver and idempotency keys

An operation is idempotent when running it twice has the same effect as running it once. An idempotent receiver identifies each request uniquely, remembers the ones it already handled, and returns the stored response for a duplicate instead of running it again. Between services, the client sends that unique ID as an idempotency key.

POST /payments
Idempotency-Key: 7f3c9a2e-41b8-4d0c-9a51-2b6f0e8d1c44
{"amount": 4900, "currency": "usd"}

→ retry with the same key returns the stored 201, no second charge

As an example, Stripe stores the status code and body of the first request with a key, and prunes keys once they're at least 24 hours old. Inside a cluster, the same pattern uses a client ID plus a request number, as Kafka's idempotent producer does with producer IDs and sequence numbers.

Use it when: a client may retry a write that creates something or moves money.

Note: you need a store for the keys, and the key and the side effect must commit together, or a crash between them breaks the guarantee. The store also needs an expiry, which caps how late a retry can arrive.

Circuit breaker and bulkhead

When a dependency's failure rate crosses a threshold, a circuit breaker stops calling it. The breaker stays open for a break period, lets a few trial calls through (half-open), and closes again if they succeed. A bulkhead caps concurrent calls to each dependency, so one slow service can't tie up all your threads or connections.

Use it when: a dependency can be slow or down for minutes, and failing fast is better than waiting.

Note: a breaker needs a fallback (a cached value, a default, a clear error), and its thresholds need tuning per dependency. Read more in the Circuit Breaker article.

Backpressure and rate limiting

Backpressure means a busy component tells its callers to slow down instead of queueing work without limit. Rate limiting is one way to apply it: a token bucket refills at a fixed rate, and each request takes a token or gets a 429.

Use it when: callers can send faster than you can process, as with public APIs, consumers, or batch jobs.

Note: the rejected work has to go somewhere: a later retry, a drop, or a queue with its own limit.

Message delivery semantics and distributed transactions

At-most-once, at-least-once, exactly-once

A message broker gives one of three delivery guarantees:

Note: Kafka gets closest to exactly-once. Since Kafka 3.0, the producer is idempotent by default when acks=all, and transactions give exactly-once for read-process-write inside Kafka. An email, an HTTP call, or a write to another database is outside that boundary, so the consumer still has to dedupe.

Outbox and Inbox patterns

A service that writes to its database and then publishes an event can crash between the two steps. The outbox pattern writes the event to an outbox table in the same database transaction as the business change. A relay then publishes it, either by polling the table or by reading the database log (change data capture). On the receiving side, the inbox pattern stores each processed message ID, so the consumer skips a redelivered message.

Use it when: a state change and its event must not get out of sync, which is almost every event-driven microservices setup.

Note: the relay delivers at least once, so the consumer still needs the inbox or another idempotency check. Read more in Transactional Inbox and Outbox Patterns and Change Data Capture (CDC).

Two-Phase Commit vs saga

Two-phase commit (2PC) updates resources on several nodes as one atomic operation. The coordinator asks every participant to prepare, then tells them all to commit. If the coordinator dies after the prepare step, the participants hold their locks and wait. A saga splits the operation into local transactions, one per service. If a step fails, the saga runs compensating transactions for the steps that already committed.

Example: an order saga reserves stock, charges the card, and books delivery. If delivery fails, it refunds the card and releases the stock. Inside one database cluster, 2PC is common: Spanner commits transactions across partitions with 2PC, and every participant is a Paxos group, so a coordinator failure doesn't block forever.

Use it when: one business operation spans services that each own their database (saga), or partitions of one database that supports distributed transactions (2PC).

Note: a saga has no isolation. Other requests see the intermediate state (stock reserved, card not charged yet), and each compensation is business logic you design and test. Read about orchestration vs choreography in Microservices and Distributed Systems Interview Questions, and about locks held across services in Deadlocks in Distributed Systems.

Conclusion

Design for the message that never arrives. Timeouts, idempotency keys, and deduplication come before any consistency choice. Without them, every retry is a guess.

Choose consistency per operation. A payment, a profile edit, and a like counter need different guarantees, and paying for the strongest one everywhere costs latency you don't need.

Borrow the hard parts. Write-ahead logs, consensus, consistent cores, and idempotent producers ship with your database, orchestrator, and broker. Use them before you write your own.

I left Byzantine fault tolerance and CRDTs out of this post. Each deserves its own post.

What I suggest you read related to the topic

• CAP Theorem
• Consistency Models for Distributed Systems
• Raft Consensus Algorithm Explained
• Transactional Inbox and Outbox Patterns
• Microservices and Distributed Systems Interview Questions