A regional network partition rarely begins as a clean, obvious outage. It starts as elevated tail latency, a few expired leases, retries that look harmless in dashboards, and replicas that can still read but no longer agree on who may write. That is where distributed systems patterns earn their keep: not by making failure disappear, but by ensuring the system fails with explicit ownership, bounded damage, and a path back to correctness.
The useful unit of design is not a catalog of named patterns. It is a failure contract. Before choosing a quorum protocol, an outbox, or a circuit breaker, determine what may be stale, what may be duplicated, what must never occur twice, and who has authority when the network lies. A service can tolerate stale product recommendations. It cannot casually tolerate a duplicate wire transfer or two control planes simultaneously mutating the same customer resource.
Distributed Systems Patterns Start With Write Authority
The most expensive distributed failures are usually conflicting writes, not failed reads. A read can be retried, served from a replica, or made temporarily unavailable. Once two actors successfully apply incompatible state transitions, reconciliation becomes a business process rather than an infrastructure task.
Single-writer partitioning is the default
Assign each logical key to one active writer at a time. The key may be a tenant, account, device, document, or shard range. The implementation can be a Raft-backed partition leader, a database primary, or a lease holder, but the invariant matters more than the mechanism: one authority decides the next ordered mutation for that key.
This is why partitioning strategy is a correctness decision, not just a scaling decision. A tenant-keyed partition may make isolation and rate limiting straightforward, while an account-keyed partition may be necessary to preserve balance invariants. Choosing a key that scatters related writes across partitions creates a coordination problem that no amount of cache capacity will solve.
Leases are attractive because they appear lightweight. They are also frequently misunderstood. Lease expiry proves only that a local clock and a timer say authority should end. It does not stop a paused process from resuming after a long garbage-collection pause and issuing writes against an old assumption. Pair leases with fencing tokens: every newly granted owner receives a monotonically increasing token, and the storage layer rejects mutations carrying a token older than the last accepted token. The resource itself must enforce fencing. Checking it only in the caller is not sufficient.
Use quorum where correctness requires a shared decision
Consensus systems such as Raft are appropriate when a small set of nodes must agree on ordered state transitions: membership, shard placement, feature-flag control planes, workflow ownership, or metadata that decides where writes belong. A quorum commit prevents a minority partition from independently establishing a competing history.
That guarantee has a price. Consensus increases write latency, reduces availability during quorum loss, and turns poorly bounded log growth into an operational hazard. Do not place every request path behind a globally replicated consensus group because “strong consistency” sounds safe. Keep the strongly coordinated state small, then use it to govern larger, partitioned data planes.
A useful split is control-plane consensus and data-plane locality. The control plane decides assignment, versions, and authority. The data plane processes traffic close to the user or workload, with explicit replication and recovery behavior. This avoids turning routine high-volume writes into global coordination events.
Make Retries Safe Before You Make Them Fast
At-least-once delivery is the normal operating condition once requests cross process, queue, or region boundaries. Timeouts do not reveal whether the remote system failed before applying work, failed after applying it, or merely failed to return the response. Retrying without a deduplication design converts uncertainty into duplicate side effects.
Idempotency keys solve this when attached to the business operation rather than the HTTP request alone. A payment authorization, provisioning request, or email send should have a durable operation identity. Store the result keyed by that identity alongside the state transition, then return the recorded outcome on duplicates. Hashes of request bodies can help detect key reuse with incompatible parameters, but they are not a replacement for a client-generated operation ID.
The transactional outbox closes a second failure window. Without it, an application can commit a database change and crash before publishing its event, or publish an event and crash before committing the change. Write the state change and an outbox record in one local transaction. A relay then publishes the outbox record, accepting that it may publish more than once. Consumers must still deduplicate.
Exactly-once is therefore rarely an end-to-end property. It is usually a composition of at-least-once transport, durable idempotency, and an atomic boundary around local state. Calling this “exactly once” without naming the boundary invites dangerous assumptions during incident response.
Prefer compensations to distributed transactions across services
A multi-service business workflow needs explicit states: reserved, confirmed, expired, compensated, and manually reviewed. A saga coordinates these transitions through events or an orchestrator, with compensating actions for work that has already committed.
Compensation is not rollback. Reversing a shipment request may mean intercepting it before fulfillment, initiating a return after shipment, or issuing a credit if the physical action cannot be reversed. That distinction should show up in the domain model and the operator tooling. If a system has no meaningful compensation for a step, it may need a stronger commit boundary before that irreversible action occurs.
Contain Overload Instead of Amplifying It
Distributed systems often fail by self-inflicted load multiplication. A downstream dependency slows down, callers hold connections longer, worker pools fill, retry timers fire together, and the original partial outage becomes thread exhaustion across the fleet.
Backpressure is the first defense. Bound queues by count and bytes, reject or defer work when budgets are exhausted, and propagate cancellation when callers no longer need results. An unbounded queue is not resilience. It is delayed failure with worse observability and larger recovery time.
Circuit breakers are useful when a dependency is predictably unhealthy, but they should not become opaque policy engines. Open the circuit based on recent errors and latency, allow limited probes through a half-open state, and expose the reason in metrics. Pair the breaker with a fallback that preserves semantics. Serving a cached entitlement may be acceptable; inventing an authorization decision is not.
Retries need a budget. Use exponential backoff with jitter, cap attempts, and make retries consume a fraction of normal request volume. Retry only transient failures and only when the operation is idempotent or protected by a deduplication key. For overloaded services, a fast, explicit rejection is often healthier than another attempt that occupies CPU, sockets, and queue capacity.
Bulkheads complement this approach. Separate connection pools, worker pools, and concurrency limits by dependency or traffic class so a slow analytics export cannot starve interactive writes. At high scale, this separation should extend to noisy tenants and expensive query shapes, not merely service names.
Reconcile for the Failures Your Protocol Cannot Prevent
No protocol eliminates every mismatch. Events can expire from retention windows, a bad deployment can corrupt derived projections, and an operator can repair one side of a system incorrectly at 3 a.m. Mature systems include reconciliation as a product capability, not an emergency script.
Maintain immutable or append-only source records where practical, and treat indexes, caches, search documents, and materialized views as rebuildable derivatives. For replicated keyspaces, anti-entropy mechanisms such as version vectors, Merkle-tree comparisons, or periodic range scans identify divergence without copying the entire dataset. The right mechanism depends on cardinality, mutation rate, and the cost of reading canonical state.
Design repair jobs to be throttled, resumable, and observable. A repair process that saturates the primary database during an incident is another outage generator. Record what was compared, what changed, which version won, and whether a human needs to review the result. Silent repair is convenient until it silently resolves a financial or security conflict in the wrong direction.
Observability should follow the same causal path as the workload. Propagate trace context across queues and asynchronous workers, attach partition and fencing-token metadata to structured logs, and measure replication lag, outbox age, retry volume, queue depth, and rejected work. Aggregated success rates are inadequate when one hot partition is timing out while the fleet average remains green.
The best distributed design is often the one that declines to distribute a decision. Keep invariants local where possible, coordinate only the state that truly requires agreement, and make every ambiguous boundary visible to the people who will operate it. When the next partition arrives, your engineers should be deciding which contract applies, not guessing which copy of reality is allowed to win.
Leave a Reply