A checkout API that handles 2,000 requests per second in steady state can still fail at 200 requests per second when one dependency begins timing out. Threads pile up behind slow calls, connection pools saturate, retries amplify the load, and a system that looked healthy in a capacity spreadsheet becomes unavailable. That is the real domain of system design: not drawing services and arrows, but deciding what happens when assumptions fail.
The useful question is rarely, “Which database should we use?” It is: which property must survive a regional impairment, a queue backlog, a bad deploy, a tenant with pathological behavior, or a 10x traffic burst? The answer determines the architecture. A design that protects payment correctness differs materially from one that optimizes feed freshness, model-serving throughput, or low-cost analytics.
Start System Design With Failure and Demand
Architecture discussions often begin with a functional decomposition: API gateway, service layer, cache, database, queue. That decomposition is familiar, but it hides the decisions that create production outcomes. Start with the workload instead.
Specify the request mix, not just aggregate RPS. A service receiving one million mostly-cacheable reads per second has a different bottleneck profile from one processing 20,000 write-heavy requests with cross-entity transactions. Include payload distributions, fan-out, hot-key risk, tail-latency objectives, tenant skew, and burst duration. Average values are convenient and routinely misleading.
Then state failure expectations in operational terms. If the primary datastore is unreachable for five minutes, may users read stale data? Can they submit writes for later reconciliation? Must a request fail closed? If an event consumer falls six hours behind, is replay safe, and does replay create duplicate external side effects? These are design inputs, not post-launch SRE concerns.
A useful constraint set contains four dimensions:
- Correctness: what cannot be lost, duplicated, reordered, or exposed.
- Availability: which operations must continue during partial dependency failure.
- Latency: the p50, p95, and p99 behavior that matters to users and upstream callers.
- Economics: the cost ceiling at ordinary load and during abnormal conditions.
Security belongs inside each dimension. A zero-trust egress policy, for example, affects availability when a required destination is blocked, latency when inspection adds work, and cost when traffic is routed through centralized controls. Treating security as a separate box on the diagram produces fragile systems and surprise operational paths.
Model the Critical Path, Not the Service Inventory
A service inventory tells you ownership. A critical-path model tells you whether a request can finish.
For every high-value operation, trace synchronous calls, asynchronous handoffs, state mutations, cache reads, authorization checks, and external dependencies. Assign each edge a timeout, concurrency bound, retry policy, and fallback behavior. If those values do not exist, the system has an implicit policy, usually “wait until resources are exhausted.”
Consider a request that calls an identity provider, inventory service, pricing engine, and payment processor. A sequential path compounds tail latency. A fan-out path can reduce median latency but increases the chance that one slow branch dominates the whole request. Parallelism is not free: it consumes connections, CPU scheduling capacity, and downstream concurrency precisely when a dependency is degraded.
Budget time deliberately. With a 500 ms end-to-end deadline, an API gateway cannot spend 450 ms waiting for one backend and leave every downstream component to improvise. Propagate deadlines, use cancellation where the runtime supports it, and reserve time for serialization, response construction, and client delivery. A timeout without cancellation often merely moves the work off the caller’s dashboard.
The same discipline applies to concurrency. Little’s Law is practical architecture, not interview trivia: concurrent work grows with arrival rate and time spent in the system. When latency rises, unconstrained concurrency rises with it, consuming memory, file descriptors, goroutines, threads, and database sessions. Bounded queues and admission control are therefore availability mechanisms. They decide which work is rejected early instead of allowing all work to fail late.
Consistency Is a Product Contract
Distributed storage is frequently discussed as a choice between strong consistency and eventual consistency. Production systems need more precise language.
Ask what a user can observe after an acknowledged write. Can they immediately read their own change? Can two users make conflicting reservations? Is a stale value merely inconvenient, or can it trigger a financial loss, access-control violation, or irreversible workflow? The required guarantee may apply only to a narrow subset of state.
A global inventory counter may tolerate delayed aggregation, while the reservation record for the final unit cannot. A social feed can accept asynchronous fan-out and ranking lag; a ledger cannot accept an ambiguous commit outcome. Put coordination where the invariant lives rather than imposing synchronous replication on unrelated data paths.
When a workflow spans services, do not pretend that a chain of local database commits is atomic. Use an explicit state machine, durable intent records, idempotency keys, and compensating actions where compensation is meaningful. An outbox pattern can make database state changes and event publication recoverable, but it does not make consumers exactly once. Consumers still need deduplication or idempotent side effects because delivery can be repeated after a crash at the worst possible moment.
Consensus systems deserve the same clarity. A Raft quorum commit protects a replicated log from acknowledged data loss under its fault assumptions. It does not eliminate application-level races, bad schema migrations, or a client retrying a timed-out operation. The system design must distinguish a durable commit from a completed business action.
Design Degradation Before You Need It
Most outages are not total failures. They are partial, slow, uneven, and contagious. One availability zone has packet loss. One shard is hot. A third-party API returns 429s. GPU workers run out of KV-cache capacity and request latency climbs until the gateway retries traffic into a saturated fleet.
The architecture should have a defined degraded mode for these conditions. A product catalog may serve a slightly stale cache. A search endpoint may return fewer results rather than waiting for an expensive ranking stage. An LLM application may reduce maximum context length, route low-priority requests to a smaller model, or reject streaming work before it exhausts accelerator memory. These are product decisions expressed through infrastructure controls.
Circuit breakers help only when they are paired with sensible fallbacks and observability. A breaker that opens after failures but redirects every request to another saturated dependency simply relocates the blast radius. Bulkheads are often more valuable: isolate pools by tenant, endpoint class, priority, or dependency so a noisy batch job cannot consume the resources required for interactive traffic.
Retries require similar skepticism. Retry only operations that are safe to repeat, only after errors likely to be transient, and only within a bounded total deadline. Add jitter. Cap attempts. Prefer retry budgets that limit retry volume as a fraction of successful traffic. Otherwise, clients turn a modest degradation into a self-sustaining overload event.
Make Cost a First-Class Architectural Signal
Cloud cost failures are commonly capacity-design failures with a delayed invoice. Autoscaling can preserve availability while multiplying spend during retry storms, unbounded cardinality, inefficient queries, or a single tenant’s abusive workload. The absence of a budget guardrail is an architectural choice.
Measure unit economics at the request boundary: cost per successful transaction, per gigabyte processed, per inference token, or per active tenant. Then connect that measure to implementation details. A cache miss rate may translate directly into database read replicas. A larger model context can increase prefill compute and KV-cache pressure. Cross-region replication can make a low-latency design materially more expensive than the business case supports.
This does not mean optimizing every component for cost. It means knowing where cost rises nonlinearly and placing controls there. Rate limits, payload caps, partition-level quotas, lifecycle policies, and backpressure often produce larger savings than micro-optimizing a serialization library.
Validate the System Design Through Operations
A design review is incomplete until it describes how operators detect and recover from its failures. Metrics should expose queue age, saturation, timeout causes, retry volume, replication lag, cache churn, and dependency-specific latency, not just aggregate request success. High availability without diagnosability creates long incidents.
Test the behavior that diagrams assume. Inject latency into a dependency and verify that deadline propagation stops useless work. Kill consumers and validate backlog growth, replay time, and duplicate handling. Remove a region from service and observe whether DNS, data placement, credentials, and capacity policies behave as designed. Load tests should include hot keys, skewed tenants, and slow downstream responses, because uniform synthetic traffic is unusually polite.
The strongest designs are not those with the most components or the cleanest diagrams. They are the ones whose invariants, overload behavior, recovery paths, and operating costs remain understandable at 3 a.m. when the traffic is real and the assumptions are not.
Leave a Reply