Article
The System Design Checklist
- Authors
- Author
- Ram Simran G
- twitter @rgarimella0124
Most system design content is written from a single point of view — usually “how would I answer this in an interview.” That’s useful, but it misses how these questions actually get asked in the real world: by a systems engineer debugging a 3am page, by a DevOps engineer trying to keep a deploy from taking down production, and by an architect trying to make a decision that a dozen teams will be stuck with for the next five years.
This post walks through all 43 questions from the original checklist, answered from all three of those perspectives at once — with the trade-offs, the “when do you actually use this” guidance, and a real industry example for each. A few questions get a small diagram where a picture genuinely clarifies things faster than more words would.
Part 1: Core Distributed Systems & Scalability
1. How do I decide when to split a monolith into services?
Answer: Don’t split along technical lines (e.g., “the database layer”) — split along team and change-frequency boundaries. A DevOps lens adds a second test: split when a single deployment unit is forcing unrelated teams to coordinate releases, or when one component’s failure mode is taking down unrelated functionality. An architect’s test: split when the data ownership genuinely diverges, not just the code.
Trade-off: Services buy independent deployability and fault isolation, but cost you network calls, distributed debugging, and operational overhead (more things to monitor, deploy, and secure) in exchange.
When to use what: Stay monolithic as long as one team can safely own the whole codebase and deploys aren’t blocking each other. Split when deployment coordination pain or blast-radius risk consistently outweighs the operational cost of a new service.
Industry example: Amazon’s well-known shift from a monolith to service-oriented architecture in the early 2000s was driven explicitly by team scaling problems, not just technical ones — Conway’s Law in action.
2. How does a request flow end-to-end through a production system?
Answer: From an SRE/DevOps perspective, this is the single most important mental model to hold, because every latency and reliability problem lives somewhere along this path: client → DNS/load balancer → API gateway (auth, rate limiting) → service mesh/network → application service → cache → database/downstream services → response, with logging, tracing, and metrics collection happening at every hop.
Trade-off: The more hops and layers you add for security, observability, and resilience, the more latency and potential failure points you introduce — every layer is a trade of reliability/control against speed and simplicity.
When to use what: Simple systems can skip API gateways and service meshes; add them once you have multiple services, multiple teams, and a genuine need for centralized auth, rate limiting, or traffic control.
Industry example: Companies like Netflix and Uber openly document their full request-path diagrams in engineering blogs specifically because understanding this flow is treated as a prerequisite for debugging incidents.
Diagram:
flowchart TD
C[Client] --> D[DNS]
D --> LB[Load Balancer]
LB --> AG["API Gateway<br/>(auth / rate limit)"]
AG --> SM[Service Mesh]
SM --> AS[App Service]
AS --> CA{Cache Hit?}
CA -->|Yes| R[Response]
CA -->|No| DB["Database / Downstream Services"]
DB --> R3. How do I design a service that can handle 10x more traffic with minimal changes?
Answer: Design statelessly from day one — push session state into a shared store (Redis, a database) rather than in-memory, so any instance can serve any request. Then scaling is “add more instances behind a load balancer,” not “rearchitect.”
Trade-off: Statelessness adds a network hop to read shared state, trading a small latency cost for near-linear horizontal scalability.
When to use what: If traffic growth is predictable and gradual, vertical scaling (bigger machines) can buy time cheaply. If growth is spiky or you need to survive traffic 10x above baseline without notice, horizontal scaling behind auto-scaling groups is the only approach that holds up.
Industry example: Most cloud-native SaaS companies (Shopify, Stripe) design every service to be stateless by policy specifically so Black Friday-style spikes can be absorbed by auto-scaling rather than emergency re-architecture.
4. What are the real trade-offs between vertical and horizontal scaling?
Answer: Vertical scaling (bigger machine) is operationally simple — no distributed systems complexity — but has a hard ceiling and a single point of failure. Horizontal scaling (more machines) removes the ceiling and improves fault tolerance, but forces you to solve state management, load balancing, and data consistency problems you didn’t have before.
Trade-off: Vertical = simplicity now, hard ceiling and downtime risk later. Horizontal = complexity now, effectively unlimited headroom and better resilience later.
When to use what: Vertical scaling is fine for early-stage products, databases that are hard to shard, or workloads with a genuine hard ceiling on data size. Horizontal scaling is necessary once you need to survive a single machine’s failure or exceed what any single machine can handle.
Industry example: Traditional relational databases (before cloud-native options like Aurora/Spanner) were often scaled vertically for years because horizontal sharding of a relational store is genuinely hard — this is a classic case where the “easy” answer persisted because the alternative was expensive.
5. How do I choose between synchronous and asynchronous communication?
Answer: Use synchronous (direct request/response) when the caller genuinely needs an immediate answer to proceed. Use asynchronous (queues, events) when the caller just needs to know the work will eventually happen, and doesn’t need to block waiting for it.
Trade-off: Synchronous is simpler to reason about and debug (a stack trace tells the whole story) but couples the caller’s availability to the callee’s availability — if the downstream service is slow or down, you’re stuck waiting or failing. Async decouples availability but adds real complexity: you now need to think about ordering, retries, and eventual consistency.
When to use what: User-facing requests needing an immediate result (checking out a cart) → synchronous. Background work the user doesn’t need to wait on (sending a confirmation email, updating an analytics pipeline) → asynchronous.
Industry example: Payment authorization is typically synchronous (the user needs to know now), while sending the receipt email is asynchronous — a pattern used across virtually every e-commerce platform.
6. When should I use a queue vs direct RPC?
Answer: Direct RPC (gRPC, REST) is right when you need a response before continuing, and the caller can tolerate the callee being temporarily unavailable causing a visible failure. A queue is right when you want to decouple producer and consumer lifecycles entirely — the producer can keep working even if the consumer is down or slow.
Trade-off: RPC gives you simplicity and low latency but zero buffering — if the downstream service can’t keep up, requests fail or pile up at the caller. Queues give you buffering and resilience to consumer downtime, at the cost of added infrastructure (the queue itself becomes a system you must operate and monitor) and eventual, not immediate, processing.
When to use what: Use RPC for tightly coupled, latency-sensitive operations. Use a queue when producers and consumers scale independently, or when you need to smooth out traffic spikes (the queue absorbs the burst; consumers drain it at their own pace).
Industry example: Uber uses queues extensively to decouple trip-event ingestion from the many downstream systems (billing, fraud detection, analytics) that consume those events at different rates.
7. How do I design for backpressure in a high-traffic system?
Answer: Backpressure means giving a struggling downstream component a way to signal “slow down” rather than silently accepting work it can’t keep up with and falling over. In practice: bounded queues (reject or shed load once full, rather than growing unboundedly), rate limiting at the producer, and load-shedding policies that explicitly decide which requests to drop when overloaded.
Trade-off: Implementing backpressure means deliberately failing some requests during overload — an uncomfortable trade, but the alternative (unbounded queues, cascading failure) is worse: you protect the whole system’s availability at the cost of some individual requests.
When to use what: Any system where a slow consumer could be overwhelmed by a fast producer needs backpressure — this is especially critical in queue-based and streaming architectures.
Industry example: Kafka consumers implementing backpressure by controlling their own poll/fetch rate is a textbook example — the broker doesn’t force-feed consumers faster than they can process.
8. How do I protect downstream services from traffic spikes?
Answer: Layer multiple defenses: rate limiting at the edge/gateway, queues to absorb bursts, circuit breakers to stop calling a service that’s already failing, and auto-scaling to add capacity where the spike is sustained rather than transient.
Trade-off: Every protective layer adds latency and operational surface area; too little protection risks cascading failure, too much (overly aggressive rate limits) risks rejecting legitimate traffic.
When to use what: Rate limiting for predictable, per-client abuse protection. Queues for bursty-but-eventually-processable traffic. Circuit breakers specifically for protecting a service that’s already showing signs of failure, to stop making things worse.
Industry example: Stripe’s public API rate limiting is a well-documented example of protecting downstream payment infrastructure from being overwhelmed by client-side traffic spikes or bugs.
9. How do I design APIs so they can evolve without breaking clients?
Answer: Version explicitly (URL or header-based), make new fields additive and optional rather than modifying existing field meanings, and never remove or repurpose a field without a formal deprecation window. From an architect’s perspective, this is a contract-design problem as much as a technical one — you’re making a promise to every client that calls your API.
Trade-off: Strict backward compatibility slows down how fast you can “clean up” an API’s design, but breaking clients erodes trust and creates emergency firefighting whenever you ship a change.
When to use what: Internal APIs between services you control fully can tolerate faster iteration (coordinated deploys). Public or cross-team APIs need strict versioning and deprecation policies because you can’t control when every client upgrades.
Industry example: Stripe is widely cited for its API versioning discipline — accounts are pinned to the API version active when they signed up, and Stripe maintains backward compatibility for years.
10. How do I handle idempotency for write APIs?
Answer: Require clients to send a unique idempotency key with write requests; the server stores the result of the first request under that key and returns the same result for any retry with the same key, instead of re-executing the write.
Trade-off: Idempotency keys add storage overhead (you need to track recent keys) and a bit of client-side discipline, but they’re what makes retries — which are unavoidable in distributed systems — safe rather than dangerous.
When to use what: Any write operation that a client might retry after a timeout (payments, order creation) needs idempotency; purely read operations don’t, since re-running a read is naturally safe.
Industry example: Stripe’s and PayPal’s payment APIs both require idempotency keys on charge creation specifically to prevent double-charging a customer when a network timeout causes a client to retry.
11. How do I model data differently for OLTP vs OLAP workloads?
Answer: OLTP (transactional) workloads need normalized schemas optimized for fast, small, frequent reads and writes of individual records. OLAP (analytical) workloads need denormalized, often columnar schemas optimized for scanning and aggregating huge volumes of data across many records at once.
Trade-off: A schema optimized for OLTP (normalized, row-based) performs poorly for OLAP-style aggregate queries, and vice versa — trying to serve both from the same schema and store usually means compromising both.
When to use what: Use your normalized transactional database for the application’s live read/write path. Pipe data into a separate OLAP store (a data warehouse, columnar store) for reporting and analytics, so heavy analytical queries never compete with production traffic for resources.
Industry example: Most companies run PostgreSQL/MySQL for OLTP and pipe data into Snowflake, BigQuery, or Redshift for OLAP — a near-universal pattern in modern data architecture.
12. How do I decide between SQL, NoSQL, and search engines?
Answer: SQL when your data is relational, you need strong consistency and transactions, and your query patterns are varied and not fully known upfront. NoSQL (key-value, document, wide-column) when you have a known, simple access pattern at very large scale and can trade some consistency/flexibility for speed and horizontal scalability. Search engines (Elasticsearch, OpenSearch) when your primary need is full-text search or complex faceted filtering, not transactional correctness.
Trade-off: SQL’s flexibility and consistency come at the cost of harder horizontal scaling. NoSQL’s scalability comes at the cost of flexible querying and often strong consistency. Search engines are excellent at search but are not your system of record — they should be a derived index, not the source of truth.
When to use what: Many production systems genuinely use all three side by side — SQL for the core transactional data, NoSQL for a specific high-scale access pattern (like session storage), and a search engine synced from the primary store for search/filter UX.
Industry example: Airbnb runs MySQL as its system of record but replicates data into Elasticsearch specifically to power listing search, rather than trying to serve complex search queries from MySQL directly.
13. How do I shard data — by user, tenant, region, or something else?
Answer: Shard by whatever dimension your access patterns query by most frequently and whatever dimension keeps related data co-located — sharding by user ID works well when nearly all queries are scoped to a single user; sharding by tenant works for B2B SaaS where each customer’s data is naturally isolated; sharding by region works when data residency or latency requirements are geography-driven.
Trade-off: A well-chosen shard key keeps most queries within a single shard (fast, simple). A poorly chosen one forces frequent cross-shard queries or joins, which are slow and complex to implement correctly.
When to use what: B2B SaaS with clear customer boundaries → shard by tenant. Consumer apps with per-user data → shard by user ID (often hashed for even distribution). Regulated industries with data residency laws → shard by region regardless of other considerations.
Industry example: Slack shards data by workspace (tenant) specifically because nearly every query a user makes is scoped to their own workspace, making cross-shard queries rare.
14. What does a good partition key look like in a distributed store?
Answer: A good partition key has high cardinality (many distinct values, to spread load evenly) and aligns with your dominant access pattern (so most reads/writes hit a single partition rather than fanning out across many).
Trade-off: A low-cardinality key (like “country” for a global user base skewed toward a few countries) creates hot partitions that bottleneck the whole system, even if the rest of your infrastructure is perfectly scaled. A high-cardinality key that doesn’t match your access pattern forces expensive cross-partition queries.
When to use what: Composite keys (e.g., tenant ID + entity ID) are common precisely because they balance even distribution with query locality — you get both good spread and single-partition reads for the common case.
Industry example: DynamoDB’s documentation explicitly warns against low-cardinality partition keys, using the classic example of partitioning a global leaderboard by “game ID” when a handful of extremely popular games would create massive hot partitions.
Part 2: Data Modeling, Storage & Indexing
15. How do I avoid hot partitions and skew?
Answer: Choose a high-cardinality, evenly-distributed key (see Q14), and where a natural key is inherently skewed (e.g., a viral user or trending item), add a random or hashed suffix to spread writes for that single logical entity across multiple physical partitions, merging results at read time.
Trade-off: Salting/sharding a hot key adds read-side complexity (you now need to query multiple physical partitions and merge results) in exchange for eliminating a write bottleneck that would otherwise cap your entire system’s throughput.
When to use what: Standard high-cardinality keys need no special handling. Known “celebrity” entities (a viral social post, a popular product on sale day) need explicit hot-key mitigation before they become a production incident.
Industry example: Instagram’s engineering team has written publicly about handling hot partitions for viral posts by distributing likes/comments counters across multiple shards and aggregating them.
16. How should I index for read-heavy vs write-heavy workloads?
Answer: Read-heavy workloads benefit from more, wider indexes (covering indexes that satisfy a query without touching the base table) since reads dominate and index maintenance cost is amortized. Write-heavy workloads should minimize indexes, since every index adds write overhead — each insert/update must also update every index on that table.
Trade-off: More indexes = faster reads, slower writes, more storage. Fewer indexes = faster writes, potentially much slower reads (full table scans).
When to use what: An analytics/reporting table that’s rarely written but frequently queried → index heavily. A high-throughput event ingestion table → index minimally, and consider a separate read-optimized replica or downstream store for query needs instead.
Industry example: Time-series/event-ingestion systems like those behind ad-tech click tracking typically write with minimal indexing and rely on downstream batch processing (Spark, Flink) for the heavy analytical querying, rather than indexing the raw ingestion table itself.
17. How do I design soft delete, archival, and retention?
Answer: Soft delete (a deleted_at flag rather than physically removing rows) preserves data for recovery, audit, and referential integrity, while application logic filters out soft-deleted rows from normal queries. Archival moves old, rarely-accessed data to cheaper storage (cold storage, a separate archive table) while retention policies define when data is permanently purged, often driven by legal/compliance requirements.
Trade-off: Soft delete adds a filter condition to every query (easy to forget, causing subtle bugs) and lets deleted data silently accumulate, bloating tables and indexes over time — it trades short-term safety for long-term storage and query-performance cost if not paired with an actual archival/purge process.
When to use what: Soft delete for anything a user or support team might need to “undo” or audit. Hard delete (with proper backups) for genuinely ephemeral or non-sensitive data where recovery isn’t a real requirement.
Industry example: Most fintech and healthcare platforms are legally required to implement formal retention policies (e.g., 7-year financial record retention) that combine soft delete, archival to cold storage, and scheduled hard purges once the legal retention window closes.
18. How do I handle multi-tenant data isolation?
Answer: Three common models, in increasing order of isolation (and cost): shared database/shared schema with a tenant_id column and row-level filtering; shared database/separate schema per tenant; and fully separate databases per tenant. From a systems/DevOps perspective, the further right you go, the more operational overhead (backups, migrations, monitoring) multiply per tenant.
Trade-off: Shared schema is cheapest to operate but riskiest — a single missing WHERE tenant_id = ? clause leaks data across tenants. Separate databases per tenant are the most secure and easiest to reason about, but multiply operational cost linearly with tenant count.
When to use what: High-volume, lower-sensitivity SaaS (many small tenants) → shared schema with strict row-level security enforced at the database or ORM layer. Regulated industries or a small number of large enterprise customers demanding strong isolation guarantees → separate databases or even separate infrastructure per tenant.
Industry example: Salesforce famously runs a shared, multi-tenant architecture with extremely rigorous row-level security enforcement, while many enterprise B2B SaaS companies serving finance/healthcare clients offer fully isolated single-tenant deployments as a premium tier.
19. How do I design a migration strategy for a huge table?
Answer: Never run a blocking schema change on a huge live table. Use online schema migration tools (gh-ost, pt-online-schema-change for MySQL) that build a shadow table, backfill it in small batches, and cut over with minimal locking. For application-level changes, use the expand-contract pattern: add the new column/table alongside the old one, dual-write to both, backfill historical data, migrate reads to the new structure, then remove the old one once fully migrated.
Trade-off: Online migration tools and expand-contract patterns take significantly longer and require more coordination than a simple ALTER TABLE, but they avoid taking production down for a multi-hour lock on a huge table.
When to use what: Small tables in low-traffic systems can often tolerate a direct, blocking migration during a maintenance window. Anything at real production scale needs the online/expand-contract approach as a default, not an exception.
Industry example: GitHub open-sourced gh-ost specifically because standard MySQL migrations were becoming operationally untenable on their largest tables — a widely-adopted tool across the industry today.
20. How do I safely backfill data in production?
Answer: Backfill in small, rate-limited batches (not one giant transaction), make the backfill job idempotent so it can safely resume after a failure, run it during low-traffic windows if possible, and monitor database load/replication lag throughout, with the ability to pause if it’s impacting production.
Trade-off: Slow, careful, batched backfills take much longer to complete than a single bulk operation, but a single bulk operation risks locking tables, exhausting connection pools, or blowing out replication lag and taking down production.
When to use what: Small tables or low-traffic systems can tolerate a faster, less cautious backfill. Any backfill touching a high-traffic production table needs batching, rate limiting, and active monitoring as non-negotiable practice.
Industry example: Large-scale backfills at companies like Shopify are typically implemented as background jobs with explicit rate limits and circuit breakers that pause the backfill automatically if database CPU or replication lag crosses a threshold.
21. When do I accept eventual consistency vs strict consistency?
Answer: Accept eventual consistency when the business impact of briefly stale data is low and the availability/performance benefit is high (a “like” count, a follower count, a search index). Require strict consistency when correctness has real consequences — financial balances, inventory counts near zero stock, anything involving money or safety.
Trade-off: Strict consistency typically costs you availability and/or latency (per CAP theorem trade-offs) since you need coordination across nodes before confirming a write. Eventual consistency buys you availability and speed at the cost of a window where different reads can return different answers.
When to use what: Read-heavy, low-stakes data (social features, counters, caches) → eventual consistency is usually the right default. Anything where a wrong answer causes real financial or safety harm → strict consistency, even at a latency/availability cost.
Industry example: Amazon’s DynamoDB explicitly offers both eventually consistent and strongly consistent read options, letting engineers choose per-query based on the specific data’s sensitivity — a direct acknowledgment that this isn’t a system-wide, one-size-fits-all decision.
Part 3: Consistency, Availability & Transactions
22. How do I model consistency levels across services and data stores?
Answer: Explicitly document, per data type, what consistency guarantee it actually needs — not what’s technically achievable, but what the business requires. Different data within the same system often needs different levels: user profile data might tolerate eventual consistency, while a payment ledger needs strict consistency.
Trade-off: Applying one consistency model uniformly across an entire system is simpler to reason about but usually means over-paying for consistency you don’t need in some places, and under-delivering it in others.
When to use what: Map data types to consistency requirements explicitly during design, not as an afterthought — this becomes a genuine architectural document, not just an implementation detail.
Industry example: Amazon’s own internal engineering culture (as described in various re:Invent talks) treats “what consistency does this specific data need” as a standard, mandatory design-review question for any new service.
23. How do I implement distributed transactions without 2PC pain?
Answer: Use the Saga pattern — break the distributed transaction into a sequence of local transactions, each with a corresponding compensating action that undoes it if a later step fails, rather than trying to lock all participants simultaneously as two-phase commit (2PC) does.
Trade-off: Sagas avoid 2PC’s blocking, single-point-of-failure coordinator and poor availability characteristics, but they trade atomicity for complexity — you must design and test compensating transactions for every step, and the system passes through genuinely inconsistent intermediate states that other parts of the system might observe.
When to use what: 2PC can still make sense for a small number of tightly coupled resources within a single, reliable network boundary. Sagas are the standard choice for distributed transactions spanning independent services, especially across service and even organizational boundaries.
Industry example: Uber’s trip-booking flow (reserve driver, charge payment, confirm trip) is a commonly cited real-world Saga implementation, with explicit compensating actions (release driver, refund payment) if any step fails.
Diagram:
flowchart LR
RD[Reserve Driver] --> CP[Charge Payment]
CP --> CT[Confirm Trip]
CP -.fails.-> C1[Compensate:<br/>Release Driver]
CT -.fails.-> C2[Compensate:<br/>Refund Payment]
C2 --> C124. How do I design an outbox/inbox pattern for reliable events?
Answer: The Outbox pattern writes an event to an “outbox” table in the same local database transaction as the business data change, then a separate process (often a change-data-capture tool like Debezium) reliably publishes that event to a message broker — guaranteeing the event is never lost even if the publish step fails, since it’s durably recorded in the same transaction as the actual change.
Trade-off: This adds infrastructure (an outbox table, a relay process) and a small publish delay, but it eliminates the classic dual-write problem — where a service writes to its database and separately publishes an event, and a crash between the two leaves them permanently inconsistent.
When to use what: Any time a service needs to atomically update its own data and reliably notify other services of that change — which is most event-driven microservice architectures.
Industry example: Debezium, an open-source change-data-capture tool built specifically to implement the outbox pattern at scale, is used across companies including Shopify and many fintech platforms to guarantee event delivery without dual-write bugs.
25. How do I handle “read your own writes” in a distributed system?
Answer: Route the user’s subsequent reads to the same node/replica that handled their write (session affinity/sticky routing), or read from the primary for a short window after a write, or track a version/timestamp and require reads to wait until a replica has caught up to that version.
Trade-off: Guaranteeing read-your-own-writes for every user adds routing complexity or forces reads toward the primary (reducing the read-scaling benefit of replicas), so it’s usually applied selectively rather than system-wide.
When to use what: Apply this guarantee specifically where users would notice and be confused by its absence — “I just posted a comment and it’s not showing up” — rather than paying the cost everywhere.
Industry example: Many social platforms use session-based routing so a user’s own posts/comments always appear correctly to them immediately, even while other users might briefly see stale replica data.
26. How do I reconcile divergent data across systems?
Answer: Build explicit reconciliation jobs that periodically compare data across systems (e.g., a source-of-truth database vs. a downstream cache or search index), flag or automatically correct discrepancies, and alert when the divergence rate exceeds an expected baseline (some divergence during normal eventual-consistency windows is expected; a growing or persistent gap signals a real bug).
Trade-off: Reconciliation jobs add operational overhead and computing cost, but without them, silent data drift between systems can go undetected for months, causing slowly compounding correctness issues that are far more expensive to untangle later.
When to use what: Any architecture with derived or duplicated data (a search index synced from a primary store, a cache, a data warehouse) needs a reconciliation process — treat it as a required component, not an optional nice-to-have.
Industry example: Financial systems universally run automated daily/nightly reconciliation between ledgers and downstream reporting systems specifically because undetected drift in financial data has serious regulatory and business consequences.
27. How do I design ID generation that works across regions?
Answer: Avoid centralized auto-increment IDs (a single point of contention and failure across regions); instead use distributed ID generation schemes like Snowflake IDs (timestamp + machine/region ID + sequence number, generated locally with no cross-region coordination) or UUIDs.
Trade-off: Distributed ID schemes (Snowflake-style) give you roughly sortable, unique, collision-free IDs generated locally with no coordination, at the cost of slightly larger ID sizes and needing to provision unique machine/region identifiers. Plain random UUIDs are simpler to generate but lose natural sort order, which can hurt database index performance.
When to use what: Multi-region, high-throughput systems where cross-region coordination for ID generation would add unacceptable latency → Snowflake-style IDs. Simpler, lower-scale systems → UUIDs are often good enough and much simpler to implement.
Industry example: Twitter’s open-sourced Snowflake ID generation scheme is the namesake and reference implementation most companies (including Discord and Instagram) base their own distributed ID systems on.
28. How do I reason about CAP trade-offs for a given product requirement?
Answer: Since network partitions are a fact of distributed systems life (P is not optional), the real question CAP poses is: during a partition, do you favor Consistency (reject requests you can’t guarantee are correct) or Availability (keep serving, accepting the risk of stale/inconsistent data)? Answer this per feature, based on what a wrong or unavailable response actually costs the business.
Trade-off: Favoring consistency means users occasionally see errors or unavailability during network issues, but never incorrect data. Favoring availability means the system stays responsive during issues, but users can occasionally see stale or conflicting data.
When to use what: Payment processing, inventory near zero-stock, anything with real financial/safety consequences → favor consistency. Social feeds, recommendation engines, most read-heavy user-facing features → favor availability, since brief staleness is a minor UX issue, not a business risk.
Industry example: Amazon’s shopping cart famously favors availability (you can almost always add to cart, even during a partition, with conflicts resolved later), while Amazon’s payment processing favors consistency — the same company making explicitly different trade-offs for different features.
flowchart TD
C((Consistency))
A((Availability))
P((Partition Tolerance))
C --- A
A --- P
P --- CPart 4: Reliability, Failures & Recovery
29. How do I design strong ordering guarantees where they really matter?
Answer: Use a single partition/shard per entity whose events must stay strictly ordered (e.g., all events for one user go to the same Kafka partition), since ordering is only guaranteed within a partition, not across an entire topic or system.
Trade-off: Enforcing strict ordering by routing all of one entity’s events to a single partition limits your ability to parallelize processing for that entity — you trade throughput for correctness where correctness genuinely requires order.
When to use what: Financial ledger updates, state machine transitions (order status: placed → shipped → delivered) genuinely need ordering. Independent, unrelated events (different users’ unrelated actions) don’t need any ordering relationship to each other and shouldn’t be forced into one.
Industry example: Kafka’s per-partition ordering guarantee (and no cross-partition guarantee) is the standard mechanism the entire event-streaming industry relies on for this exact trade-off.
30. What are all the ways my system can fail under real traffic?
Answer: A systems/SRE-minded checklist: resource exhaustion (CPU, memory, connection pools, file descriptors), dependency failures (a downstream service or database goes down or slows down), cascading failures (one slow component causes retries/backpressure that take down others), network partitions, deployment-related failures (a bad rollout), and traffic pattern failures (a spike, a thundering herd, an unexpected access pattern).
Trade-off: You cannot design against every possible failure mode with equal investment — this list is genuinely the starting point for a risk-prioritization exercise, not a checklist to fully “solve.”
When to use what: Prioritize defenses based on blast radius and likelihood — cascading failures and dependency failures are usually the highest-priority to defend against first, since they tend to have the largest blast radius relative to how often they occur.
Industry example: Netflix’s Chaos Engineering practice (Chaos Monkey and its successors) exists specifically to proactively surface these failure modes before real traffic does, rather than discovering them during an actual incident.
31. How do I design for graceful degradation instead of hard failure?
Answer: Identify which parts of a request/page/response are essential vs. nice-to-have, and design fallbacks for the nice-to-have parts (show cached/stale recommendations if the live recommendation service is down, rather than failing the entire page load) while protecting the essential path.
Trade-off: Graceful degradation adds real design and code complexity — you now need to build and test fallback behavior for every non-critical dependency — but it converts what would be a full outage into a partial, often barely-noticeable degradation.
When to use what: User-facing systems with many non-critical dependencies (recommendations, “related items,” secondary widgets) are the clearest candidates. Systems with few, all-essential dependencies have less room for this pattern.
Industry example: Amazon’s product pages are a textbook example — if the “customers also bought” recommendation service is down, the page still loads and sells the product; only that one section degrades or disappears.
32. How do I implement retries without causing storms and loops?
Answer: Use exponential backoff with jitter (randomized delay added to each retry interval, so many clients don’t retry in lockstep), cap the maximum number of retries, and make sure retries respect circuit breaker state (don’t keep retrying a service a circuit breaker has already marked as down).
Trade-off: More aggressive retrying improves individual request success rates during transient blips, but naive retrying (fixed interval, no jitter, no cap) is one of the most common causes of self-inflicted outages — a struggling service gets hit with a synchronized wave of retries right when it can least handle it.
When to use what: Transient, likely-to-resolve-quickly failures (a brief network blip) → retry with backoff and jitter. Failures signaling sustained downstream trouble (repeated timeouts, a circuit breaker already open) → stop retrying and fail fast instead.
Industry example: AWS’s own SDKs implement exponential backoff with jitter by default specifically because AWS observed retry storms as a recurring cause of cascading failures across customer systems calling their APIs.
33. How do I design circuit breakers and bulkheads correctly?
Answer: A circuit breaker monitors calls to a dependency and “opens” (stops sending traffic, failing fast) once error rates cross a threshold, giving the struggling dependency room to recover, then periodically tests (“half-open” state) whether it’s safe to resume. A bulkhead isolates resources (connection pools, thread pools) per dependency, so one slow/failing dependency can’t exhaust resources needed by unrelated calls.
Trade-off: Circuit breakers add configuration complexity (thresholds, timeout windows) and can occasionally trip on transient blips if tuned too sensitively, cutting off traffic that would have actually succeeded. Bulkheads use more resources overall (dedicated pools per dependency rather than one shared pool) in exchange for isolating failures.
When to use what: Apply circuit breakers to every external dependency call in a production service, as close to a default as a retry policy. Apply bulkheads specifically where one flaky dependency has historically caused resource exhaustion affecting unrelated functionality.
Industry example: Netflix’s Hystrix library (and its modern successor, resilience4j) popularized this exact pattern industry-wide, born directly out of Netflix’s own experience with cascading failures in their microservices architecture.
stateDiagram-v2
[*] --> Closed
Closed --> Open: error rate exceeds threshold
Open --> HalfOpen: after timeout, try again
HalfOpen --> Closed: success
HalfOpen --> Open: failure34. How do I run in multiple regions and fail over cleanly?
Answer: Decide between active-active (multiple regions serving live traffic simultaneously, requiring data replication and conflict resolution) and active-passive (one primary region, one or more standby regions ready to take over), based on your RTO/RPO requirements. Implement health-checked, automated failover (via DNS or a global load balancer) rather than relying on manual intervention during an actual regional outage.
Trade-off: Active-active gives you the best resilience and lowest failover time (no failover needed — traffic is already distributed) but requires solving multi-region data consistency, which is genuinely hard. Active-passive is simpler to reason about and implement but has a real recovery time gap during failover, and the standby region’s readiness needs continuous verification (an untested failover path often doesn’t actually work when needed).
When to use what: Mission-critical systems where even minutes of downtime is unacceptable → active-active, accepting the added consistency complexity. Most other systems → active-passive is a reasonable, much simpler default.
Industry example: Netflix runs active-active across multiple AWS regions specifically so a full regional AWS outage doesn’t take Netflix down — a well-documented architectural choice driven directly by past regional AWS outages that did affect them.
35. How do I design for data center outage scenarios?
Answer: This is a superset of multi-region failover (Q34) plus operational readiness: regularly test failover (don’t just build it and hope), maintain runbooks that are actually practiced (not just written), ensure monitoring/alerting itself doesn’t have a single point of failure in the affected data center, and have a clear, pre-agreed decision process for when to declare a regional failover (since hesitation during a real incident is a common, costly failure mode).
Trade-off: Regularly testing full data-center failover (game days, chaos engineering at the regional level) has real operational cost and risk of its own, but untested failover procedures fail at a strikingly high rate when actually needed — the cost of not testing is usually far higher, just deferred and invisible until the real incident.
When to use what: The rigor here should scale with the actual cost of an outage — a company where downtime means direct, large revenue loss or safety risk should invest heavily in tested, practiced failover; smaller-stakes systems can reasonably accept a documented-but-less-frequently-tested plan.
Industry example: Google and AWS both run regular, deliberate disaster-recovery-testing exercises that intentionally simulate data center failures in production-adjacent environments specifically to validate failover actually works, rather than trusting it does.
36. Where should I cache — client, edge, service, or DB layer?
Answer: Cache as close to the requester as the data’s staleness tolerance allows: client-side (browser/app) caching for data that rarely changes and is user-specific; edge/CDN caching for shared, publicly cacheable content; service-layer caching (in-memory or Redis) for computed results or hot data shared across requests; database-layer caching (query cache, buffer pool) as the last line, closest to the source of truth.
Trade-off: Caching closer to the user gives the biggest latency win but is the hardest to invalidate correctly and consistently across many distributed clients. Caching closer to the database is easier to keep consistent but delivers a smaller latency improvement.
When to use what: Static or rarely-changing public content → CDN/edge. User-specific, moderately dynamic data → service-layer cache. Expensive, frequently-repeated computations → wherever in the stack that computation actually happens.
Industry example: Cloudflare and Fastly’s entire business model is built around edge caching — pushing cacheable content as close to end users globally as possible, specifically to minimize the latency and origin-server load that layer eliminates.
37. How do I pick between write-through, write-back, and write-around caching?
Answer: Write-through writes to cache and the underlying store simultaneously (strong consistency between the two, simplest to reason about, but every write pays the full latency of both). Write-back writes to cache immediately and asynchronously flushes to the store later (fastest writes, but risks data loss if the cache fails before flushing). Write-around writes directly to the store, bypassing the cache (good for data that’s written once but rarely re-read soon after, avoiding cache pollution).
Trade-off: Write-through trades write latency for consistency and durability. Write-back trades durability risk for write speed. Write-around avoids cache pollution from write-heavy, read-rarely data but means the first read after a write is always a cache miss.
When to use what: Financial or other durability-critical writes → write-through. High-throughput writes where brief data loss risk is acceptable (e.g., non-critical metrics/counters) → write-back. Bulk data loads or logs rarely re-read immediately → write-around.
Industry example: Most e-commerce inventory systems use write-through caching for stock counts specifically because a cache/database mismatch on inventory can mean overselling a product.
38. How do I design cache invalidation rules that don’t become a nightmare?
Answer: Prefer short TTLs (time-to-live) over manual invalidation wherever staleness tolerance allows — a TTL-based cache self-heals without any invalidation logic at all. Where you genuinely need immediate invalidation, use explicit, targeted invalidation (invalidate the specific key that changed) rather than broad invalidation (clearing entire cache regions), and consider event-driven invalidation (the write path publishes an event that triggers cache invalidation) to keep the logic centralized rather than scattered across every write path.
Trade-off: TTL-based expiry is simple and self-healing but means some staleness is always possible within the TTL window. Explicit invalidation is more precise (near-zero staleness) but adds real complexity and more ways to get it subtly wrong — a famous joke in computer science calls cache invalidation one of the genuinely hardest problems for a reason.
When to use what: Default to TTL-based expiry unless a specific feature genuinely can’t tolerate the staleness window; reserve explicit invalidation for cases where correctness truly requires it.
Industry example: Most CDN configurations default to TTL-based cache expiry precisely because coordinating explicit invalidation across a globally distributed edge network at scale is exponentially harder than accepting a bounded staleness window.
39. How do I handle cache stampede and thundering herd?
Answer: Use request coalescing (only the first request for an expired key actually queries the source; concurrent requests for the same key wait for that one result rather than all hitting the database simultaneously), or probabilistic early expiration (refresh a cache entry slightly before it actually expires, staggered randomly, so many keys don’t expire at the exact same moment).
Trade-off: Request coalescing adds implementation complexity (locking/queueing logic in the cache layer) but directly prevents the exact failure mode where a popular key’s expiration causes a sudden spike of simultaneous requests to hit the database at once.
When to use what: Any cached value that’s both expensive to recompute and likely to have many concurrent readers (a popular product page, a trending post) is a candidate for stampede protection; low-traffic or cheap-to-recompute keys usually don’t need it.
Industry example: Facebook’s Memcache paper (widely cited in distributed systems literature) explicitly describes techniques for handling exactly this problem at massive scale, since a naive cache-expiry approach at Facebook’s traffic volume would reliably cause database overload on every popular key’s expiration.
40. What is my caching story for hot keys and large objects?
Answer: For hot keys (a small number of keys receiving disproportionate traffic), replicate the hot key’s value across multiple cache nodes or add client-side local caching layered on top of the shared cache, rather than letting one cache node/shard absorb all the traffic for that key. For large objects, consider whether the whole object needs caching or just a computed summary, and be mindful of cache memory pressure — a few very large cached objects can evict many smaller, still-useful entries.
Trade-off: Replicating hot keys across nodes trades some memory duplication for eliminating a single-node bottleneck. Caching large objects wholesale is simple but risks cache memory pressure and eviction thrashing affecting unrelated cached data.
When to use what: Genuinely hot keys (identifiable via monitoring, not guesswork) warrant special handling; most keys in a well-designed system don’t need it.
Industry example: Twitter’s caching infrastructure has publicly documented handling celebrity accounts as an explicit hot-key case, since a single viral tweet’s engagement data can dwarf normal traffic patterns by orders of magnitude.
41. How do I measure whether a cache is actually helping?
Answer: Track cache hit ratio (the percentage of requests served from cache vs. falling through to the source), latency comparison between cache hits and misses, and — critically — the actual load reduction on the underlying data store, not just the cache’s own metrics in isolation.
Trade-off: A high hit ratio alone doesn’t prove value if the underlying query was already fast — the real question is whether the cache is meaningfully reducing load or latency where it actually matters, which requires measuring the source system too, not just the cache.
When to use what: Treat cache effectiveness as an ongoing metric to monitor, not a one-time design decision — access patterns shift over time, and a cache that was valuable at launch can become dead weight (or vice versa) as usage evolves.
Industry example: Most mature engineering orgs track cache hit ratio as a first-class SRE dashboard metric precisely because a silently degrading hit ratio is an early warning sign of either a capacity problem or a shifting access pattern that needs investigation.
42. How do I design pagination for massive datasets efficiently?
Answer: Avoid offset-based pagination (OFFSET 100000 LIMIT 20) at scale — the database still has to scan and discard all the skipped rows, getting slower as the offset grows. Use cursor/keyset pagination instead (WHERE id > last_seen_id LIMIT 20), which uses an index to jump directly to the right starting point regardless of how deep into the dataset you are.
Trade-off: Cursor-based pagination is more efficient at scale and doesn’t degrade with depth, but it doesn’t support jumping to an arbitrary page number (“go to page 500”) the way offset pagination naturally does — you trade that specific UX capability for consistent performance.
When to use what: Small datasets or UIs that genuinely need arbitrary page-jumping → offset pagination is fine. Large or infinite-scroll-style datasets (social feeds, large API result sets) → cursor-based pagination is close to mandatory at scale.
Industry example: Twitter’s and most major social platforms’ public APIs use cursor-based pagination for exactly this reason — timelines are far too large for offset pagination to remain performant.
43. How do I design rate limiting that doesn’t break good users?
Answer: Use algorithms like token bucket or sliding window (rather than a naive fixed window, which allows bursty abuse right at window boundaries), set limits per meaningful identity (per API key/user, not just per IP, since legitimate users can share IPs behind NAT), and return clear, actionable responses (429 status with a Retry-After header) so well-behaved clients can back off gracefully rather than just failing.
Trade-off: More sophisticated rate-limiting algorithms (sliding window, token bucket with burst allowance) are fairer to legitimate bursty usage but are more complex to implement and reason about than a naive fixed-window counter.
When to use what: Public APIs need per-client rate limiting as a baseline defense regardless of traffic volume. Internal service-to-service calls may need looser or no rate limiting if both sides are trusted and capacity-planned together, reserving strict limiting for the actual trust boundary.
Industry example: GitHub’s public API rate limiting (documented extensively, including response headers showing remaining quota and reset time) is a widely referenced example of rate limiting designed explicitly to let well-behaved clients self-regulate rather than being surprised by failures.
Closing Thoughts
Read straight through, these 43 questions form something close to a complete mental checklist for designing, operating, and evolving a real production system — not a system that merely works in a demo, but one that survives real traffic, real failures, and real organizational change over years.
A pattern worth noticing: almost none of these questions have a single universally correct answer. Nearly every one resolves to “it depends on what this specific data or feature actually needs” — which is precisely why system design is a discipline of trade-offs, not a checklist of best practices to apply uniformly. The engineers and architects who get good at this aren’t the ones who’ve memorized the “right” answer to each question — they’re the ones who’ve gotten fast and rigorous at working out which answer fits the specific system in front of them.
Cheers,
Sim