The Pizza That Changed How I Read Dashboards

I used to be a senior engineer who trusted averages. Now I lead a systems team, and the first thing I do in any incident review is ask someone to stop showing me the mean.

Here’s the story I use to make the point, because it works on engineers and on product managers in the same meeting.

The Pizza Problem

Domino’s promises 30-minute delivery and ships 5,000 orders a day.

• P50 (median): 2,500 pizzas arrive in 30 minutes or less. Half the customers got what was promised. Half didn’t. This is the “typical” experience — and it’s already a coin flip.
• P95: 4,750 pizzas arrive in 45 minutes or less. That sounds fine, until you notice 250 customers had a worse night than that. Cold pizza, no call, no apology.
• P99: 4,950 pizzas arrive in an hour or less. Great number on a slide. Terrible number if you’re one of the 50 people who just switched to Pizza Hut.

An average delivery time can look healthy while 1% of your customers are quietly furious. Averages don’t lie exactly — they just don’t tell you who got hurt.

Why This Story Follows Me Into Production

Swap “pizza” for “API request” and the math doesn’t change.

Your dashboard says average response time is 80ms. Everyone nods. Meanwhile your P99 is sitting at 3 seconds for a slice of traffic nobody’s paying attention to. At 1 million requests a day, that 1% is 10,000 requests getting the cold-pizza treatment — every single day, silently, with no ticket filed and no page fired.

This is the gap between feeling like a system is healthy and knowing it is. Averages measure the center of a distribution. Incidents live in the tail.

What Changes When You Move From Senior to Lead

As a senior engineer, my job was mostly: is my service fast, correct, and shipped on time? I looked at P95/P99 for my own service and moved on.

As a lead, the job flipped. I’m no longer asking “is this fast” — I’m asking “fast for whom, under what conditions, and what happens when it isn’t.” That’s a different kind of question, and it shows up in three places:

1. Which percentile actually matters for this workload. P50 tells you about the common case. P95 tells you where your engineering effort should go next. P99 (and increasingly P99.9) tells you who’s about to churn, file a support ticket, or trip a downstream timeout. A lead engineer picks the percentile that matches the decision being made — capacity planning cares about P95, customer trust cares about P99, SLA compliance often cares about both plus the count behind them.

2. What’s causing the tail, not just measuring it. A senior engineer reports “P99 is 3 seconds.” A lead engineer asks: is that one noisy dependency, a specific customer’s data shape, a cold cache path, GC pauses, a hot partition, or retries stacking on top of retries? The percentile is a symptom. Leadership work is chasing the mechanism.

3. Whether the org is even structured to notice. This is the least technical and most important shift. Averages get you a green dashboard and a quiet Slack channel. Somebody has to decide that P95/P99 (and the raw counts behind them) are first-class numbers on every dashboard, every SLO, and every postmortem — not an appendix. That’s a leadership decision, not a metrics decision.

The Questions That Actually Expose How Someone Thinks

I started collecting the questions I ask architecture candidates and my own team, because the answers tell you more about someone’s maturity than any resume line. A few that consistently separate “can write code” from “can own a system”:

• How do you detect and isolate a slow service before it takes the rest of the system down with it? (Tail latency is often contagion — one slow dependency plus synchronous calls plus no timeout budget equals a full outage.)
• How do you design caching layers to avoid stale data and thundering herds? (Caches fix the average and can wreck the tail if everyone misses at once.)
• What are the signals that a system needs vertical vs. horizontal scaling? (People who’ve only read about this reach for “add more servers.” People who’ve lived it talk about hot partitions, lock contention, and where the actual bottleneck sits.)
• How do you design idempotent operations for systems with retries? (Retries are the single biggest hidden cause of tail-latency spikes turning into data-correctness incidents.)
• How do you design an audit-friendly system without killing performance? (This is where “I know the theory” and “I’ve paid the tax of doing it wrong once” diverge.)
• How do you handle partial failures in distributed systems? (Not “if it fails,” but “when part of it fails” — that framing alone tells you a lot.)

None of these have a single right answer. What I’m listening for is whether someone reasons from first principles about distributions and failure modes, or whether they reach for the textbook answer and stop. The pizza story is the same test in disguise: do you see the average, or do you see the 50 people who got cold pizza?

A Small Practical Framework

If you’re making the same transition — senior to lead, or just trying to get your team to stop trusting averages — here’s what I actually changed:

1. Every SLO gets a percentile and a count, not just a percentage. “99% under 200ms” means nothing until someone says “that’s 10,000 requests a day” out loud.
2. Every dashboard shows P50, P95, and P99 side by side, not the mean with P99 buried in a drill-down nobody clicks.
3. Every postmortem asks “who was in the tail” before it asks “what was the average impact.” The tail is where your angriest customers and your next outage both live.
4. Every architecture review includes at least one question from the list above. Not to trap anyone — to see how they think about the parts of the system that don’t show up in the happy path.

Averages look good. Percentiles tell the real story. If you’re leading a system instead of just building one, your job is making sure the whole team watches P95 and P99 — because that’s where the cold pizza lives.

Cheers,

Sim