Can this fleet survive a zone loss?
The normal fleet has spare capacity. Decide whether that is enough.
The question
Northline Dispatch runs equal worker counts in three independent zones. The packet models constant attempt throughput, not a latency percentile.
| Value | Unit | |
|---|---|---|
| Peak notification jobs | 12000 | jobs/second |
| Additional retry work | 25 | % of original attempts |
| Worker throughput | 2000 | attempts/second |
| Zone impairment | 120 | seconds |
| Worker price | 60 | USD/worker/month |
| Worker budget | 750 | USD/month |
All numbers are authored for this exercise. No cloud price or production benchmark is claimed.
All notification jobs must keep processing at peak during a single-zone loss, with no growing attempt backlog. No emergency worker launch is assumed. Durable enqueue and processing correctness are separate requirements.
“Choose a fleet within budget. Explain what must be tested before release.”
Not saved by this site. Your browser may restore text when you navigate back.
A strong answer, step by step
Open each step in turn. This is one strong answer, not the only one.
Step 1: State the demand
What would you do here? Open the move you’d make.
Size for 12,000 jobs/second because retries are exceptional.
Tempting, but it misses something. The failure-state packet explicitly includes retry work.
Include the stated 25% retry work in demand.
The strong move. That gives 15,000 attempts/second.
See how a strong candidate says it
“The demand is 15,000 attempts per second: 12,000 jobs × 1.25 attempts per job.”
Why it works: Jobs and attempts have different units. Retry work uses worker capacity.
Step 2: Remove one zone
What would you do here? Open the move you’d make.
Calculate capacity using only the two surviving zones.
The strong move. This exposes a 3,000-attempt/second shortfall.
Keep the normal 18,000-attempt/second capacity and rely on auto-scaling.
Tempting, but it misses something. The packet does not permit a new worker dependency during impairment.
See how a strong candidate says it
“Three workers per zone leave six workers, or 12,000 attempts/second, during a loss.”
Why it works: Normal capacity of 18,000 does not satisfy the surviving fleet constraint.
Step 3: Choose the bounded fleet
What would you do here? Open the move you’d make.
Use five per zone so there is no need to discuss risk.
Tempting, but it misses something. The $900 worker cost exceeds the $750 budget and still does not prove reliability.
Use four per zone, then test the small remaining margin.
The strong move. It meets the given capacity and worker-budget envelope.
See how a strong candidate says it
“Four workers per zone cost $720/month and leave 16,000 attempts/second after a zone loss.”
Why it works: The proposed fleet leaves 1,000 attempts/second of modeled headroom; five workers per zone cost $900.
Step 4: Set the release test
What would you do here? Open the move you’d make.
Exercise the actual loss, queue behavior and limiting dependency.
The strong move. A normal-fleet load test cannot establish the failure envelope.
Quote the worker RPS as proof of a latency SLO.
Tempting, but it misses something. Throughput does not determine a latency distribution.
See how a strong candidate says it
“Load-test the surviving zones at 15,000 attempts/second, including dependency limits and recovery.”
Why it works: The fluid arithmetic does not establish queue durability, failover time, p99 or correlated-failure behavior.
The answer, in short
- Decision
- Propose four workers per zone for the stated packet.
- Because
- The failure fleet can serve 16,000 attempts/second against 15,000 demand, within the authored worker budget.
- What would change it
- Higher retry work, a correlated dependency failure, or a measured capacity below the assumed throughput changes the result.
Common traps
- Sizing only the healthy fleet.
- Treating attempt capacity as a latency or availability guarantee.
Check your answer
- Did I convert jobs into attempts?
- Did I remove one whole zone?
- Did I include all workers in cost?
- Did I name the failure test and assumptions?
Last reviewed Nov 10, 2026