Design a rate limiter for an API
System design, Medium
Design a service that limits each API key to 100 requests per minute. The API runs on 20 servers behind a load balancer in one region. Peak traffic is 20,000 requests per second across about 50,000 active keys. The check must add less than 2 ms at the 99th percentile. Cover: the algorithm and why you chose it, where the counters live, what you return to a blocked caller, what happens when the counter store is down, and how you know it works. Out of scope: billing and per-plan limit configuration.
Examples
Input One API key sends 101 requests within one minute, spread over 20 servers.
Output Requests 1 to 100 are allowed. Request 101 is refused with HTTP 429 and a Retry-After header.
Why The limit is per key, not per server, and the caller learns when to try again.
Input A caller sends 100 requests at 0:59 and 100 more at 1:00.
Output Your design must say how many pass. A strict 100 per minute allows 100 of the 200.
Why This is the edge case where a counter that resets on the minute and a counter that moves with time give different answers.
Input The counter store does not answer for 30 seconds while requests keep coming.
Output Your design must say what each request gets in that time: allowed or refused, and why.
Why This is a choice the answer must state, with its cost.
Constraints
- The limit is 100 requests per minute per API key.
- 20 API servers behind one load balancer in one region. Peak 20,000 requests per second across about 50,000 active keys.
- The check adds less than 2 ms at the 99th percentile.
- Out of scope: billing and per-plan limit configuration.
Your answer must contain
- Rough load and memory numbers, the algorithm with its main weakness, the data layout and the atomic step, the failure choice, and a test or metric plan.