Opening system design
Progress is saved as you study.
Opening system design
Progress is saved as you study.
Research rank: #8 most asked
Asked at: Amazon, Stripe, Anthropic, OpenAI
Estimated interview time: 45 minutes
Concepts exercised: Redis, distributed counting, algorithms, API design
Related patterns: Caching, Networking & APIs
Design a rate limiter that protects APIs from excessive traffic. Clients can be users, tenants, API keys, IP addresses, or internal services. For each request, the limiter returns allow or reject.
The system must make low-latency decisions at high QPS, return clear retry data, and enforce fair usage across tenants. In an interview, use a Redis-backed token bucket with atomic updates as the default design.
Start by narrowing scope. Rate limiting is easy on one server. The problem becomes distributed once traffic reaches many API servers.
100 requests / second10,000 requests / hour500< 2ms in the same AZ.< 1ms.State the default clearly:
Use token bucket for normal API rate limiting. Token bucket allows short bursts and enforces a long-term average rate.
There are two API surfaces:
High-throughput systems often run the limiter as a library, sidecar, Envoy filter, or gateway plugin. For interview clarity, expose the limiter as a service API first. Then mention that production deployments often move the limiter closer to the request path.
POST /v1/rate-limit/check
Request:
{
"key": "tenant:stripe_demo",
"scope": "POST:/v1/payments",
"cost": 1,
"now_ms": 1767225600123
}
Response when allowed:
{
"allowed": true,
"limit": 100,
"remaining": 42,
"retry_after_ms": 0,
"reset_after_ms": 580
}
Response when blocked:
{
"allowed": false,
"limit": 100,
"remaining": 0,
"retry_after_ms": 240,
"reset_after_ms": 240
}
The cost field supports weighted requests. A cheap GET /status can cost 1. A large model inference, payment batch, or export job can cost 100. Strong candidates mention weighted requests because production systems rarely have uniform work per request.
PUT /v1/rate-limit/policies/{policy_id}
Request:
{
"policy_id": "pro_payments_api",
"algorithm": "token_bucket",
"rate_per_sec": 100,
"burst": 500,
"scope": "POST:/v1/payments"
}
Response:
{
"policy_id": "pro_payments_api",
"version": 17,
"updated_at": "2026-01-01T00:00:00Z"
}
PUT /v1/rate-limit/assignments/{subject}
Request:
{
"subject": "tenant:stripe_demo",
"policy_id": "pro_payments_api"
}
Response:
{
"subject": "tenant:stripe_demo",
"policy_id": "pro_payments_api",
"version": 9
}
When the API server rejects a request, return:
HTTP/1.1 429 Too Many Requests
X-RateLimit-Limit: 100
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1767225601
Retry-After: 1
Use Retry-After because clients and SDKs understand it. Use X-RateLimit-* headers to help humans and client libraries debug throttling.
Rate limiting has two data types:
Store configuration in a durable database. Store runtime state in Redis because it changes on every request and needs sub-millisecond operations.
CREATE TABLE rate_limit_policies (
policy_id TEXT PRIMARY KEY,
algorithm TEXT NOT NULL,
rate_per_sec DOUBLE PRECISION NOT NULL,
burst BIGINT NOT NULL,
scope TEXT NOT NULL,
created_at TIMESTAMP NOT NULL,
updated_at TIMESTAMP NOT NULL,
version BIGINT NOT NULL
);
Example:
policy_id: pro_payments_api
algorithm: token_bucket
rate_per_sec: 100
burst: 500
scope: POST:/v1/payments
CREATE TABLE rate_limit_assignments (
subject TEXT NOT NULL,
scope TEXT NOT NULL,
policy_id TEXT NOT NULL REFERENCES rate_limit_policies(policy_id),
version BIGINT NOT NULL,
updated_at TIMESTAMP NOT NULL,
PRIMARY KEY (subject, scope)
);
Example:
subject: tenant:acme
scope: POST:/v1/payments
policy_id: pro_payments_api
Key:
rl:{tenant:acme}:{POST:/v1/payments}
Value:
{
"tokens": 381.5,
"last_refill_ms": 1767225600123
}
In production, store the value as Redis hash fields or as a compact string. The encoding matters less than atomic updates.
Set TTL on bucket keys. A good default is:
ttl = max(2 * burst / rate_per_sec, 60 seconds)
TTL prevents idle tenants from consuming Redis memory forever. For example, a bucket with rate = 100/sec and burst = 500 refills in 5s, so a 60s TTL is safe.
Request path:
Client
-> API Gateway
-> Rate Limiter Client / Filter
-> Rate Limiter Service
-> Redis Cluster
-> API Service
Configuration path:
Admin / Billing System
-> Policy Management API
-> Postgres
-> Config Cache / PubSub
-> Rate Limiter Service
Named components:
API Gateway
Rate Limiter Client
Rate Limiter Service
Redis Cluster
Postgres
Config Cache
Default production design:
The gateway calls a rate limiter service. The service uses Redis Lua scripts for atomic token bucket updates. Policies live in Postgres and are cached in memory.

Token bucket is the best default for API rate limiting because it enforces an average rate and allows short bursts.
A bucket has:
For each request:
elapsed * refill_rate tokens.Example:
rate = 100 tokens/sec
burst = 500 tokens
cost = 1 token/request
The tenant can send 500 requests immediately after being idle. After that, the tenant can sustain 100 requests per second.
Product teams usually want that behavior. A tenant should be allowed to send a short burst when long-term usage stays within contract.
Core formula:
new_tokens = min(burst, old_tokens + (now - last_refill) * rate)
allowed = new_tokens >= cost
If blocked:
retry_after = (cost - new_tokens) / rate
Use millisecond timestamps. Integer seconds create visible boundary edge cases unless the limit is coarse.
Fixed window counts requests in a time bucket:
key = tenant:123:2026-01-01T00:00:00Z
limit = 1000/min
Fixed window is simple and allows boundary bursts. A client can send 1000 requests at 00:00:59 and another 1000 at 00:01:00, creating 2000 requests in two seconds.
Use fixed window for simple administrative limits where boundary bursts are acceptable. Do not use fixed window as the main answer for a serious API.
Sliding window counters are more precise. A common Redis implementation keeps counts for the current and previous windows and weights them by elapsed time. Another implementation uses sorted sets with one entry per request.
Sliding window fits login attempts and fraud-sensitive operations where precision matters more than Redis cost. For general API rate limiting, pick token bucket because it gives the desired burst behavior with less state and lower Redis load.
Interview recommendation:
Default: token bucket.
Use sliding window for security-sensitive limits.
Use fixed window only for simple coarse limits.
Many candidates lose the interview here. They describe a correct algorithm on one server and skip the behavior across 100 API servers.
A local limiter stores counters in each API server’s memory. It is very fast because there is no network call. The result is approximate.
If the global limit is 1000 requests/sec and there are 10 servers, you might assign each server 100 requests/sec. That works only when traffic is evenly balanced. If one tenant’s traffic lands mostly on one server, the tenant gets throttled too early. If every server gives the tenant a full 1000/sec, the tenant can get 10,000/sec.
A centralized limiter stores state in Redis. All API servers check the same bucket.
API Server 1 \
API Server 2 -> Redis bucket for tenant:acme
API Server 3 /
Centralized Redis is accurate because every request updates the same state. It costs one network hop, usually < 1ms in the same AZ, plus Redis execution time. For external APIs with latency in the tens of milliseconds, that cost is usually acceptable.
Default interview answer:
Use centralized Redis for correctness. Add local pre-limiting as an optimization.
A strong production design often uses hybrid limiting:
Example:
global limit = 10,000/sec
each gateway local soft limit = 2,000/sec
Redis global hard limit = 10,000/sec
The local limiter protects Redis from abusive spikes. Redis remains the source of truth. If local and global disagree, Redis wins.
For very high-scale systems, each server can lease tokens from Redis. Each server periodically requests a batch, such as 1000 tokens, and spends them locally. Token leasing reduces Redis calls and allows some overshoot when servers crash or traffic shifts.
Do not lead with token leasing in a 45-minute interview. Mention it as an optimization after presenting the Redis-backed design.

graph TD
subgraph TokenBucket["Token Bucket Algorithm"]
Bucket["Bucket: N tokens\nRefill: R tokens/sec"]
Req["Request arrives"] --> Check{"Tokens > 0?"}
Check -->|"Yes"| Allow["Allow request\ntokens -= 1"]
Check -->|"No"| Reject["429 Too Many Requests\nRetry-After header"]
Refill["Background: refill\nR tokens/sec up to N"] --> Bucket
end
subgraph Architecture["Production Architecture"]
Client["Client request"] --> API["API Gateway"]
API --> RL["Rate Limiter\n(middleware)"]
RL --> Redis["Redis\nLua script for atomicity\nkey: user_id:endpoint"]
Redis -->|"allowed"| Backend["Backend Service"]
Redis -->|"rejected"| R429["429 + X-RateLimit-Remaining"]
end
subgraph Modes["Local vs Global"]
Local["Per-server counter\nfast, approximate"] --> Hybrid["Hybrid for production"]
Global["Centralized Redis\naccurate, +1ms latency"] --> Hybrid
end
Rate limiting is a read-modify-write problem. If two requests read the same bucket at the same time, both might think a token is available and both might decrement it.
Unsafe implementation:
GET tokens
GET last_refill
compute new value in app
SET tokens
SET last_refill
Concurrent requests can overwrite each other and allow too many requests.
Use a Redis Lua script. Redis executes a Lua script atomically, so no other command modifies the key during the calculation.
Pseudo-Lua behavior:
-- inputs:
-- KEYS[1] = bucket key
-- ARGV[1] = now_ms
-- ARGV[2] = rate_per_sec
-- ARGV[3] = burst
-- ARGV[4] = cost
-- ARGV[5] = ttl_ms
local bucket = redis.call("HMGET", KEYS[1], "tokens", "last_refill_ms")
local tokens = tonumber(bucket[1])
local last_refill = tonumber(bucket[2])
if tokens == nil then
tokens = tonumber(ARGV[3])
last_refill = tonumber(ARGV[1])
end
local now = tonumber(ARGV[1])
local rate = tonumber(ARGV[2])
local burst = tonumber(ARGV[3])
local cost = tonumber(ARGV[4])
local elapsed_ms = math.max(0, now - last_refill)
local refill = elapsed_ms * rate / 1000.0
tokens = math.min(burst, tokens + refill)
local allowed = 0
local retry_after_ms = 0
if tokens >= cost then
tokens = tokens - cost
allowed = 1
else
retry_after_ms = math.ceil((cost - tokens) * 1000.0 / rate)
end
redis.call("HMSET", KEYS[1],
"tokens", tokens,
"last_refill_ms", now
)
redis.call("PEXPIRE", KEYS[1], tonumber(ARGV[5]))
return {allowed, tokens, retry_after_ms}
The script gives atomicity, low latency, and one Redis round trip.
Senior candidates usually call out two details.
First, handle clock skew. If API servers have skewed clocks, one server may refill the bucket incorrectly. Redis has a TIME command, although Lua usage depends on Redis version and replication mode. A practical default is to synchronize service clocks with NTP and clamp negative elapsed time to zero. For stricter correctness, route all updates for a key to the same Redis primary and use Redis time in a supported way.
Second, handle Redis Cluster hash slots. If a script touches multiple keys, all keys must be in the same hash slot. For the base design, each script touches one bucket key, so clustering is simple. If you later enforce multiple limits in one script, use hash tags:
rl:{tenant:acme}:global
rl:{tenant:acme}:payments
The {tenant:acme} portion forces keys into the same Redis Cluster slot.
A rate limiter protects the backend and defines part of the public API contract. If clients lack retry data, they often retry aggressively and make overload worse.
On allowed responses, return:
X-RateLimit-Limit: 100
X-RateLimit-Remaining: 42
X-RateLimit-Reset: 1767225601
On rejected responses, return:
HTTP/1.1 429 Too Many Requests
Retry-After: 1
X-RateLimit-Limit: 100
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1767225601
Use Retry-After in seconds because it is a standard HTTP header. If the internal calculation is in milliseconds, round up. Avoid Retry-After: 0 because many clients will retry immediately. Use at least 1 second for HTTP responses.
For internal APIs or SDKs, return a machine-readable body:
{
"error": {
"code": "rate_limited",
"message": "Too many requests",
"retry_after_ms": 240
}
}
Make backoff deterministic and visible. A client that receives Retry-After: 1 should wait at least one second before retrying. SDKs should apply jitter because thousands of clients retrying at the exact same time create a new burst.
Recommended SDK behavior:
sleep = retry_after_ms + random(0, 100ms)
Do not hide rate limiting behind generic 500 errors. Use 429 for client-specific throttling. Use 503 only when the service itself is overloaded.
Per-tenant fairness prevents one large or abusive tenant from consuming capacity needed by everyone else.
The basic key should include tenant identity:
rl:{tenant_id}:{scope}
Example:
rl:{tenant:acme}:{POST:/v1/payments}
Each tenant gets its own bucket. That is the first fairness boundary.
You usually also need a global service-level limiter:
rl:{global}:{POST:/v1/payments}
A request must pass both:
If the tenant is within quota and the service is overloaded, the global limiter protects the backend. If the service is healthy and one tenant is abusive, the tenant limiter protects everyone else.
For expensive APIs, add endpoint-specific limits:
tenant:acme + GET:/v1/search -> 100/sec
tenant:acme + POST:/v1/exports -> 5/min
tenant:acme + POST:/v1/inference -> 1000 tokens/sec cost-weighted
Do not use a single tenant-wide limit for all endpoints when request cost varies widely. A tenant could spend quota on cheap reads or overwhelm the system with expensive writes.
Good default:
tenant-level global bucket + endpoint-specific bucket for expensive routes
Store plan-based limits in the policy table:
free: 10/sec, burst 20
pro: 100/sec, burst 500
enterprise: custom
Enterprise tenants may need dedicated Redis capacity or isolated keys if they become hot keys. A single Redis instance can handle roughly 100K-500K ops/sec, but one tenant doing a large share of that traffic can create imbalance.
For very large tenants, shard by sub-key:
rl:{tenant:acme}:shard:0
rl:{tenant:acme}:shard:1
...
rl:{tenant:acme}:shard:15
Sharded buckets trade strict accuracy for throughput. If requests spread randomly across 16 buckets, the tenant can overshoot because each shard has its own burst. Divide burst and rate across shards:
total rate = 160,000/sec
16 shards
per-shard rate = 10,000/sec
Do not introduce sharded buckets unless the interviewer pushes on hot keys. The clean default is one bucket per tenant per scope.
Real systems often enforce more than one limit per request. For example:
tenant global: 1000/sec
endpoint: POST /payments 100/sec
IP: 20/sec
global service: 50,000/sec
A request must pass every required limit.
The implementation detail is tricky. If you decrement one bucket and then fail another, you consumed quota for a rejected request.
There are three common approaches.
The simplest approach checks from most restrictive to least restrictive and accepts minor overcharging. That is acceptable for abuse prevention and poor for billing-sensitive quotas.
The better Redis approach evaluates all relevant buckets in one Lua script and commits decrements only if every bucket can pass. That requires all keys to live in the same Redis hash slot. Tenant hash tags make that manageable:
rl:{tenant:acme}:global
rl:{tenant:acme}:payments
rl:{tenant:acme}:ip:1.2.3.4
For a 45-minute interview, say:
For one limit, one Lua script per bucket is enough.
For multiple limits on the same request, use one Lua script that checks all buckets and commits only if all pass.
Use quick numbers. Do not spend 10 minutes estimating. The goal is to size Redis and validate the design.
Assume:
100M requests/day
peak = 3x average
one rate limit check per request
Average QPS:
100M / 86,400 = ~1,160 QPS
Peak QPS:
1,160 * 3 = ~3,500 QPS
Redis can handle this load easily. A single Redis primary with a replica is enough.
Now assume a larger API:
10B requests/day
average = 10B / 86,400 = ~116K QPS
peak = ~350K QPS
At this size, use Redis Cluster. If each request needs one Lua script, peak load is around 350K Redis ops/sec. A small cluster can handle that if keys are evenly sharded.
Memory estimate:
Assume active buckets:
10M active tenant+scope buckets
200 bytes per bucket including overhead is optimistic
500 bytes per bucket is safer
Storage:
10M * 500 bytes = 5 GB
Even with Redis overhead and replication, 5 GB is manageable. Redis CPU and hot keys are more likely bottlenecks than memory.
Bandwidth estimate:
Assume Redis request plus response is around 500 bytes.
At 350K QPS:
350K * 500 bytes = 175 MB/sec
= 1.4 Gbps
Datacenter networks commonly run at 25-100 Gbps, so bandwidth is usually not the main constraint.
Latency estimate:
API server -> Redis same AZ: <1ms
Lua execution: sub-ms for simple script
added latency target: ~1ms
If the API latency budget is very tight, place Redis in the same AZ and keep the limiter in the gateway process. Do not call cross-region Redis for per-request limiting. Cross-region latency, such as US-EU 70-90ms, is unacceptable on the hot path.
If Redis is down, the limiter cannot make accurate global decisions.
Choose a default failure policy by traffic class:
Good default answer:
Fail closed for unauthenticated and expensive endpoints.
Fail open for trusted internal traffic with alerts and local emergency limits.
Do not give one failure mode for all traffic. State which traffic gets which behavior.
If Redis latency rises from <1ms to 20ms, API latency degrades.
Mitigations:
5ms.The timeout must be lower than the API latency budget. A rate limiter that waits forever becomes an outage amplifier.
A single tenant can create a hot Redis key.
Mitigations:
Default recommendation:
Start with one bucket per tenant per scope.
Move only the top tenants to sharded or dedicated handling when metrics show hot keys.
If API servers send inconsistent timestamps, buckets can refill incorrectly.
Mitigations:
For most interviews, NTP plus clamping is an acceptable default.
If a tenant upgrades or gets blocked, old policy values may remain in memory briefly.
Mitigations:
For billing plan changes, 30 seconds of staleness is usually acceptable. For abuse blocks, push invalidation immediately.
If a tenant sends traffic to multiple regions, each region may enforce only local limits.
Options:
Default recommendation:
Use per-region limiters with allocated quota per region.
For example, split a 10,000/sec tenant limit into 5,000/sec in us-east and 5,000/sec in eu-west, then rebalance periodically.
Do not put cross-region Redis in the request path. Latency is too high and failure behavior is poor.
Clients that receive 429s may retry immediately.
Mitigations:
Retry-After.The rate limiter should reduce overload. Clients that retry without backoff can increase load unless the API contract is clear.
An L5 candidate should produce a correct, practical design for the normal case.
Strong L5 signals:
429 with Retry-After.A solid L5 answer sounds like:
I would use a Redis-backed token bucket. Each tenant and endpoint has a bucket with capacity and refill rate. API servers call a rate limiter before handling the request. The Redis update is a Lua script so refill and decrement are atomic. Policies live in Postgres and are cached in the limiter. On reject, return 429 with Retry-After and remaining quota headers.
That is enough for a hire-level answer if the candidate can defend the details.
An L6 candidate should show production judgment in addition to algorithm knowledge.
Strong L6 signals:
A strong L6 answer sounds like:
The default is centralized Redis token bucket for correctness, with local pre-limits to protect Redis during spikes. Each request may check tenant, endpoint, and global service buckets. For one bucket, a Lua script is enough; for multiple buckets, use one script that checks all and commits only if all pass. Policies are durable in Postgres and cached in memory. For multi-region, I would avoid cross-region Redis and allocate quota per region, then rebalance. Redis failure behavior depends on endpoint risk: fail closed for expensive public APIs, fail open with local emergency limits for trusted internal APIs.
The most common interview failure is stopping at “use Redis counters.” That answer is incomplete. Cover distributed enforcement, atomicity, burst behavior, and failure behavior for Redis and clients. Keep the base design simple. Go deeper where the interviewer pushes.
A worked system design answer for Rate Limiter, timed at about 40 minutes — the length of the design portion of a real round. Reported asked at Stripe, Anthropic.
It walks the same sequence an interviewer expects: clarify the requirements, size the load, settle the API, choose the storage and the caching, then defend the trade-offs. It builds on the Caching, Scalability families and belongs to Caching.