Case Study / 2023
Distributed Rate Limiting Service
Redis-based token bucket implementation handling 2M+ RPM with sub-millisecond overhead, rebuilt after the legacy PHP system caused production incidents.
2.1M RPM
Peak Throughput
Sustained during flash sale
<0.8ms p99
Rate limit overhead
Added latency per request
99.97%
Accuracy
Measured over 30 days
94% down
Abuse reduction
Scraping incidents after deployment
problem
Problem Statement
Business Context
Tokopedia's API gateway handled around 800k RPM at steady state with larger spikes during flash sale events. The old limiter kept counters per instance, effectively multiplying real limits by the number of gateway replicas.
Constraints
- Do not add more than 2ms p99 latency.
- Support per-client, per-endpoint, and per-IP policies.
- Degrade gracefully when Redis is unavailable.
architecture
System Architecture
interactive diagram
System Architecture
High-Level Design
The rate limiter runs as a sidecar beside each API gateway. Checks are synchronous gRPC calls, while Redis performs a single atomic Lua operation for the counter update.
tradeoffs
Key Trade-offs
Lua Script vs WATCH/MULTI/EXEC
Decision: use server-side Lua scripts. Lua avoids retry loops under contention while keeping the operation atomic.
Fail-open vs Fail-closed
Decision: fail-open with alerts. A rate-limiter failure should not take down customer traffic.
results
Results and Impact
The service handled 2.1M RPM during the first major sale after deployment with zero rate-limit accuracy incidents and a 94% reduction in abuse incidents.
Architecture Decisions
ADR 1 / accepted
Use Redis Lua Scripts for Atomic Rate Limit Operations
Rate limit check-and-increment must be atomic under concurrent requests from the same client.
Use Redis server-side Lua scripts, which execute atomically inside Redis.
