Ridhuan Rangga KusumaRidhuan Rangga
Case StudiesSystem Design
Home
Writing

Case Study / 2023

Distributed Rate Limiting Service

Redis-based token bucket implementation handling 2M+ RPM with sub-millisecond overhead, rebuilt after the legacy PHP system caused production incidents.

2.1M RPM

Peak Throughput

Sustained during flash sale

<0.8ms p99

Rate limit overhead

Added latency per request

99.97%

Accuracy

Measured over 30 days

94% down

Abuse reduction

Scraping incidents after deployment

problem

Problem Statement

Business Context

Tokopedia's API gateway handled around 800k RPM at steady state with larger spikes during flash sale events. The old limiter kept counters per instance, effectively multiplying real limits by the number of gateway replicas.

Constraints

  • Do not add more than 2ms p99 latency.
  • Support per-client, per-endpoint, and per-IP policies.
  • Degrade gracefully when Redis is unavailable.

architecture

System Architecture

interactive diagram

System Architecture

ClientPolicy EngineCache LayerTelemetry

High-Level Design

The rate limiter runs as a sidecar beside each API gateway. Checks are synchronous gRPC calls, while Redis performs a single atomic Lua operation for the counter update.

tradeoffs

Key Trade-offs

Lua Script vs WATCH/MULTI/EXEC

Decision: use server-side Lua scripts. Lua avoids retry loops under contention while keeping the operation atomic.

Fail-open vs Fail-closed

Decision: fail-open with alerts. A rate-limiter failure should not take down customer traffic.

results

Results and Impact

The service handled 2.1M RPM during the first major sale after deployment with zero rate-limit accuracy incidents and a 94% reduction in abuse incidents.

Architecture Decisions

ADR 1 / accepted

Use Redis Lua Scripts for Atomic Rate Limit Operations

Rate limit check-and-increment must be atomic under concurrent requests from the same client.

Use Redis server-side Lua scripts, which execute atomically inside Redis.