Design a Rate-Limited API Gateway (Kong / Envoy)

An API Gateway serves as the single perimeter entry point for all external web, mobile, and third-party API traffic, centralizing cross-cutting concerns (authentication, rate limiting, routing, SSL offloading, observability) before forwarding requests to internal microservices.

Functional Requirements

1. Dynamic Path-Based Routing: Route requests (e.g., /api/v1/orders/* → Order Service) based on path, HTTP methods, or headers.
2. Authentication & Authorization: Verify JWT signatures and API keys at the perimeter.
3. Distributed Rate Limiting: Enforce per-user, per-IP, and per-tenant rate limits using Token Bucket algorithms.
4. Protocol Translation & SSL Offloading: Terminate external TLS 1.3 HTTPS/HTTP2 traffic and route internally via gRPC or HTTP/1.1 with mutual TLS (mTLS).
5. Circuit Breaking & Outlier Detection: Automatically shed load or return fallback responses when a downstream microservice degrades.

Non-Functional Requirements

• Ultra-Low Latency Overhead: Gateway filter pipeline must execute in < 2 ms (p99).
• High Throughput: Handle > 100,000 requests/second across an autoscaling gateway fleet.
• High Availability & Statelessness: Zero local state on gateway nodes; all rate limit counters stored externally in Redis.