Design a Global Notification System

A Global Notification System coordinates, formats, and dispatches billions of transactional, promotional, and real-time alerts across multiple channels (iOS/Android Push, SMS, Email, In-App).

Functional Requirements

1. Multi-Channel Dispatch:
   • Mobile Push: Apple Push Notification service (APNs), Google Firebase Cloud Messaging (FCM).
   • SMS: Twilio, MessageBird, Infobip (for 2FA codes, order updates).
   • Email: SendGrid, Amazon SES, Mailgun (receipts, marketing campaigns).
   • In-App / Webhook: WebSocket feeds and partner webhook integrations.
2. User Preference Management: Respect user opt-in/opt-out settings per category (Marketing vs Transactional) and local time zone Do Not Disturb (DND) quiet hours.
3. Template & Localization: Support dynamic parameter substitution and multi-language localization (i18n).
4. Delivery Tracking & Webhooks: Capture real-time delivery receipts, open rates, and click tracking.

Non-Functional Requirements

• High Throughput & Scale: Handle 100 million notifications/day (~1,200 QPS, peak 10,000 QPS).
• Low Latency for High-Priority Alerts: Critical transactional alerts (e.g., 2FA verification SMS) delivered in < 3 seconds.
• Fault Domain Isolation: A failure or rate-limit throttle in one channel (e.g., Twilio outage) must never stall or impact other channels (e.g., APNs push notifications).
• At-Least-Once Delivery with Deduplication: Ensure messages are delivered without sending duplicate marketing blasts.

Daily Volume & Throughput

• Total Notifications: 100 million notifications/day.

Average Throughput = \frac{100,000,000}{86,400} ≈ 1,157 notifications/sec

Peak Throughput (Breaking News / Flash Sale) = 10,000 to 25,000 notifications/sec

Channel Breakdown:

• Push Notifications: 60\% (60M/day)
• Email: 30\% (30M/day)
• SMS: 10\% (10M/day)

Storage & Payload Size:

• Average notification metadata payload: 500 bytes.
• Daily storage for delivery logs and analytics:

100M × 500 bytes = 50 GB/day \implies 1.5 TB/month

To guarantee fast checks before enqueuing notifications, cache user preferences and device tokens in Redis while storing master records in PostgreSQL or DynamoDB:

sql
Copy
-- User Device Tokens Table (for Push)
CREATE TABLE user_devices (
    device_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    user_id UUID NOT NULL,
    platform VARCHAR(10) NOT NULL, -- 'IOS', 'ANDROID', 'WEB'
    device_token TEXT NOT NULL,     -- APNs / FCM registration token
    is_active BOOLEAN DEFAULT TRUE,
    updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);

-- User Notification Preferences
CREATE TABLE notification_preferences (
    user_id UUID NOT NULL,
    channel VARCHAR(20) NOT NULL,    -- 'PUSH', 'SMS', 'EMAIL'
    category VARCHAR(50) NOT NULL,   -- 'SECURITY', 'ORDERS', 'MARKETING'
    is_enabled BOOLEAN DEFAULT TRUE,
    dnd_start_time TIME,             -- e.g. '22:00:00'
    dnd_end_time TIME,               -- e.g. '08:00:00'
    timezone VARCHAR(50) DEFAULT 'UTC',
    PRIMARY KEY (user_id, channel, category)
);

Endpoints

1. Send Notification API (Internal Microservice Call)
   • POST /api/v1/notifications/send
   • Request Body:

     json
     Copy
     {
       "user_id": "usr_99812",
       "priority": "HIGH",
       "category": "ORDERS",
       "template_id": "order_shipped_v2",
       "params": {
         "customer_name": "Alice",
         "order_id": "ORD-10948",
         "tracking_url": "https://carrier.com/track/10948"
       },
       "idempotency_key": "order_shipped_ORD-10948_push"
     }

   • Response (202 Accepted):

     json
     Copy
     {
       "notification_id": "notif_77a98bc1",
       "status": "QUEUED",
       "channels": ["PUSH", "EMAIL"]
     }

Why Channel Queue Isolation is Mandatory:

Third-party providers (Twilio, SendGrid, APNs) enforce strict per-account rate limits (e.g., Twilio standard limits are 100 SMS/sec).

• If all notification types shared a single monolithic queue, a sudden burst of 100,000 marketing emails would block time-sensitive 2FA verification SMS messages for minutes.
• Solution: Maintain completely separate message queues (Amazon SQS / RabbitMQ / Kafka) for each channel:
  1. push_notifications_queue
  2. sms_priority_queue (2FA & Security)
  3. sms_marketing_queue
  4. email_transactional_queue
  5. email_promotional_queue

Anti-Spam Rate Limiting

To prevent notification fatigue (e.g., an active group chat triggering 500 phone vibrations in 2 minutes):

• Redis Sliding Window Filter: Enforces maximum limits (e.g., max 1 push alert per user per 30 seconds for social events).
• Consecutive alerts are batched into a single aggregated message: "Alice and 4 others liked your post".

Deduplication via Idempotency Keys

• Network timeouts can cause the Order Service to retry the sendNotification call.
• The Ingestion Gateway checks Redis for SET NX "idemp:{idempotency_key}" EX 86400. If the key exists, return the existing status without queuing duplicate alerts.

Exponential Backoff & Dead-Letter Queues (DLQ)

• If APNs or Twilio returns a transient 5xx server error, workers retry with exponential backoff and jitter (1s, 2s, 4s, 8s).
• If an alert fails after 5 attempts (or fails with a permanent 4xx error like "Invalid Device Token" or "Unregistered Phone Number"), it is routed to a Dead-Letter Queue (DLQ) for automated alerting and invalid token unregistration.