WhatsApp: Extreme Efficiency with Erlang/OTP at Massive Scale

When Facebook acquired WhatsApp for $19 Billion in February 2014, the engineering team consisted of just 32 software engineers supporting 450 million active users and over 50 billion messages per day. By 2016, a team of roughly 50 engineers supported 900 million users.

This unprecedented level of engineering leverage was achieved through two core principles:

1. Radical Architectural Simplicity: WhatsApp intentionally avoided building complex social graphs, advertising tracking networks, server-side search indexes, or heavy relational data pipelines.
2. The Erlang/OTP Runtime Environment: Choosing Erlang and the BEAM Virtual Machine allowed WhatsApp to build a distributed, fault-tolerant messaging switch capable of handling millions of concurrent persistent socket connections per physical server.

Traditional web architectures (such as Java, C#, or Python WSGI) allocate an Operating System (OS) thread or process per user connection. An OS thread incurs a default memory stack allocation of 1 MB - 2 MB, meaning a single server with 64GB of RAM runs out of memory after just 30,000 - 50,000 connections. Furthermore, OS thread context switching causes severe CPU cache thrashing.

Erlang solves this through the BEAM Actor Model:

• Microscopic Memory Footprint: An Erlang actor process is allocated in user space and consumes only ~ 2 KB of RAM at creation.
• Preemptive Multi-Core Schedulers: BEAM runs one OS thread per physical CPU core as an Erlang scheduler. Schedulers manage queues of millions of Erlang processes, allocating execution time based on a strict reduction budget (typically 2,000 function calls/reductions). No single runaway task or calculation can monopolize CPU resources or starve other connections.
• Share-Nothing Message Passing: Erlang processes have private heaps and never share memory. Communication occurs purely via asynchronous, non-blocking message passing. Because there is zero shared mutable state, there are no database table locks, mutex deadlocks, or thread synchronization bottlenecks.
• OTP Supervisor Trees & "Let It Crash": Instead of writing thousands of lines of defensive error handling for every edge case, Erlang uses supervisor trees (one_for_one, one_for_all). If an unexpected runtime exception crashes an actor process handling a single user session, the supervisor immediately restarts that specific actor in microseconds without impacting any other connected users.

To achieve maximum hardware density, WhatsApp ran its messaging infrastructure on bare-metal servers running FreeBSD rather than standard Linux distributions.

Through aggressive OS kernel tuning, WhatsApp achieved over 2.8 million active, concurrent TCP connections on a single dual-socket server:

• Event Multiplexing via kqueue: FreeBSD's kqueue kernel event notification mechanism provided non-blocking I/O with O(1) scaling across millions of open file descriptors.
• TCP Socket Buffer Optimization: The default TCP buffer size in operating systems is often 64 KB - 128 KB per socket. Holding 2 million sockets at 128KB would require 256GB of pure socket buffer RAM. WhatsApp tuned tcp_sendspace and tcp_recvspace down to 4 KB - 8 KB, dynamically growing buffer size only when active data transmission was occurring.
• File Descriptor & Port Scaling: WhatsApp expanded system-wide limits (kern.maxfiles = 10000000, kern.ipc.maxsockets = 10000000) and assigned multiple IP aliases to network interfaces to overcome the 65,535 ephemeral port limit per IP.
• Custom BEAM VM Patches: WhatsApp engineers wrote custom C patches for the Erlang BEAM runtime to eliminate lock contention in internal process tables and timer wheels.

A primary reason large distributed systems collapse under load is continuous database accumulation: storing petabytes of user messages, indexing them for full-text search, and maintaining secondary indexes across billions of historical chat records.

WhatsApp bypassed this entire class of problems with its Ephemeral Messaging Architecture:

1. In-Flight Message Routing: WhatsApp acts as a store-and-forward routing pipeline rather than a permanent database.
2. Delivery & Instant Deletion: When Alice sends a message to Bob:
   • Alice's Erlang connection actor passes the encrypted payload to the routing layer.
   • If Bob is connected, the message is routed to Bob's Erlang connection actor and pushed to his device immediately.
   • As soon as Bob's phone sends back a cryptographic Delivery ACK (the "Double Tick"), the message is permanently and immediately deleted from server memory and temporary storage.
3. Offline Spooling: If Bob is offline, the encrypted message is temporarily spooled in an ephemeral Mnesia / database store. Once Bob reconnects and acknowledges receipt, the spooled messages are delivered and purged.
4. End-to-End Encryption (Signal Protocol): WhatsApp servers only route encrypted ciphertext blobs. The servers do not hold decryption keys, making server-side chat indexing architecturally impossible and eliminating compliance/privacy storage burdens.