Instagram: Scaling Python/Django & PostgreSQL Sharding

Instagram was founded in 2010 by Kevin Systrom and Mike Krieger, scaling to 30 million users within 18 months with only three backend engineers. Rather than rewriting their codebase in C++ or Go, Instagram scaled to hundreds of millions of daily active users while maintaining its Python and Django monolithic codebase.

They achieved this by strictly adhering to foundational scaling patterns:

• 100% Stateless Web Tier: Django application servers store no user session state, uploaded images, or temporary state in local memory or local disks. All requests are completely stateless, allowing hundreds of Django servers to auto-scale behind NGINX load balancers.
• Asynchronous Task Offloading: Time-consuming operations (image resizing, push notification dispatching, activity feed fan-out, spam scoring) are packaged into background jobs and queued into RabbitMQ, executed asynchronously by a fleet of Celery workers.
• Python Runtime Optimizations: To reduce memory consumption across thousands of multi-threaded uWSGI worker processes, Instagram famously disabled Python's cyclic garbage collector on shared pre-forked memory pages, saving over 8GB of RAM per server through Copy-on-Write (CoW) preservation.

When Instagram's single PostgreSQL primary database hit hardware CPU and I/O capacity in 2011, standard vertical scaling was no longer possible. Instead of migrating to an unproven NoSQL database (which would sacrifice relational data integrity and ACID transactions), Instagram implemented Logical Sharding in PostgreSQL:

The Logical Shard Concept:

• Instead of sharding data across a fixed number of physical database machines, Instagram pre-split their relational data into thousands of Logical Shards (PostgreSQL Schemas)—typically 1,024 to 4,096 schemas (shard_0001 to shard_1024).
• Each logical schema contains its own independent set of tables: photos, likes, comments, follows.
• Physical Server Mapping: Initially, Instagram ran all 1,024 logical schemas on just 16 physical PostgreSQL database servers (each physical machine hosting 64 schemas).
• Zero-Downtime Hardware Expansion: As user traffic doubled, database engineers provisioned 16 additional physical servers. They moved 32 schemas from each original machine to the new machines using standard PostgreSQL streaming replication and pg_dump. Not a single line of application code needed to change, because the Django application only tracks a lightweight routing map: logical_shard_id -> physical_hostname.

In a horizontally sharded database environment, traditional auto-incrementing integer primary keys (AUTO_INCREMENT or SERIAL) fail because different database shards will generate conflicting duplicate IDs. UUIDs (128-bit strings) are poorly suited as primary keys because they are non-sequential, causing severe index fragmentation and cache misses in B-Tree indexes.

To solve this, Instagram engineered a custom 64-bit ID generator using PostgreSQL stored procedures:

The 64-Bit ID Bit Allocation:

• 41 Bits (Timestamp): Milliseconds elapsed since a custom Instagram epoch (giving > 69 years of unique timestamps).
• 13 Bits (Logical Shard ID): Represents the logical shard where the record was created (supporting up to 2^{13} = 8,192 logical schemas).
• 10 Bits (Sequence Counter): Auto-incrementing modulo sequence counter (2^{10} = 1,024 IDs per shard per millisecond).

ID = (Timestamp \ll 23) \mid (Shard ID \ll 10) \mid (Sequence \pmod{1024})

Architectural Advantages:

1. Self-Routing Primary Keys: When a client requests photo ID 3459821873912, the application extracts the 13-bit Shard ID using a bitwise mask. It routes the SQL query directly to the correct physical database without querying a central index.
2. Chronologically Sortable: Because the most significant 41 bits represent time, IDs are naturally ordered chronologically. Paginating user feeds requires a simple WHERE id < last_seen_id ORDER BY id DESC LIMIT 20 query.

Instagram is intensely read-heavy, with a read-to-write ratio exceeding 100:1. To prevent read queries from overwhelming PostgreSQL database shards:

1. Memcached Caching Tier:
   • Memcached clusters cache rendered user profiles, photo metadata, and friendship relationships in DRAM.
   • Django uses a Cache-Aside (Lazy Loading) pattern: Check Memcached -> On Miss, query PostgreSQL -> Populate Memcached with a 24-hour TTL.
2. Redis for Activity Feeds & Counters:
   • Real-time notification counters (unread notifications, like counts) are maintained in in-memory Redis Hashes and Sorted Sets.
3. Primary-Replica Read Splitting:
   • Each physical PostgreSQL shard operates with multiple read replicas. Read-heavy feed generation queries target local replicas, while mutating transactions (creating a post, deleting a comment) execute on the primary shard.