Real-Time Recommendation Systems & Vector Similarity Search

Large-scale content and commerce platforms (YouTube, TikTok, Netflix, Amazon, Spotify) maintain global catalogs containing tens to hundreds of millions of items. Running a state-of-the-art transformer or deep neural ranking model across the entire catalog for every user feed refresh would require billions of floating-point operations per millisecond—exceeding both GPU server budgets and strict human latency thresholds (< 50ms).

To reconcile massive scale with sub-50ms response times, production recommendation systems employ a multi-stage funnel architecture:

1. Candidate Generation (Retrieval): Sifts through the entire corpus (10^7 - 10^8 items) to extract ~ 1,000 relevant candidate items in < 10ms using fast vector embeddings and heuristic indices.
2. Hard Filtering: Discards invalid candidates (\~ 1,000 → 400 items) based on deterministic business rules (already watched content, blocked creators, out-of-stock SKUs, age and geographic restrictions).
3. Heavy Deep Ranking: Applies a complex, multi-layer neural network (such as Deep & Cross Network [DCN-v2] or Multi-gate Mixture-of-Experts [MMoE]) with hundreds of rich cross-features to score and sort the remaining ~ 400 items down to ~ 80 in < 25ms.
4. Re-Ranking & Diversity: Applies Maximal Marginal Relevance (MMR), contextual exploration bandits, creator deduplication, and ad-insertion logic to produce the final top-20 items served to the client in < 10ms.

The retrieval stage relies heavily on Two-Tower Neural Architectures:

• User Tower: Encodes real-time user context (user demographics, past 50 watched video IDs, search queries, current time, country) into a low-dimensional dense embedding vector u \in \mathbb{R}^d (d=128 to 256).
• Item Tower: Encodes static and semi-static item metadata (video title, creator, tags, transcript embeddings) into an item embedding vector v_i \in \mathbb{R}^d.

Because item embeddings v_i do not depend on who is searching, they are precomputed asynchronously offline and indexed in a Vector Database (Milvus, Pinecone, Qdrant, or Faiss). At runtime, computing candidate relevance reduces to an inner product or cosine similarity:

Score(u, v_i) = \langle u, v_i \rangle = \sum_{k=1}^{d} u_k \cdot v_{i,k}

Vector Indexing Algorithms: Flat vs. IVF-PQ vs. HNSW

Exhaustively computing dot products against 100 million vectors requires 100M vector distance calculations—far too slow for real-time traffic. Production vector engines use Approximate Nearest Neighbor (ANN) indexing:

1. Hierarchical Navigable Small World (HNSW): Constructs a multi-layer geometric graph. Search begins on a sparse top layer with long-range links and descends to dense bottom layers, achieving logarithmic search complexity O(log N) with >98\% recall in sub-5ms at the cost of high RAM consumption (\~ 1.5-2× raw vector size).
2. Inverted File with Product Quantization (IVF-PQ): Clusters the vector space into Voronoi cells (IVF) and quantizes 128-dimensional float32 vectors into 8-byte codebooks (PQ). Compresses memory by 16× to 32×, enabling billions of vectors to fit on a single node with slight loss in recall (90-95\%).
3. SCaNN (Anisotropic Vector Quantization): Google's state-of-the-art vector quantization optimized for Maximum Inner Product Search (MIPS) by penalizing quantization error parallel to the query vector.

import numpy as np
import hnswlib
import time

class RealTimeVectorRetriever:
    def __init__(self, dim: int = 256, max_items: int = 10_000_000):
        self.dim = dim
        # Initialize HNSW index with Cosine distance metric
        self.index = hnswlib.Index(space='cosine', dim=dim)
        # M = 32 (links per node), ef_construction = 200 (indexing quality vs speed)
        self.index.init_index(max_elements=max_items, ef_construction=200, M=32)
        # ef = 64 balances search latency (~3ms) with high recall (>97%)
        self.index.set_ef(64)

    def retrieve_top_k(self, user_embedding: np.ndarray, k: int = 500) -> tuple[list[int], list[float]]:
        """Retrieves top-k candidate item IDs and similarity scores in sub-5ms."""
        t0 = time.perf_counter()
        
        # Ensure user embedding matches expected dimensions
        query_vector = user_embedding.reshape(1, self.dim).astype(np.float32)
        
        # Fast approximate k-nearest neighbor query
        labels, distances = self.index.knn_query(query_vector, k=k)
        
        # Distances are cosine distances (1 - cosine_similarity)
        similarity_scores = (1.0 - distances[0]).tolist()
        candidate_ids = labels[0].tolist()
        
        elapsed_ms = (time.perf_counter() - t0) * 1000.0
        return candidate_ids, similarity_scores

Optimizing a recommendation system exclusively for a single engagement metric (such as Click-Through Rate [CTR]) results in clickbait optimization—the system recommends sensationalist headlines that users click but immediately abandon with dissatisfaction.

Modern recommendation engines utilize Multi-Task Learning (MTL), specifically Multi-gate Mixture-of-Experts (MMoE) architectures:

• Shared Experts Tier: Multiple specialized feed-forward neural networks (experts) learn fundamental representations of user preferences, content themes, and cross-feature interactions.
• Task-Specific Softmax Gating: Each prediction objective has a dedicated gating network that assigns dynamic attention weights across the shared experts.
• Simultaneous Multi-Objective Prediction:
  • p(Click): Probability the user clicks or taps the item.
  • p(Complete): Probability the user watches >80\% of the video duration.
  • p(Share): Probability the user forwards the item to external contacts.
  • p(Dislike): Probability the user flags or skips within 2 seconds.

The Composite Value Score Formula

The ranking engine computes a unified composite utility score for each candidate item i:

Score_i = w_1 \cdot p(Click) + w_2 \cdot p(Complete) + w_3 \cdot p(Share) - w_4 \cdot p(Dislike) + \alpha \cdot FreshnessBoost_i

The weights w_1, w_2, w_3, w_4 are tuned dynamically via offline business experimentation to maximize long-term 30-day user retention rather than short-term vanity clicks.

Even a highly accurate ranking model will fail in production if it creates filter bubbles—recommending 20 identical videos from the same creator or topic because the user clicked one item. Furthermore, newly published items have zero interaction history and will never be recommended unless explicitly given exposure.

Maximal Marginal Relevance (MMR) Algorithm

To guarantee diversity in the final feed, systems use Maximal Marginal Relevance (MMR) during the final re-ranking stage:

MMR(u, C) = \arg\max_{d_i \in C \setminus S} ≤ft[ \lambda \cdot Sim_1(d_i, u) - (1 - \lambda) \cdot \max_{d_j \in S} Sim_2(d_i, d_j) \right]

Where:

• C is the candidate pool from the ranking stage.
• S is the set of already selected items in the user feed.
• Sim_1(d_i, u) is the relevance of item d_i to user u.
• Sim_2(d_i, d_j) is the pairwise similarity between candidate d_i and already selected item d_j.
• \lambda \in [0, 1] is the diversity trade-off hyperparameter (typically \lambda ≈ 0.65).

Solving the Cold-Start Problem with Contextual Bandits

For newly uploaded videos or products with zero interaction data:

1. Content-Based Visual/Text Embeddings: Extract multi-modal embeddings from video frames, audio tracks, and descriptions via CLIP/Whisper models to place new items into the vector space immediately.
2. Exploration Bandits (LinUCB / Thompson Sampling): Allocate 5% to 10% of recommendation slots to explore fresh items. The bandit models the upper confidence bound of expected engagement, granting an exploration bonus to items with high variance (few impressions). Once confidence intervals tighten, items graduate to regular ranking funnels.