Recommender systems (RecSys) are one of the most ubiquitous machine learning problems in the consumer internet industry yet notoriously difficult to train and serve at scale. The advent of LLMs has inspired a shift from the traditional embedding-similarity-based objective to a generative one, where the goal is to predict the next action or item in a large catalog given a sequence of user histories.

This post covers the architectural shift toward generative recommenders (GRs), the production challenges it introduces, and how the NVIDIA recsys-examples and nv-embedding-cache address them.

Why traditional RecSys breaks at scale

Data type and volume

User histories, the primary RecSys data type, represent a record of how users interact with items in a catalog. Unlike modalities such as text or images, user histories involve a mix of categorical and continuous features that change frequently over time. At industry scale, this data can get to the order of terabytes or petabytes every day.…