devops
Newly open sourced XSE handles six million queries a second across services including Search, Ads, and YouTube Shorts
"Train in Python, serve in C++" might as well be Google's motto when it comes to much of its ML work.
Python is easy for data scientists to wield, but keeping its interpreter in the serving pipeline can add overhead and deployment headaches. The industry-backed OpenXLA Project provides tools for compiling machine learning computations into native code for different processors.
The latest component of this setup to be open sourced is XSE , short for the XLA Serving Engine.
One-shot inference
XSE is not so much suited for large-scale LLM inference (which would be better served by vLLM or SGLang ), but rather for smaller, more sharply task-oriented models, of which there are plenty at Google.
"LLM serving gets most of the attention right now, but a significant amount of production inference is still ranking, recommendation, embedding, and vision models.…