Joab Jackson at The Register reports that Google has open sourced XSE, the XLA Serving Engine, a C++ runtime for serving machine learning models without a Python interpreter in the path. It’s the newest piece of the OpenXLA Project, which compiles machine learning math into native code for different processors.
Models are written in Python with Google’s JAX. An export tool packages the computations in StableHLO, a portable intermediate format, along with the trained weights, in what Google calls an XLA SavedModel. The C++ runtime then uses XLA to compile that into native code for the target chip. Google also plans integration with TorchTPU so PyTorch models can be exported for C++ serving on TPUs.
XSE is aimed at small, task-focused models for ranking, recommendation, embedding and vision, not large language model inference. Those requests are short, arrive at very high rates, and have tight latency budgets.…