Originally published on four-leaf.ai .

"Design ChatGPT" is a backend question wearing a model's name. The vLLM paper by Woosuk Kwon and colleagues reports 2-4 times the throughput of state-of-the-art serving systems such as FasterTransformer and Orca at the same latency, mostly by managing the model's key-value cache memory better. That result is a good summary of where this interview is actually scored. The model is a box you are allowed to treat as given. The system around it is the answer.

It is tempting to spend the round on transformers, training data and reinforcement learning from human feedback. In our view, a panel testing a backend or infrastructure engineer usually cares about something else. They want to know how a token gets from a GPU to a browser, what happens when a thousand people hit send at once, and where the conversation history lives.…