Use DPO first: it learns from the same preference labels as RLHF but drops the reward model and the online sampling loop, so it is cheaper, faster and easier to debug. On Nodus the whole DPO path runs on managed runtimes ( nodus/sft , then nodus/dpo ), while RLHF adds a nodus/reward-model stage and a PPO loop you run as your own Job. Move to RLHF with PPO when evaluations show you need online exploration, as on math and code, and you can afford the tuning.
What does each pipeline actually involve?
Both pipelines start from the same two ingredients: a supervised fine-tuned (SFT) model and a set of preference pairs, each a prompt with a chosen and a rejected answer. They differ in what happens next.
RLHF, as in InstructGPT , trains a reward model on the pairs, then runs Proximal Policy Optimization (PPO). PPO samples fresh answers from the policy, scores them with the reward model, and updates the policy.…