The right method follows from the training signal you have: preference pairs point to DPO, a program that scores answers points to GRPO, and a learned reward model points to PPO. Nodus runs the first two as managed training runtimes, nodus/dpo and nodus/grpo-lora , each priced by a dry run and stopped at a dollar cap you set. The cost gap between them is mostly about how many models sit in GPU memory and how much text the trainer has to generate.
What does each method actually train on?
Each method learns from a different kind of signal, and that signal is the first thing to check.
PPO (Proximal Policy Optimization) is a general reinforcement learning algorithm. It alternates between sampling from the model and optimizing a clipped surrogate objective. For language models it became the RL step of RLHF.…