torchtune.rlhf.loss.SimPOLoss¶
- torchtune.rlhf.loss.SimPOLoss = <function SimPOLoss>[source]¶
Simple Preference Optimization with a Reference-Free Reward: https://arxiv.org/abs/2405.14734. Intuition from the paper:
The effectiveness of SimPO is attributed to a key design: using the average log probability of a sequence as the implicit reward. Additionally, we introduce a target reward margin to the Bradley-Terry objective to encourage a larger margin between the winning and losing responses, further enhancing the algorithm’s performance.
Based on the TRL implementation: https://github.com/huggingface/trl/blob/98ad01ddfd1e1b67ec018014b83cba40e0caea66/trl/trainer/cpo_trainer.py#L603
SimPO is pretty much identitcal to DPO but uses average logprobs to eliminate the need for a reference model to regularize the policy during training. It also uses a target reward margin to guide the policy towards better responses. This is kind of the same intuition as in
IPOLoss, but instead of optimizing against a margin between the reference policy and policy models, we’re optimizing against a margin between the chosen and rejected responses.- Parameters:
beta (float) – Equivalent temperature scaling parameter to DPO loss, typically in the range of 2.0 to 2.5. Default is 2.0.
gamma (float) – Target reward margin hyperparameter, typically we have
gamma in (0, 1.5]. Default is 0.5.label_smoothing (float) – Parameter encoding uncertainty about the labels. Default is 0.
- Type:
SimPO
