{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 10｜直接观察 DPO Objective\n",
    "\n",
    "目标：不训练大模型，只使用 chosen/rejected 的序列 log probability，观察 policy margin、reference margin、β 和标签交换如何影响 DPO loss。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn.functional as F\n",
    "\n",
    "policy_chosen = torch.tensor([-4.0, -6.0, -3.5])\n",
    "policy_rejected = torch.tensor([-5.0, -5.5, -4.0])\n",
    "reference_chosen = torch.tensor([-4.5, -5.8, -3.8])\n",
    "reference_rejected = torch.tensor([-4.8, -5.6, -4.1])\n",
    "\n",
    "policy_margin = policy_chosen - policy_rejected\n",
    "reference_margin = reference_chosen - reference_rejected\n",
    "advantage = policy_margin - reference_margin\n",
    "print('policy margin   :', policy_margin.tolist())\n",
    "print('reference margin:', reference_margin.tolist())\n",
    "print('relative margin :', advantage.tolist())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## β 如何改变损失敏感度\n",
    "\n",
    "此处使用 `-log σ(β × relative_margin)`。真实训练还涉及 token mask、序列聚合、batch 和具体实现配置。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def dpo_loss(relative_margin, beta):\n",
    "    return -F.logsigmoid(beta * relative_margin)\n",
    "\n",
    "for beta in (0.1, 0.5, 1.0):\n",
    "    losses = dpo_loss(advantage, beta)\n",
    "    print(f'beta={beta:.1f} losses={losses.tolist()} mean={losses.mean().item():.6f}')\n",
    "assert torch.all(dpo_loss(torch.tensor([1.0]), 1.0) < dpo_loss(torch.tensor([-1.0]), 1.0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 故意交换 chosen/rejected\n",
    "\n",
    "交换标签会让 relative margin 取反。若原样本符合偏好，交换后的平均 loss 应更高；这也是数据方向错误的最小测试。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "beta = 1.0\n",
    "original = dpo_loss(advantage, beta).mean()\n",
    "swapped = dpo_loss(-advantage, beta).mean()\n",
    "print(f'original={original.item():.6f}, swapped={swapped.item():.6f}')\n",
    "assert torch.isfinite(original) and torch.isfinite(swapped)\n",
    "assert not torch.allclose(original, swapped)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 练习与验收\n",
    "\n",
    "1. 构造 policy 与 reference 完全相同的样本，解释 relative margin 和 loss。\n",
    "2. 让 policy 同时提高 chosen 与 rejected 的概率，但相对 margin 不变，观察 DPO 信号。\n",
    "3. 思考序列长度不同导致 log probability 求和偏差时，应怎样审计数据和实现。\n",
    "4. 写出 DPO 训练前必须检查的 chat template、prompt mask、chosen/rejected 方向和独立评估集。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {"display_name": "Python 3", "language": "python", "name": "python3"},
  "language_info": {"name": "python", "version": "3.12"}
 },
 "nbformat": 4,
 "nbformat_minor": 5
}