{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 08｜从交叉熵到成对评估\n",
    "\n",
    "目标：计算 token 级 NLL/Perplexity，并用 paired bootstrap 估计两个系统在同一题集上的差异区间。数值为教学夹具，不是模型排行榜。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import math\n",
    "import numpy as np\n",
    "\n",
    "token_nll = np.array([1.2, 0.7, 1.0, 0.8, 1.3], dtype=np.float64)\n",
    "mean_nll = float(token_nll.mean())\n",
    "perplexity = math.exp(mean_nll)\n",
    "print(f'mean NLL={mean_nll:.4f}')\n",
    "print(f'perplexity={perplexity:.4f}')\n",
    "assert math.isclose(perplexity, math.exp(mean_nll))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 为什么必须成对比较\n",
    "\n",
    "baseline 和 treatment 必须在同一批样本上评分。对每条样本先求差，再对差值重采样，才能保留题目难度的配对关系。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "baseline = np.array([0, 1, 1, 0, 1, 0, 1, 0, 1, 0], dtype=np.float64)\n",
    "treatment = np.array([1, 1, 1, 0, 1, 1, 1, 0, 1, 0], dtype=np.float64)\n",
    "differences = treatment - baseline\n",
    "rng = np.random.default_rng(42)\n",
    "samples = rng.choice(differences, size=(5000, len(differences)), replace=True).mean(axis=1)\n",
    "low, high = np.quantile(samples, [0.025, 0.975])\n",
    "delta = float(differences.mean())\n",
    "print(f'baseline accuracy={baseline.mean():.3f}')\n",
    "print(f'treatment accuracy={treatment.mean():.3f}')\n",
    "print(f'paired delta={delta:+.3f}, bootstrap 95% interval=[{low:+.3f}, {high:+.3f}]')\n",
    "assert len(baseline) == len(treatment)\n",
    "assert low <= delta <= high"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 验收与边界\n",
    "\n",
    "1. 解释为什么不同 tokenizer 的 perplexity 不宜直接比较。\n",
    "2. 注意当前区间可能包含 0：小样本下不能把正均值直接写成稳定提升。\n",
    "3. 把二元正确率替换成真实题集的逐题分数，保留每条原始记录。\n",
    "4. 生成任务还需增加盲评、格式合规、安全和延迟指标，不能只看 perplexity。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {"display_name": "Python 3", "language": "python", "name": "python3"},
  "language_info": {"name": "python", "version": "3.12"}
 },
 "nbformat": 4,
 "nbformat_minor": 5
}