{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 07｜先评估检索，再连接生成模型\n",
    "\n",
    "目标：运行本地 BM25 检索器，检查单条结果，并用标注 query 计算 Hit@k 与 MRR。这个顺序能避免把检索错误误判为模型生成错误。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "import subprocess\n",
    "import sys\n",
    "\n",
    "roots = [Path('llm/downloads'), Path('../')]\n",
    "lab_root = next((root for root in roots if (root / 'code/08_rag_retrieval.py').exists()), None)\n",
    "assert lab_root is not None, '找不到实验脚本；请从实验资料包的 notebooks/ 目录启动 Jupyter'\n",
    "script = lab_root / 'code/08_rag_retrieval.py'\n",
    "documents = lab_root / 'data/rag_documents.jsonl'\n",
    "queries = lab_root / 'data/rag_queries.jsonl'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 观察单条检索\n",
    "\n",
    "预期第一名是 `kv-cache`。分数只在本次索引和查询中有意义，不应跨数据集直接比较。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "single = subprocess.run(\n",
    "    [sys.executable, str(script), '--documents', str(documents), '--query', 'KV Cache 为什么占用内存？', '--top-k', '3'],\n",
    "    check=True, capture_output=True, text=True,\n",
    ")\n",
    "print(single.stdout)\n",
    "assert '1. [kv-cache]' in single.stdout"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 运行离线评估\n",
    "\n",
    "Hit@3 检查正确文档是否进入前三名；MRR 同时奖励更靠前的正确结果。当前小型夹具应达到 1.000，它只是管线测试，不代表真实业务质量。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "evaluation = subprocess.run(\n",
    "    [sys.executable, str(script), '--documents', str(documents), '--queries', str(queries), '--evaluate', '--top-k', '3'],\n",
    "    check=True, capture_output=True, text=True,\n",
    ")\n",
    "print(evaluation.stdout)\n",
    "assert 'hit@3: 1.000' in evaluation.stdout\n",
    "assert 'mrr@3: 1.000' in evaluation.stdout"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 练习与验收\n",
    "\n",
    "- 把 `top_k` 改为 1，比较 Hit@1 与 Hit@3。\n",
    "- 新增两个含义相近的文档，观察排名是否变差。\n",
    "- 为真实业务准备至少 30 条人工标注 query，再决定是否引入 embedding 或 reranker。\n",
    "- 只有检索指标过关后，才把返回文档拼进生成提示，并继续评估答案忠实度。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {"display_name": "Python 3", "language": "python", "name": "python3"},
  "language_info": {"name": "python", "version": "3.12"}
 },
 "nbformat": 4,
 "nbformat_minor": 5
}