bench: refuse to report a comparison whose two sides have different calibers

Two runs on different calibers produce a report that looks exactly like a
valid one — no step errors, the numbers just mean something else. This repo
has published three figures that were an artifact of such a difference:
router_gemm -8.2% (record vs record), mm "945 variants -> 33" (shared vs
inherited libtuner cache), replication_pad3d +43% vs +124.5% (timer).

batch_summary --ab / --ab-rounds now reads every participating run's
run_identity.json and compares backend/device/timer/bench_level/shape_mode/
tuner_cache/direct_repeat before parsing a single latency. Mismatch, or a
missing identity, prints !!! [caliber] and exits 4. autotune_mode is judged
separately: record pinning a replay is the normal shape, both sides record
is the exact trap above and is refused by name. --allow-caliber-mismatch
releases it, still printing what it found.

Recording on a *shared* tuner cache warns rather than refuses: the pinned
config depends on what else ran on the machine, which leaves relative
conclusions intact and absolute latencies unquotable. Refusing it would
reject the default every A/B here currently uses.

ab_repeat.sh closes the one axis that had no check: after round 1 it
fingerprints both sides' ttgir (relative path + content, so a changed
variant *set* counts too). Byte-identical means the switch changed no
generated code for this op, and the remaining N-1 rounds would measure
noise and report it as "no difference" — so the op is skipped with
SWITCH_NO_IR_DIFF unless SWITCH_ALLOW_NO_IR_DIFF=1.

README leads with a caliber table: each axis, its measured magnitude, what
pins it, and which code checks it. The fourth column is code because
documentation did not hold — the README already said "the B side must use
REPLAY_FROM" when the record-vs-record number was published.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

README.md

@@ -6,6 +6,30 @@
 2. **可解释**：自动按 shape 收集每次运行实际使用的 ttgir（带可读的变体命名），供 IR 级 diff；
 3. **口径统一**：cudagraph 计时消除 launch 开销，小 kernel 的对比不被 CPU 侧噪声淹没；capture 前按 warmup 预算显式预热吸收 autotune/JIT，首轮即稳态、多轮一致。
 
+## 先读这个：口径决定数字的含义
+
+同一个算子、同一台机器、同一份代码，换一条测试口径，测出来的数字可以完全不同。
+下表每一行都是本仓踩过的，量级是实测的，不是估计：
+
+| 口径轴 | 改一下的量级 | 由谁固定 | 由谁检查 |
+|---|---|---|---|
+| autotune config（record vs replay） | router_gemm 差 8pp | `REPLAY_FROM` | `caliber_complaints`：两侧都 record 直接拒绝出报告 |
+| timer（cuda_graph vs direct） | replication_pad3d 从 +43% 变成 +124.5% | `ZL_TIMER` / backend 能力 | 同上，`timer` 字段不一致即拒绝 |
+| libtuner 持久缓存（共享 vs 隔离） | mm 看起来从 945 个变体降到 33 个——实际是 B 侧继承了 A 侧的缓存 | `ZL_ISOLATE_TUNER_CACHE=1` | 共享缓存上 record 会打 `警告 [caliber]` |
+| `--level core` vs `comprehensive` | mm 的 shape 数 5 → 29，能不能覆盖到大 GEMM 全看这个 | `BENCH_LEVEL` | `bench_level` 字段不一致即拒绝 |
+| 重复轮数 | 收益落在零附近时，单次测量给不出方向：topk 42 个 shape 有 17 个在两次等效运行间翻了符号 | `ROUNDS` / `ZL_DIRECT_REPEAT` | `direct_repeat` 字段不一致即拒绝 |
+| 被测开关本身是否真的接上 | 开关名拼错，两侧跑同一份代码，脚本照样报出"无差异" | 开关名正确 | `ab_repeat.sh`：第 1 轮后比两侧 ttgir 指纹，相同即跳过该算子 |
+
+**结论：一次运行的数字没有意义，只有一对同口径运行的差值有意义。** 每次运行把自己的
+口径写进 `runs/<run>/run_identity.json`；`batch_summary.py --ab` / `--ab-rounds` 在出
+报告前核对这些字段，不一致就拒绝，因为**口径不同的两次运行会产出一份和有效报告长得
+一模一样的表**——没有任何一步会报错，只是数字换了含义。确认两次运行确实可比时，用
+`--allow-caliber-mismatch` 显式放行。
+
+文档拦不住人：README 里早就写着"B 侧必须走 REPLAY_FROM"和"只有 record 数据支撑的
+结论按未验证处理"，本仓仍然发布过一个 record 对 record 得到的 router_gemm 数字。
+所以上表第四列是代码，不是提醒。
+
 单算子精测流程之上另有批量筛查层（`run_batch.py`，见「批量多算子测试」），可对双仓库上千个算子做例行体检，产物结构与单算子完全同构。
 
 ## 快速开始
@@ -89,7 +113,9 @@ softmax:
 ```
 runs/softmax_20260716-031752/
 ├── run.log                  # 完整日志（含 SUCCESS 行的 latency/speedup 表；已去 ANSI 色的纯文本）
-├── run_identity.json        # provider/backend/device/timer/vendor（插件合并写入）
+├── run_identity.json        # 本次运行的口径：provider/backend/device/timer/vendor/
+│                          # bench_level/shape_mode/tuner_cache/direct_repeat/autotune_mode
+│                          # （shell 写 stub，插件合并写入；对比工具据此判可比性）
 ├── shapes.yaml              # 本次实际使用的 shape（存档）
 ├── autotune_records/        # Triton autotune 选中的 config（供 replay）
 │   ├── softmax.json
@@ -162,6 +188,27 @@ benchmark 目录，注册 `--level`/`--mode` 的 `conftest.py` 就没被加载
 只校验节点的文件那一半；`::test_<fn>` 交给 pytest，函数不存在时它自己会清楚地报
 `not found: ...`。
 
+### 口径核对：出报告前的拒绝
+
+`batch_summary.py --ab` / `--ab-rounds` 在解析任何 latency 之前，先读参与对比的每个
+run 目录的 `run_identity.json`，逐字段核对 `backend / device / timer / bench_level /
+shape_mode / tuner_cache / direct_repeat`。任何一项不一致，或任何一侧缺 identity，
+就打 `!!! [caliber] ...` 并以 **4** 退出，不出报告。
+
+```
+!!! [caliber] sort: timer 不一致：before=cuda_graph, after=direct
+!!! [caliber] sort: 拒绝出报告。确认口径确实可比后加 --allow-caliber-mismatch
+```
+
+单独判 `autotune_mode`：一侧 record 定 config、另一侧 replay 它是正常形态；**两侧都
+record 直接拒绝**——那正是本仓发布过错误 router_gemm 数字的那次形态。
+
+不拒绝但会警告的一类：在**共享** libtuner 缓存上 record。被 pin 的 config 取决于此前
+在这台机器上跑过什么，相对结论仍成立，绝对延迟与变体数不可跨机器引用。
+
+`--allow-caliber-mismatch` 放行，仍会把问题打出来。用它之前先能说清为什么这两次运行
+可比；跨口径引用别人的数字，比自己重跑一遍贵得多。
+
 ### 多轮重复：ab_repeat.sh 与 --ab-rounds
 
 单对观测无法说明自己稳不稳。S5000 实测 `topk` 消除了 32 个 `convert_layout`，
@@ -177,6 +224,12 @@ SWITCH=FLAGTREE_MUSA_RLC_ENHANCE OPS="topk masked_select" ROUNDS=6   ZL_BACKEND=
 `REPLAY_FROM` 那一份，config 因此跨轮跨侧全部锁死；逐轮交替先后顺序抵消单向漂移。
 有轮次失败的算子不出报告——缺轮的报告会让人误以为是完整的 N 轮。
 
+第 1 轮 A/B 跑完后，脚本比两侧 `ttgir/` 的指纹（相对路径 + 内容）。**逐字节相同就
+跳过这个算子**，并打 `!!! SWITCH_NO_IR_DIFF op=... switch=...`：这个开关没改变该算子的
+生成代码，剩下 N-1 轮测的全是噪声，最后还会报成一个"无差异"的结论。先查开关名拼写
+与它在被测源码里的读取点；确属只影响运行时、不影响 IR 的开关，用
+`SWITCH_ALLOW_NO_IR_DIFF=1` 放行。
+
 报告分开给两种离散度，它们回答不同的问题：
 
 | 指标 | 含义 |

ab_repeat.sh

@@ -17,8 +17,10 @@
 #   OPS 元素为 OP 或 OP:OP_FILE（省略时 OP_FILE=OP）。
 #   SWITCH 是被测环境变量，A 侧设 0、B 侧设 1。
 #
-# 套用前先确认对比轴有效：grep 该开关名于被测源码应有命中，否则两侧执行同一份
-# 代码，脚本照样跑完并报出一个"无差异"的假结论（同 ab_fold_test.sh 的告诫）。
+# 对比轴有效性由脚本自己查：第 1 轮 A/B 跑完后比两侧的 ttgir 指纹，完全一致说明
+# 这个开关在这个算子上没改变任何生成代码——继续跑满 N 轮只会得到一个"无差异"的假
+# 结论（同 ab_fold_test.sh 的告诫）。这类算子默认跳过；开关确实只影响运行时而不影响
+# IR 时，用 SWITCH_ALLOW_NO_IR_DIFF=1 放行。
 #
 # 产物：runs/ab_repeat_<开关>_<时间戳>/<op>/r<k>_<A|B>/，跑完对每个算子调用
 # batch_summary.py --ab-rounds 出报告。
@@ -26,6 +28,21 @@ set -uo pipefail
 
 cd "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 
+SWITCH_ALLOW_NO_IR_DIFF="${SWITCH_ALLOW_NO_IR_DIFF:-0}"
+
+# 一个 run 生成代码的指纹：ttgir 相对路径 + 内容。路径进指纹是因为"变体集合变了"
+# 与"某个变体内容变了"同样算改变，而只 cat 内容会把前者洗掉。
+ir_fingerprint() {
+  local dir="$1/ttgir"
+  [[ -d "$dir" ]] || return 1
+  local out
+  out=$( { find "$dir" -type f | LC_ALL=C sort | while IFS= read -r f; do
+             printf '%s
+' "${f#"$dir"/}"; cat "$f"
+           done; } | sha256sum | cut -d' ' -f1 )
+  [[ -n "$out" ]] && printf '%s' "$out"
+}
+
 : "${SWITCH:?SWITCH is required, e.g. FLAGTREE_MUSA_RLC_ENHANCE}"
 OPS="${OPS:-}"
 : "${OPS:?OPS is required, e.g. \"topk masked_select\"}"
@@ -57,7 +74,7 @@ LOG="$ROOT/summary.log"
     fi
     REC="$out/r1_A"
 
-    ok=1
+    ok=1; reason="有失败轮次"
     for ((k = 1; k <= ROUNDS; k++)); do
       # 奇数轮先 A 后 B，偶数轮反过来
       if (( k % 2 == 1 )); then order="A B"; else order="B A"; fi
@@ -71,6 +88,26 @@ LOG="$ROOT/summary.log"
           ok=0
         fi
       done
+
+      if (( k == 1 && ok )); then
+        fa=$(ir_fingerprint "$out/r1_A") || fa=""
+        fb=$(ir_fingerprint "$out/r1_B") || fb=""
+        if [[ -z "$fa" || -z "$fb" ]]; then
+          echo "??? $op: 没有 ttgir 产物，跳过对比轴有效性检查"
+        elif [[ "$fa" == "$fb" ]]; then
+          echo "!!! SWITCH_NO_IR_DIFF op=$op switch=$SWITCH"
+          echo "!!! 两侧 ttgir 逐字节相同：这个开关没有改变 $op 的生成代码。"
+          echo "!!! 继续跑 $ROUNDS 轮只会测出噪声，并报成一个'无差异'结论。"
+          echo "!!! 先确认开关名拼写与它在被测源码里的读取点；确属只影响运行时的"
+          echo "!!! 开关，用 SWITCH_ALLOW_NO_IR_DIFF=1 重跑。"
+          if [[ "$SWITCH_ALLOW_NO_IR_DIFF" != "1" ]]; then
+            ok=0; reason="开关未改变该算子的生成代码"
+            break
+          fi
+        else
+          echo ">>> 对比轴有效：$op 两侧 ttgir 不同（A=${fa:0:12} B=${fb:0:12}）"
+        fi
+      fi
     done
 
     echo "########## $op | report ##########"
@@ -79,7 +116,7 @@ LOG="$ROOT/summary.log"
         || echo "!!! report failed: $op"
     else
       # 缺轮的算子照样出报告会让人误以为是完整的 N 轮
-      echo "!!! $op 有失败轮次，不出报告；产物在 $out"
+      echo "!!! $op $reason，不出报告；产物在 $out"
     fi
   done
   echo "########## ALL DONE ##########"

batch_summary.py

@@ -222,8 +222,16 @@ def ab_rows(before_text: str, after_text: str) -> List[Dict[str, str]]:
     return paired
 
 
-def report_ab(before: Path, after: Path, label: str = "") -> List[Dict[str, str]]:
-    """打印一个算子的前后对比：逐行 speedup 与整体区间。"""
+def report_ab(before: Path, after: Path, label: str = "",
+              strict: bool = True) -> List[Dict[str, str]]:
+    """打印一个算子的前后对比：逐行 speedup 与整体区间。
+
+    先核对两侧口径；不一致时默认拒绝出报告，因为不可比的两次运行会产出
+    一份和有效报告长得一模一样的表。
+    """
+    if not _report_caliber([("before", before.parent), ("after", after.parent)],
+                           label or "ab", strict):
+        raise CaliberMismatch()
     before_text = before.read_text(encoding="utf-8", errors="replace")
     after_text = after.read_text(encoding="utf-8", errors="replace")
     rows = ab_rows(before_text, after_text)
@@ -246,6 +254,109 @@ def report_ab(before: Path, after: Path, label: str = "") -> List[Dict[str, str]
     return rows
 
 
+# Fields whose disagreement makes two runs incomparable. Measured magnitudes,
+# so the list is not precautionary: an unpinned autotune config moved
+# router_gemm by 8pp; the timer moved replication_pad3d from +43% to +124.5%;
+# a shared tuner cache made mm look like 945 compiled variants had become 33;
+# --level core sees 5 mm shapes where comprehensive sees 29.
+CALIBER_FIELDS = (
+    "backend", "device", "timer", "bench_level", "shape_mode",
+    "tuner_cache", "direct_repeat",
+)
+
+
+def caliber_of(run_dir: Path) -> Optional[Dict[str, str]]:
+    """Caliber fields from a run's run_identity.json, or None if unreadable."""
+    if not (run_dir / "run_identity.json").is_file():
+        return None
+    try:
+        from _run_identity import load_identity
+        ident = load_identity(str(run_dir))
+    except Exception:
+        return None
+    return {k: str(ident.get(k, "")) for k in CALIBER_FIELDS} | {
+        "autotune_mode": str(ident.get("autotune_mode", ""))
+    }
+
+
+def caliber_complaints(runs: List[Tuple[str, Path]]) -> List[str]:
+    """Everything about this set of runs that makes their comparison unsound.
+
+    Reported rather than silently tolerated because the failure is invisible in
+    the numbers: two runs on different calibers produce a report that looks
+    exactly like a valid one. This repository has already published three
+    figures that were an artifact of one such difference.
+    """
+    out: List[str] = []
+    cals = [(name, caliber_of(d)) for name, d in runs]
+    missing = [n for n, c in cals if c is None]
+    if missing:
+        out.append(f"口径不可核实：{', '.join(missing)} 缺 run_identity.json"
+                   "（早于该字段的产物）")
+    known = [(n, c) for n, c in cals if c is not None]
+    if len(known) < 2:
+        return out
+
+    for field in CALIBER_FIELDS:
+        seen = {c[field] for _, c in known}
+        if len(seen) > 1:
+            vals = ", ".join(f"{n}={c[field] or '<空>'}" for n, c in known)
+            out.append(f"{field} 不一致：{vals}")
+
+    # A/B needs one side to pin the configs the other replays. Two record runs
+    # each sweep independently, and the winner can differ; the README puts that
+    # at the 10% level and one-directional per shape, which is enough to swamp
+    # what is being measured.
+    modes = [(n, c["autotune_mode"]) for n, c in known]
+    if all(m.startswith("record") for _, m in modes):
+        out.append("两侧都是 record 模式：config 未锁定，各自 sweep 可能选出不同 "
+                   "winner。按 README，只有 record 数据支撑的结论按未验证处理")
+    return out
+
+
+def caliber_hazards(runs: List[Tuple[str, Path]]) -> List[str]:
+    """Not mismatches — shared state that makes a matched caliber less solid.
+
+    Warned about rather than refused, because refusing would reject the
+    default configuration every A/B in this repo currently uses.
+    """
+    cals = [(n, c) for n, c in ((n, caliber_of(d)) for n, d in runs) if c]
+    recording = [n for n, c in cals if c["autotune_mode"].startswith("record")]
+    shared = [n for n, c in cals if c["tuner_cache"].startswith("shared")]
+    if recording and shared:
+        # 一次 record 跑在共享 libtuner cache 上，sweep 的起点是别的运行留下的
+        # 结果。曾把 mm 的 945 个变体读成 33 个，误报为编译期收益。A/B 两侧都
+        # replay 同一份 config 时结论仍成立，绝对值不成立。
+        return [f"{', '.join(recording)} 在共享 tuner cache 上 record："
+                "被 pin 的 config 取决于此前跑过什么。相对结论不受影响，"
+                "绝对延迟/变体数不可跨机器引用（ZL_ISOLATE_TUNER_CACHE=1 可消除）"]
+    return []
+
+
+class CaliberMismatch(SystemExit):
+    """Strict-mode refusal. Exit 4 so a driver script can tell it from a crash."""
+
+    def __init__(self) -> None:
+        super().__init__(4)
+
+
+def _report_caliber(runs: List[Tuple[str, Path]], label: str, strict: bool) -> bool:
+    """Print caliber problems. Returns False when strict mode should stop."""
+    for h in caliber_hazards(runs):
+        print(f"警告 [caliber] {label}: {h}")
+    problems = caliber_complaints(runs)
+    if not problems:
+        return True
+    head = "!!!" if strict else "警告"
+    for p in problems:
+        print(f"{head} [caliber] {label}: {p}")
+    if strict:
+        print(f"!!! [caliber] {label}: 拒绝出报告。确认口径确实可比后加 "
+              "--allow-caliber-mismatch")
+        return False
+    return True
+
+
 def _lat_index(text: str) -> Dict[Tuple[str, str, int], float]:
     """(dtype, size_detail, occurrence) -> gems latency, SUCCESS rows only.
 
@@ -326,10 +437,14 @@ def ab_round_rows(root: Path) -> Tuple[List[Dict[str, str]], List[float], int]:
     return rows, per_round, len(rounds)
 
 
-def report_ab_rounds(root: Path, label: str = "") -> List[Dict[str, str]]:
+def report_ab_rounds(root: Path, label: str = "",
+                     strict: bool = True) -> List[Dict[str, str]]:
     """Print the multi-round A/B: per-shape medians plus both dispersions."""
-    rows, per_round, n = ab_round_rows(root)
     name = label or root.name
+    cells = sorted(d for d in root.glob("r*_[AB]") if (d / "run.log").is_file())
+    if cells and not _report_caliber([(d.name, d) for d in cells], name, strict):
+        raise CaliberMismatch()
+    rows, per_round, n = ab_round_rows(root)
     if n < 2:
         print(f"[ab-rounds] {name}: 需要至少 2 轮 <root>/r<k>_<A|B>/run.log"
               f"（找到 {n} 轮）")
@@ -370,17 +485,22 @@ def main() -> None:
                    help="多轮 A/B 目录，含 r<k>_A/run.log 与 r<k>_B/run.log；"
                         "逐 shape 取轮次中位，并报逐轮区间与符号稳定性")
     p.add_argument("--label", default="", help="--ab / --ab-rounds 报告里的算子名")
+    p.add_argument("--allow-caliber-mismatch", action="store_true",
+                   help="口径不一致时仍出报告（默认拒绝）。只在确认两次运行"
+                        "确实可比时使用")
     args = p.parse_args()
     if args.ab_rounds:
         if not args.ab_rounds.is_dir():
             raise SystemExit(f"目录不存在: {args.ab_rounds}")
-        report_ab_rounds(args.ab_rounds, args.label)
+        report_ab_rounds(args.ab_rounds, args.label,
+                         strict=not args.allow_caliber_mismatch)
         return
     if args.ab:
         for path in args.ab:
             if not path.is_file():
                 raise SystemExit(f"run.log 不存在: {path}")
-        report_ab(args.ab[0], args.ab[1], args.label)
+        report_ab(args.ab[0], args.ab[1], args.label,
+                  strict=not args.allow_caliber_mismatch)
         return
     if args.batch_dir is None:
         raise SystemExit("需要 batch_dir，或用 --ab BEFORE AFTER")

test_backends.py

@@ -431,5 +431,115 @@ class AutotuneReplayBackendTests(unittest.TestCase):
             self.assertFalse(_replay_backend_ok(tmp))
 
 
+class CaliberGuardTests(unittest.TestCase):
+    """两次运行口径不同产出的报告，和有效报告长得一模一样。
+
+    这些用例钉住的是本仓已经踩过的三次：record 对 record、共享 tuner cache、
+    timer 不同。它们都不会让任何一步报错，只会让数字变成另一个含义。
+    """
+
+    BASE = {
+        "backend": "musa", "device": "S5000", "timer": "cuda_graph",
+        "bench_level": "core", "shape_mode": "default",
+        "tuner_cache": "isolated", "direct_repeat": 5,
+        "autotune_mode": "replay",
+    }
+
+    def _run(self, root, name, **over):
+        d = Path(root, name)
+        d.mkdir(parents=True)
+        if over.get("_no_identity"):
+            return d
+        ident = dict(self.BASE)
+        ident.update({k: v for k, v in over.items() if not k.startswith("_")})
+        Path(d, "run_identity.json").write_text(
+            json.dumps(ident), encoding="utf-8")
+        return d
+
+    def test_same_caliber_has_no_complaints(self):
+        from batch_summary import caliber_complaints
+        with tempfile.TemporaryDirectory() as tmp:
+            runs = [("A", self._run(tmp, "a")), ("B", self._run(tmp, "b"))]
+            self.assertEqual(caliber_complaints(runs), [])
+
+    def test_record_side_pinning_the_other_is_normal(self):
+        """A/B 的正常形态：一侧 record 定 config，另一侧 replay 它。"""
+        from batch_summary import caliber_complaints
+        with tempfile.TemporaryDirectory() as tmp:
+            runs = [("r1_A", self._run(tmp, "a", autotune_mode="record")),
+                    ("r1_B", self._run(tmp, "b"))]
+            self.assertEqual(caliber_complaints(runs), [])
+
+    def test_record_against_record_is_flagged(self):
+        from batch_summary import caliber_complaints
+        with tempfile.TemporaryDirectory() as tmp:
+            runs = [("A", self._run(tmp, "a", autotune_mode="record")),
+                    ("B", self._run(tmp, "b", autotune_mode="record"))]
+            self.assertTrue(any("record" in c for c in caliber_complaints(runs)))
+
+    def test_each_caliber_field_is_checked(self):
+        from batch_summary import CALIBER_FIELDS, caliber_complaints
+        others = {"backend": "cuda", "device": "H20", "timer": "direct",
+                  "bench_level": "comprehensive", "shape_mode": "yaml",
+                  "tuner_cache": "shared", "direct_repeat": 1}
+        self.assertEqual(set(others), set(CALIBER_FIELDS))
+        for field, other in others.items():
+            with tempfile.TemporaryDirectory() as tmp:
+                runs = [("A", self._run(tmp, "a")),
+                        ("B", self._run(tmp, "b", **{field: other}))]
+                got = caliber_complaints(runs)
+                self.assertTrue(any(c.startswith(field) for c in got),
+                                f"{field} 不一致时没被拦下: {got}")
+
+    def test_missing_identity_is_flagged_not_assumed_equal(self):
+        from batch_summary import caliber_complaints
+        with tempfile.TemporaryDirectory() as tmp:
+            runs = [("A", self._run(tmp, "a")),
+                    ("B", self._run(tmp, "b", _no_identity=True))]
+            self.assertTrue(any("run_identity" in c
+                                for c in caliber_complaints(runs)))
+
+    def test_shared_cache_record_warns_but_does_not_block(self):
+        from batch_summary import caliber_complaints, caliber_hazards
+        with tempfile.TemporaryDirectory() as tmp:
+            runs = [("r1_A", self._run(tmp, "a", autotune_mode="record",
+                                       tuner_cache="shared(~/.flaggems)")),
+                    ("r1_B", self._run(tmp, "b",
+                                       tuner_cache="shared(~/.flaggems)"))]
+            self.assertEqual(caliber_complaints(runs), [])
+            self.assertTrue(caliber_hazards(runs))
+
+    def test_isolated_cache_has_no_hazard(self):
+        from batch_summary import caliber_hazards
+        with tempfile.TemporaryDirectory() as tmp:
+            runs = [("A", self._run(tmp, "a", autotune_mode="record")),
+                    ("B", self._run(tmp, "b"))]
+            self.assertEqual(caliber_hazards(runs), [])
+
+    def test_strict_refuses_and_permissive_continues(self):
+        from batch_summary import _report_caliber
+        with tempfile.TemporaryDirectory() as tmp:
+            runs = [("A", self._run(tmp, "a")),
+                    ("B", self._run(tmp, "b", timer="direct"))]
+            self.assertFalse(_report_caliber(runs, "op", strict=True))
+            self.assertTrue(_report_caliber(runs, "op", strict=False))
+
+    LOG = ("Operator: op  (dtype=torch.float16, mode=kernel, level=core)\n"
+           "SUCCESS 1.000 2.000 0.500 shape\n")
+
+    def test_report_ab_refuses_mismatched_pair(self):
+        from batch_summary import CaliberMismatch, report_ab
+        with tempfile.TemporaryDirectory() as tmp:
+            a = self._run(tmp, "a")
+            b = self._run(tmp, "b", timer="direct")
+            for d in (a, b):
+                Path(d, "run.log").write_text(self.LOG, encoding="utf-8")
+            with self.assertRaises(CaliberMismatch):
+                report_ab(a / "run.log", b / "run.log", "op")
+            # 放行开关在，但必须是显式的
+            self.assertEqual(
+                len(report_ab(a / "run.log", b / "run.log", "op", strict=False)), 1)
+
+
 if __name__ == "__main__":
     unittest.main()