bench: refuse to report a comparison whose two sides have different calibers

Two runs on different calibers produce a report that looks exactly like a
valid one — no step errors, the numbers just mean something else. This repo
has published three figures that were an artifact of such a difference:
router_gemm -8.2% (record vs record), mm "945 variants -> 33" (shared vs
inherited libtuner cache), replication_pad3d +43% vs +124.5% (timer).

batch_summary --ab / --ab-rounds now reads every participating run's
run_identity.json and compares backend/device/timer/bench_level/shape_mode/
tuner_cache/direct_repeat before parsing a single latency. Mismatch, or a
missing identity, prints !!! [caliber] and exits 4. autotune_mode is judged
separately: record pinning a replay is the normal shape, both sides record
is the exact trap above and is refused by name. --allow-caliber-mismatch
releases it, still printing what it found.

Recording on a *shared* tuner cache warns rather than refuses: the pinned
config depends on what else ran on the machine, which leaves relative
conclusions intact and absolute latencies unquotable. Refusing it would
reject the default every A/B here currently uses.

ab_repeat.sh closes the one axis that had no check: after round 1 it
fingerprints both sides' ttgir (relative path + content, so a changed
variant *set* counts too). Byte-identical means the switch changed no
generated code for this op, and the remaining N-1 rounds would measure
noise and report it as "no difference" — so the op is skipped with
SWITCH_NO_IR_DIFF unless SWITCH_ALLOW_NO_IR_DIFF=1.

README leads with a caliber table: each axis, its measured magnitude, what
pins it, and which code checks it. The fourth column is code because
documentation did not hold — the README already said "the B side must use
REPLAY_FROM" when the record-vs-record number was published.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

ab_repeat.sh

@@ -17,8 +17,10 @@
 #   OPS 元素为 OP 或 OP:OP_FILE（省略时 OP_FILE=OP）。
 #   SWITCH 是被测环境变量，A 侧设 0、B 侧设 1。
 #
-# 套用前先确认对比轴有效：grep 该开关名于被测源码应有命中，否则两侧执行同一份
-# 代码，脚本照样跑完并报出一个"无差异"的假结论（同 ab_fold_test.sh 的告诫）。
+# 对比轴有效性由脚本自己查：第 1 轮 A/B 跑完后比两侧的 ttgir 指纹，完全一致说明
+# 这个开关在这个算子上没改变任何生成代码——继续跑满 N 轮只会得到一个"无差异"的假
+# 结论（同 ab_fold_test.sh 的告诫）。这类算子默认跳过；开关确实只影响运行时而不影响
+# IR 时，用 SWITCH_ALLOW_NO_IR_DIFF=1 放行。
 #
 # 产物：runs/ab_repeat_<开关>_<时间戳>/<op>/r<k>_<A|B>/，跑完对每个算子调用
 # batch_summary.py --ab-rounds 出报告。
@@ -26,6 +28,21 @@ set -uo pipefail
 
 cd "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 
+SWITCH_ALLOW_NO_IR_DIFF="${SWITCH_ALLOW_NO_IR_DIFF:-0}"
+
+# 一个 run 生成代码的指纹：ttgir 相对路径 + 内容。路径进指纹是因为"变体集合变了"
+# 与"某个变体内容变了"同样算改变，而只 cat 内容会把前者洗掉。
+ir_fingerprint() {
+  local dir="$1/ttgir"
+  [[ -d "$dir" ]] || return 1
+  local out
+  out=$( { find "$dir" -type f | LC_ALL=C sort | while IFS= read -r f; do
+             printf '%s
+' "${f#"$dir"/}"; cat "$f"
+           done; } | sha256sum | cut -d' ' -f1 )
+  [[ -n "$out" ]] && printf '%s' "$out"
+}
+
 : "${SWITCH:?SWITCH is required, e.g. FLAGTREE_MUSA_RLC_ENHANCE}"
 OPS="${OPS:-}"
 : "${OPS:?OPS is required, e.g. \"topk masked_select\"}"
@@ -57,7 +74,7 @@ LOG="$ROOT/summary.log"
     fi
     REC="$out/r1_A"
 
-    ok=1
+    ok=1; reason="有失败轮次"
     for ((k = 1; k <= ROUNDS; k++)); do
       # 奇数轮先 A 后 B，偶数轮反过来
       if (( k % 2 == 1 )); then order="A B"; else order="B A"; fi
@@ -71,6 +88,26 @@ LOG="$ROOT/summary.log"
           ok=0
         fi
       done
+
+      if (( k == 1 && ok )); then
+        fa=$(ir_fingerprint "$out/r1_A") || fa=""
+        fb=$(ir_fingerprint "$out/r1_B") || fb=""
+        if [[ -z "$fa" || -z "$fb" ]]; then
+          echo "??? $op: 没有 ttgir 产物，跳过对比轴有效性检查"
+        elif [[ "$fa" == "$fb" ]]; then
+          echo "!!! SWITCH_NO_IR_DIFF op=$op switch=$SWITCH"
+          echo "!!! 两侧 ttgir 逐字节相同：这个开关没有改变 $op 的生成代码。"
+          echo "!!! 继续跑 $ROUNDS 轮只会测出噪声，并报成一个'无差异'结论。"
+          echo "!!! 先确认开关名拼写与它在被测源码里的读取点；确属只影响运行时的"
+          echo "!!! 开关，用 SWITCH_ALLOW_NO_IR_DIFF=1 重跑。"
+          if [[ "$SWITCH_ALLOW_NO_IR_DIFF" != "1" ]]; then
+            ok=0; reason="开关未改变该算子的生成代码"
+            break
+          fi
+        else
+          echo ">>> 对比轴有效：$op 两侧 ttgir 不同（A=${fa:0:12} B=${fb:0:12}）"
+        fi
+      fi
     done
 
     echo "########## $op | report ##########"
@@ -79,7 +116,7 @@ LOG="$ROOT/summary.log"
         || echo "!!! report failed: $op"
     else
       # 缺轮的算子照样出报告会让人误以为是完整的 N 轮
-      echo "!!! $op 有失败轮次，不出报告；产物在 $out"
+      echo "!!! $op $reason，不出报告；产物在 $out"
     fi
   done
   echo "########## ALL DONE ##########"