diff --git a/README.md b/README.md index 3808760..86ff2d7 100644 --- a/README.md +++ b/README.md @@ -6,6 +6,30 @@ 2. **可解释**：自动按 shape 收集每次运行实际使用的 ttgir（带可读的变体命名），供 IR 级 diff； 3. **口径统一**：cudagraph 计时消除 launch 开销，小 kernel 的对比不被 CPU 侧噪声淹没；capture 前按 warmup 预算显式预热吸收 autotune/JIT，首轮即稳态、多轮一致。 +## 先读这个：口径决定数字的含义 + +同一个算子、同一台机器、同一份代码，换一条测试口径，测出来的数字可以完全不同。 +下表每一行都是本仓踩过的，量级是实测的，不是估计： + +| 口径轴 | 改一下的量级 | 由谁固定 | 由谁检查 | +|---|---|---|---| +| autotune config（record vs replay） | router_gemm 差 8pp | `REPLAY_FROM` | `caliber_complaints`：两侧都 record 直接拒绝出报告 | +| timer（cuda_graph vs direct） | replication_pad3d 从 +43% 变成 +124.5% | `ZL_TIMER` / backend 能力 | 同上，`timer` 字段不一致即拒绝 | +| libtuner 持久缓存（共享 vs 隔离） | mm 看起来从 945 个变体降到 33 个——实际是 B 侧继承了 A 侧的缓存 | `ZL_ISOLATE_TUNER_CACHE=1` | 共享缓存上 record 会打 `警告 [caliber]` | +| `--level core` vs `comprehensive` | mm 的 shape 数 5 → 29，能不能覆盖到大 GEMM 全看这个 | `BENCH_LEVEL` | `bench_level` 字段不一致即拒绝 | +| 重复轮数 | 收益落在零附近时，单次测量给不出方向：topk 42 个 shape 有 17 个在两次等效运行间翻了符号 | `ROUNDS` / `ZL_DIRECT_REPEAT` | `direct_repeat` 字段不一致即拒绝 | +| 被测开关本身是否真的接上 | 开关名拼错，两侧跑同一份代码，脚本照样报出"无差异" | 开关名正确 | `ab_repeat.sh`：第 1 轮后比两侧 ttgir 指纹，相同即跳过该算子 | + +**结论：一次运行的数字没有意义，只有一对同口径运行的差值有意义。** 每次运行把自己的 +口径写进 `runs//run_identity.json`；`batch_summary.py --ab` / `--ab-rounds` 在出 +报告前核对这些字段，不一致就拒绝，因为**口径不同的两次运行会产出一份和有效报告长得 +一模一样的表**——没有任何一步会报错，只是数字换了含义。确认两次运行确实可比时，用 +`--allow-caliber-mismatch` 显式放行。 + +文档拦不住人：README 里早就写着"B 侧必须走 REPLAY_FROM"和"只有 record 数据支撑的 +结论按未验证处理"，本仓仍然发布过一个 record 对 record 得到的 router_gemm 数字。 +所以上表第四列是代码，不是提醒。 + 单算子精测流程之上另有批量筛查层（`run_batch.py`，见「批量多算子测试」），可对双仓库上千个算子做例行体检，产物结构与单算子完全同构。 ## 快速开始 @@ -89,7 +113,9 @@ softmax: ``` runs/softmax_20260716-031752/ ├── run.log # 完整日志（含 SUCCESS 行的 latency/speedup 表；已去 ANSI 色的纯文本） -├── run_identity.json # provider/backend/device/timer/vendor（插件合并写入） +├── run_identity.json # 本次运行的口径：provider/backend/device/timer/vendor/ +│ # bench_level/shape_mode/tuner_cache/direct_repeat/autotune_mode +│ # （shell 写 stub，插件合并写入；对比工具据此判可比性） ├── shapes.yaml # 本次实际使用的 shape（存档） ├── autotune_records/ # Triton autotune 选中的 config（供 replay） │ ├── softmax.json @@ -162,6 +188,27 @@ benchmark 目录，注册 `--level`/`--mode` 的 `conftest.py` 就没被加载 只校验节点的文件那一半；`::test_` 交给 pytest，函数不存在时它自己会清楚地报 `not found: ...`。 +### 口径核对：出报告前的拒绝 + +`batch_summary.py --ab` / `--ab-rounds` 在解析任何 latency 之前，先读参与对比的每个 +run 目录的 `run_identity.json`，逐字段核对 `backend / device / timer / bench_level / +shape_mode / tuner_cache / direct_repeat`。任何一项不一致，或任何一侧缺 identity， +就打 `!!! [caliber] ...` 并以 **4** 退出，不出报告。 + +``` +!!! [caliber] sort: timer 不一致：before=cuda_graph, after=direct +!!! [caliber] sort: 拒绝出报告。确认口径确实可比后加 --allow-caliber-mismatch +``` + +单独判 `autotune_mode`：一侧 record 定 config、另一侧 replay 它是正常形态；**两侧都 +record 直接拒绝**——那正是本仓发布过错误 router_gemm 数字的那次形态。 + +不拒绝但会警告的一类：在**共享** libtuner 缓存上 record。被 pin 的 config 取决于此前 +在这台机器上跑过什么，相对结论仍成立，绝对延迟与变体数不可跨机器引用。 + +`--allow-caliber-mismatch` 放行，仍会把问题打出来。用它之前先能说清为什么这两次运行 +可比；跨口径引用别人的数字，比自己重跑一遍贵得多。 + ### 多轮重复：ab_repeat.sh 与 --ab-rounds 单对观测无法说明自己稳不稳。S5000 实测 `topk` 消除了 32 个 `convert_layout`， @@ -177,6 +224,12 @@ SWITCH=FLAGTREE_MUSA_RLC_ENHANCE OPS="topk masked_select" ROUNDS=6 ZL_BACKEND= `REPLAY_FROM` 那一份，config 因此跨轮跨侧全部锁死；逐轮交替先后顺序抵消单向漂移。 有轮次失败的算子不出报告——缺轮的报告会让人误以为是完整的 N 轮。 +第 1 轮 A/B 跑完后，脚本比两侧 `ttgir/` 的指纹（相对路径 + 内容）。**逐字节相同就 +跳过这个算子**，并打 `!!! SWITCH_NO_IR_DIFF op=... switch=...`：这个开关没改变该算子的 +生成代码，剩下 N-1 轮测的全是噪声，最后还会报成一个"无差异"的结论。先查开关名拼写 +与它在被测源码里的读取点；确属只影响运行时、不影响 IR 的开关，用 +`SWITCH_ALLOW_NO_IR_DIFF=1` 放行。 + 报告分开给两种离散度，它们回答不同的问题： | 指标 | 含义 | diff --git a/ab_repeat.sh b/ab_repeat.sh index 941a24e..f52aefc 100644 --- a/ab_repeat.sh +++ b/ab_repeat.sh @@ -17,8 +17,10 @@ # OPS 元素为 OP 或 OP:OP_FILE（省略时 OP_FILE=OP）。 # SWITCH 是被测环境变量，A 侧设 0、B 侧设 1。 # -# 套用前先确认对比轴有效：grep 该开关名于被测源码应有命中，否则两侧执行同一份 -# 代码，脚本照样跑完并报出一个"无差异"的假结论（同 ab_fold_test.sh 的告诫）。 +# 对比轴有效性由脚本自己查：第 1 轮 A/B 跑完后比两侧的 ttgir 指纹，完全一致说明 +# 这个开关在这个算子上没改变任何生成代码——继续跑满 N 轮只会得到一个"无差异"的假 +# 结论（同 ab_fold_test.sh 的告诫）。这类算子默认跳过；开关确实只影响运行时而不影响 +# IR 时，用 SWITCH_ALLOW_NO_IR_DIFF=1 放行。 # # 产物：runs/ab_repeat_<开关>_<时间戳>//r_/，跑完对每个算子调用 # batch_summary.py --ab-rounds 出报告。 @@ -26,6 +28,21 @@ set -uo pipefail cd "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +SWITCH_ALLOW_NO_IR_DIFF="${SWITCH_ALLOW_NO_IR_DIFF:-0}" + +# 一个 run 生成代码的指纹：ttgir 相对路径 + 内容。路径进指纹是因为"变体集合变了" +# 与"某个变体内容变了"同样算改变，而只 cat 内容会把前者洗掉。 +ir_fingerprint() { + local dir="$1/ttgir" + [[ -d "$dir" ]] || return 1 + local out + out=$( { find "$dir" -type f | LC_ALL=C sort | while IFS= read -r f; do + printf '%s +' "${f#"$dir"/}"; cat "$f" + done; } | sha256sum | cut -d' ' -f1 ) + [[ -n "$out" ]] && printf '%s' "$out" +} + : "${SWITCH:?SWITCH is required, e.g. FLAGTREE_MUSA_RLC_ENHANCE}" OPS="${OPS:-}" : "${OPS:?OPS is required, e.g. \"topk masked_select\"}" @@ -57,7 +74,7 @@ LOG="$ROOT/summary.log" fi REC="$out/r1_A" - ok=1 + ok=1; reason="有失败轮次" for ((k = 1; k <= ROUNDS; k++)); do # 奇数轮先 A 后 B，偶数轮反过来 if (( k % 2 == 1 )); then order="A B"; else order="B A"; fi @@ -71,6 +88,26 @@ LOG="$ROOT/summary.log" ok=0 fi done + + if (( k == 1 && ok )); then + fa=$(ir_fingerprint "$out/r1_A") || fa="" + fb=$(ir_fingerprint "$out/r1_B") || fb="" + if [[ -z "$fa" || -z "$fb" ]]; then + echo "??? $op: 没有 ttgir 产物，跳过对比轴有效性检查" + elif [[ "$fa" == "$fb" ]]; then + echo "!!! SWITCH_NO_IR_DIFF op=$op switch=$SWITCH" + echo "!!! 两侧 ttgir 逐字节相同：这个开关没有改变 $op 的生成代码。" + echo "!!! 继续跑 $ROUNDS 轮只会测出噪声，并报成一个'无差异'结论。" + echo "!!! 先确认开关名拼写与它在被测源码里的读取点；确属只影响运行时的" + echo "!!! 开关，用 SWITCH_ALLOW_NO_IR_DIFF=1 重跑。" + if [[ "$SWITCH_ALLOW_NO_IR_DIFF" != "1" ]]; then + ok=0; reason="开关未改变该算子的生成代码" + break + fi + else + echo ">>> 对比轴有效：$op 两侧 ttgir 不同（A=${fa:0:12} B=${fb:0:12}）" + fi + fi done echo "########## $op | report ##########" @@ -79,7 +116,7 @@ LOG="$ROOT/summary.log" || echo "!!! report failed: $op" else # 缺轮的算子照样出报告会让人误以为是完整的 N 轮 - echo "!!! $op 有失败轮次，不出报告；产物在 $out" + echo "!!! $op $reason，不出报告；产物在 $out" fi done echo "########## ALL DONE ##########" diff --git a/batch_summary.py b/batch_summary.py index 46c0295..550f356 100644 --- a/batch_summary.py +++ b/batch_summary.py @@ -222,8 +222,16 @@ def ab_rows(before_text: str, after_text: str) -> List[Dict[str, str]]: return paired -def report_ab(before: Path, after: Path, label: str = "") -> List[Dict[str, str]]: - """打印一个算子的前后对比：逐行 speedup 与整体区间。""" +def report_ab(before: Path, after: Path, label: str = "", + strict: bool = True) -> List[Dict[str, str]]: + """打印一个算子的前后对比：逐行 speedup 与整体区间。 + + 先核对两侧口径；不一致时默认拒绝出报告，因为不可比的两次运行会产出 + 一份和有效报告长得一模一样的表。 + """ + if not _report_caliber([("before", before.parent), ("after", after.parent)], + label or "ab", strict): + raise CaliberMismatch() before_text = before.read_text(encoding="utf-8", errors="replace") after_text = after.read_text(encoding="utf-8", errors="replace") rows = ab_rows(before_text, after_text) @@ -246,6 +254,109 @@ def report_ab(before: Path, after: Path, label: str = "") -> List[Dict[str, str] return rows +# Fields whose disagreement makes two runs incomparable. Measured magnitudes, +# so the list is not precautionary: an unpinned autotune config moved +# router_gemm by 8pp; the timer moved replication_pad3d from +43% to +124.5%; +# a shared tuner cache made mm look like 945 compiled variants had become 33; +# --level core sees 5 mm shapes where comprehensive sees 29. +CALIBER_FIELDS = ( + "backend", "device", "timer", "bench_level", "shape_mode", + "tuner_cache", "direct_repeat", +) + + +def caliber_of(run_dir: Path) -> Optional[Dict[str, str]]: + """Caliber fields from a run's run_identity.json, or None if unreadable.""" + if not (run_dir / "run_identity.json").is_file(): + return None + try: + from _run_identity import load_identity + ident = load_identity(str(run_dir)) + except Exception: + return None + return {k: str(ident.get(k, "")) for k in CALIBER_FIELDS} | { + "autotune_mode": str(ident.get("autotune_mode", "")) + } + + +def caliber_complaints(runs: List[Tuple[str, Path]]) -> List[str]: + """Everything about this set of runs that makes their comparison unsound. + + Reported rather than silently tolerated because the failure is invisible in + the numbers: two runs on different calibers produce a report that looks + exactly like a valid one. This repository has already published three + figures that were an artifact of one such difference. + """ + out: List[str] = [] + cals = [(name, caliber_of(d)) for name, d in runs] + missing = [n for n, c in cals if c is None] + if missing: + out.append(f"口径不可核实：{', '.join(missing)} 缺 run_identity.json" + "（早于该字段的产物）") + known = [(n, c) for n, c in cals if c is not None] + if len(known) < 2: + return out + + for field in CALIBER_FIELDS: + seen = {c[field] for _, c in known} + if len(seen) > 1: + vals = ", ".join(f"{n}={c[field] or '<空>'}" for n, c in known) + out.append(f"{field} 不一致：{vals}") + + # A/B needs one side to pin the configs the other replays. Two record runs + # each sweep independently, and the winner can differ; the README puts that + # at the 10% level and one-directional per shape, which is enough to swamp + # what is being measured. + modes = [(n, c["autotune_mode"]) for n, c in known] + if all(m.startswith("record") for _, m in modes): + out.append("两侧都是 record 模式：config 未锁定，各自 sweep 可能选出不同 " + "winner。按 README，只有 record 数据支撑的结论按未验证处理") + return out + + +def caliber_hazards(runs: List[Tuple[str, Path]]) -> List[str]: + """Not mismatches — shared state that makes a matched caliber less solid. + + Warned about rather than refused, because refusing would reject the + default configuration every A/B in this repo currently uses. + """ + cals = [(n, c) for n, c in ((n, caliber_of(d)) for n, d in runs) if c] + recording = [n for n, c in cals if c["autotune_mode"].startswith("record")] + shared = [n for n, c in cals if c["tuner_cache"].startswith("shared")] + if recording and shared: + # 一次 record 跑在共享 libtuner cache 上，sweep 的起点是别的运行留下的 + # 结果。曾把 mm 的 945 个变体读成 33 个，误报为编译期收益。A/B 两侧都 + # replay 同一份 config 时结论仍成立，绝对值不成立。 + return [f"{', '.join(recording)} 在共享 tuner cache 上 record：" + "被 pin 的 config 取决于此前跑过什么。相对结论不受影响，" + "绝对延迟/变体数不可跨机器引用（ZL_ISOLATE_TUNER_CACHE=1 可消除）"] + return [] + + +class CaliberMismatch(SystemExit): + """Strict-mode refusal. Exit 4 so a driver script can tell it from a crash.""" + + def __init__(self) -> None: + super().__init__(4) + + +def _report_caliber(runs: List[Tuple[str, Path]], label: str, strict: bool) -> bool: + """Print caliber problems. Returns False when strict mode should stop.""" + for h in caliber_hazards(runs): + print(f"警告 [caliber] {label}: {h}") + problems = caliber_complaints(runs) + if not problems: + return True + head = "!!!" if strict else "警告" + for p in problems: + print(f"{head} [caliber] {label}: {p}") + if strict: + print(f"!!! [caliber] {label}: 拒绝出报告。确认口径确实可比后加 " + "--allow-caliber-mismatch") + return False + return True + + def _lat_index(text: str) -> Dict[Tuple[str, str, int], float]: """(dtype, size_detail, occurrence) -> gems latency, SUCCESS rows only. @@ -326,10 +437,14 @@ def ab_round_rows(root: Path) -> Tuple[List[Dict[str, str]], List[float], int]: return rows, per_round, len(rounds) -def report_ab_rounds(root: Path, label: str = "") -> List[Dict[str, str]]: +def report_ab_rounds(root: Path, label: str = "", + strict: bool = True) -> List[Dict[str, str]]: """Print the multi-round A/B: per-shape medians plus both dispersions.""" - rows, per_round, n = ab_round_rows(root) name = label or root.name + cells = sorted(d for d in root.glob("r*_[AB]") if (d / "run.log").is_file()) + if cells and not _report_caliber([(d.name, d) for d in cells], name, strict): + raise CaliberMismatch() + rows, per_round, n = ab_round_rows(root) if n < 2: print(f"[ab-rounds] {name}: 需要至少 2 轮 /r_/run.log" f"（找到 {n} 轮）") @@ -370,17 +485,22 @@ def main() -> None: help="多轮 A/B 目录，含 r_A/run.log 与 r_B/run.log；" "逐 shape 取轮次中位，并报逐轮区间与符号稳定性") p.add_argument("--label", default="", help="--ab / --ab-rounds 报告里的算子名") + p.add_argument("--allow-caliber-mismatch", action="store_true", + help="口径不一致时仍出报告（默认拒绝）。只在确认两次运行" + "确实可比时使用") args = p.parse_args() if args.ab_rounds: if not args.ab_rounds.is_dir(): raise SystemExit(f"目录不存在: {args.ab_rounds}") - report_ab_rounds(args.ab_rounds, args.label) + report_ab_rounds(args.ab_rounds, args.label, + strict=not args.allow_caliber_mismatch) return if args.ab: for path in args.ab: if not path.is_file(): raise SystemExit(f"run.log 不存在: {path}") - report_ab(args.ab[0], args.ab[1], args.label) + report_ab(args.ab[0], args.ab[1], args.label, + strict=not args.allow_caliber_mismatch) return if args.batch_dir is None: raise SystemExit("需要 batch_dir，或用 --ab BEFORE AFTER") diff --git a/test_backends.py b/test_backends.py index 9af9f51..03e7315 100644 --- a/test_backends.py +++ b/test_backends.py @@ -431,5 +431,115 @@ class AutotuneReplayBackendTests(unittest.TestCase): self.assertFalse(_replay_backend_ok(tmp)) +class CaliberGuardTests(unittest.TestCase): + """两次运行口径不同产出的报告，和有效报告长得一模一样。 + + 这些用例钉住的是本仓已经踩过的三次：record 对 record、共享 tuner cache、 + timer 不同。它们都不会让任何一步报错，只会让数字变成另一个含义。 + """ + + BASE = { + "backend": "musa", "device": "S5000", "timer": "cuda_graph", + "bench_level": "core", "shape_mode": "default", + "tuner_cache": "isolated", "direct_repeat": 5, + "autotune_mode": "replay", + } + + def _run(self, root, name, **over): + d = Path(root, name) + d.mkdir(parents=True) + if over.get("_no_identity"): + return d + ident = dict(self.BASE) + ident.update({k: v for k, v in over.items() if not k.startswith("_")}) + Path(d, "run_identity.json").write_text( + json.dumps(ident), encoding="utf-8") + return d + + def test_same_caliber_has_no_complaints(self): + from batch_summary import caliber_complaints + with tempfile.TemporaryDirectory() as tmp: + runs = [("A", self._run(tmp, "a")), ("B", self._run(tmp, "b"))] + self.assertEqual(caliber_complaints(runs), []) + + def test_record_side_pinning_the_other_is_normal(self): + """A/B 的正常形态：一侧 record 定 config，另一侧 replay 它。""" + from batch_summary import caliber_complaints + with tempfile.TemporaryDirectory() as tmp: + runs = [("r1_A", self._run(tmp, "a", autotune_mode="record")), + ("r1_B", self._run(tmp, "b"))] + self.assertEqual(caliber_complaints(runs), []) + + def test_record_against_record_is_flagged(self): + from batch_summary import caliber_complaints + with tempfile.TemporaryDirectory() as tmp: + runs = [("A", self._run(tmp, "a", autotune_mode="record")), + ("B", self._run(tmp, "b", autotune_mode="record"))] + self.assertTrue(any("record" in c for c in caliber_complaints(runs))) + + def test_each_caliber_field_is_checked(self): + from batch_summary import CALIBER_FIELDS, caliber_complaints + others = {"backend": "cuda", "device": "H20", "timer": "direct", + "bench_level": "comprehensive", "shape_mode": "yaml", + "tuner_cache": "shared", "direct_repeat": 1} + self.assertEqual(set(others), set(CALIBER_FIELDS)) + for field, other in others.items(): + with tempfile.TemporaryDirectory() as tmp: + runs = [("A", self._run(tmp, "a")), + ("B", self._run(tmp, "b", **{field: other}))] + got = caliber_complaints(runs) + self.assertTrue(any(c.startswith(field) for c in got), + f"{field} 不一致时没被拦下: {got}") + + def test_missing_identity_is_flagged_not_assumed_equal(self): + from batch_summary import caliber_complaints + with tempfile.TemporaryDirectory() as tmp: + runs = [("A", self._run(tmp, "a")), + ("B", self._run(tmp, "b", _no_identity=True))] + self.assertTrue(any("run_identity" in c + for c in caliber_complaints(runs))) + + def test_shared_cache_record_warns_but_does_not_block(self): + from batch_summary import caliber_complaints, caliber_hazards + with tempfile.TemporaryDirectory() as tmp: + runs = [("r1_A", self._run(tmp, "a", autotune_mode="record", + tuner_cache="shared(~/.flaggems)")), + ("r1_B", self._run(tmp, "b", + tuner_cache="shared(~/.flaggems)"))] + self.assertEqual(caliber_complaints(runs), []) + self.assertTrue(caliber_hazards(runs)) + + def test_isolated_cache_has_no_hazard(self): + from batch_summary import caliber_hazards + with tempfile.TemporaryDirectory() as tmp: + runs = [("A", self._run(tmp, "a", autotune_mode="record")), + ("B", self._run(tmp, "b"))] + self.assertEqual(caliber_hazards(runs), []) + + def test_strict_refuses_and_permissive_continues(self): + from batch_summary import _report_caliber + with tempfile.TemporaryDirectory() as tmp: + runs = [("A", self._run(tmp, "a")), + ("B", self._run(tmp, "b", timer="direct"))] + self.assertFalse(_report_caliber(runs, "op", strict=True)) + self.assertTrue(_report_caliber(runs, "op", strict=False)) + + LOG = ("Operator: op (dtype=torch.float16, mode=kernel, level=core)\n" + "SUCCESS 1.000 2.000 0.500 shape\n") + + def test_report_ab_refuses_mismatched_pair(self): + from batch_summary import CaliberMismatch, report_ab + with tempfile.TemporaryDirectory() as tmp: + a = self._run(tmp, "a") + b = self._run(tmp, "b", timer="direct") + for d in (a, b): + Path(d, "run.log").write_text(self.LOG, encoding="utf-8") + with self.assertRaises(CaliberMismatch): + report_ab(a / "run.log", b / "run.log", "op") + # 放行开关在，但必须是显式的 + self.assertEqual( + len(report_ab(a / "run.log", b / "run.log", "op", strict=False)), 1) + + if __name__ == "__main__": unittest.main()