From aa147c938e4f9a64f8ab3b9ab78c68d25368c4c1 Mon Sep 17 00:00:00 2001 From: Aharrypotter Date: Thu, 10 Sep 2026 13:11:37 +0800 Subject: [PATCH] feat: repeat an A/B over rounds and report whether the verdict holds A single pair cannot say whether its own verdict is stable, and on this device that is not a theoretical gap. Measured on an S5000, topk eliminated 32 convert_layout ops yet 17 of its 42 shapes changed the SIGN of their delta between two equally valid runs. The cause is not the timer: graph capture and direct timing agree to 1.00x on that operator, and the verdict moved a median of 1.1pp between them, inside the 1.7pp noise floor a same-timer control established. The deltas are simply near zero, and one measurement cannot orient them. ab_repeat.sh runs an A/B over N rounds in independent processes. Round 1's A side records the autotune configs and every later run -- including A's own later rounds -- replays that one set, so configs are pinned across rounds and sides and only the switch under test and measurement noise remain. Round order alternates to cancel one-directional drift. An operator with a failed round gets no report, since a report missing rounds reads exactly like a complete one. batch_summary --ab-rounds aggregates /r_/run.log and separates the two dispersions, which answer different questions and were previously conflated: the spread within one side's samples says how noisy the machine was, while the spread of the per-round deltas says whether the verdict holds. Shapes whose delta changes sign across rounds are counted and called out, because their direction is not a conclusion at any timer. The path is already validated on NVIDIA: six rounds narrowed router_gemm to -16.6% ~ -16.2% with 0.4% side noise, which is what turned a single pair into a statement. Co-Authored-By: Claude Opus 5 --- README.md | 32 ++++++++++++ ab_repeat.sh | 92 +++++++++++++++++++++++++++++++++++ batch_summary.py | 124 ++++++++++++++++++++++++++++++++++++++++++++++- 3 files changed, 247 insertions(+), 1 deletion(-) create mode 100644 ab_repeat.sh diff --git a/README.md b/README.md index 4da1c0a..3808760 100644 --- a/README.md +++ b/README.md @@ -162,6 +162,38 @@ benchmark 目录，注册 `--level`/`--mode` 的 `conftest.py` 就没被加载 只校验节点的文件那一半；`::test_` 交给 pytest，函数不存在时它自己会清楚地报 `not found: ...`。 +### 多轮重复：ab_repeat.sh 与 --ab-rounds + +单对观测无法说明自己稳不稳。S5000 实测 `topk` 消除了 32 个 `convert_layout`， +42 个 shape 里却有 17 个在两次同等有效的运行之间**改变了 delta 的符号**——那不是 +计时口径问题（图捕获与 direct 的延迟比是 1.00x），是收益本身落在零附近、单次 +测量给不出方向。 + +```bash +SWITCH=FLAGTREE_MUSA_RLC_ENHANCE OPS="topk masked_select" ROUNDS=6 ZL_BACKEND=musa FLAGGEMS_DIR=/path/to/FlagGems bash ab_repeat.sh +``` + +口径：第 1 轮 A 侧 record config，此后每一次运行（含 A 侧的后续轮次）都 +`REPLAY_FROM` 那一份，config 因此跨轮跨侧全部锁死；逐轮交替先后顺序抵消单向漂移。 +有轮次失败的算子不出报告——缺轮的报告会让人误以为是完整的 N 轮。 + +报告分开给两种离散度，它们回答不同的问题： + +| 指标 | 含义 | +|---|---| +| 逐轮区间 | N 轮各自 delta 的 min~max，说明**判定**稳不稳 | +| 单侧噪声 | 同一侧 N 次延迟的极差，说明**机器**吵不吵 | +| 符号不稳行数 | delta 在轮次间改变符号的 shape 数，这些行的方向不可结论 | + +已有产物可以单独重出报告，目录形态是 `/r_/run.log`： + +```bash +python batch_summary.py --ab-rounds runs/ab_repeat_XXX/topk --label topk +``` + +NV 上验证过这条路径：`router_gemm` 六轮把逐轮区间收窄到 -16.6% ~ -16.2%、单侧 +噪声 0.4%，把单对观测升级成了结论。 + ### ab_fold_test.sh / ab_rlc_test.sh：A/B 测试参考示例 `ab_fold_test.sh` 是上述 A/B 流程的现成参考：以 FlagGems `FLAGGEMS_MXFP4_FOLDSCALE`（fold_scale 优化）开关为对比对象，对 4 个 DeepSeek-V4-Flash 真实 trace shape 集（`shape/DeepSeek-V4-Flash-p{1024,4096,32768,65536}d1024.yaml`，随仓库提供）各跑一对 FOLD=0（基线，record）→ FOLD=1（replay 同 config），共 8 轮： diff --git a/ab_repeat.sh b/ab_repeat.sh new file mode 100644 index 0000000..941a24e --- /dev/null +++ b/ab_repeat.sh @@ -0,0 +1,92 @@ +#!/bin/bash +# 多轮 A/B：同一对比在 N 轮独立进程里各跑一次，逐 shape 取轮次中位。 +# +# 为什么需要它：单对观测无法说明自己稳不稳。S5000 实测 topk 消除了 32 个 +# convert_layout，42 个 shape 里却有 17 个在两次同等有效的运行之间改变了 delta +# 的符号——那不是计时口径的问题（图捕获与 direct 的延迟比是 1.00x），是收益本身 +# 落在零附近、单次测量给不出方向。多轮之后才谈得上"这个判定成立"。 +# +# 口径：第 1 轮 A 侧 record autotune config，此后每一次运行（含 A 侧自己的后续 +# 轮次）都 REPLAY_FROM 那一份。config 因此跨轮跨侧全部锁死，剩下的变量只有被测 +# 开关与测量噪声。逐轮交替先后顺序，抵消单向漂移。 +# +# 用法： +# SWITCH=FLAGTREE_MUSA_RLC_ENHANCE OPS="topk masked_select" ROUNDS=6 \ +# ZL_BACKEND=musa FLAGGEMS_DIR=/path/to/FlagGems bash ab_repeat.sh +# +# OPS 元素为 OP 或 OP:OP_FILE（省略时 OP_FILE=OP）。 +# SWITCH 是被测环境变量，A 侧设 0、B 侧设 1。 +# +# 套用前先确认对比轴有效：grep 该开关名于被测源码应有命中，否则两侧执行同一份 +# 代码，脚本照样跑完并报出一个"无差异"的假结论（同 ab_fold_test.sh 的告诫）。 +# +# 产物：runs/ab_repeat_<开关>_<时间戳>//r_/，跑完对每个算子调用 +# batch_summary.py --ab-rounds 出报告。 +set -uo pipefail + +cd "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +: "${SWITCH:?SWITCH is required, e.g. FLAGTREE_MUSA_RLC_ENHANCE}" +OPS="${OPS:-}" +: "${OPS:?OPS is required, e.g. \"topk masked_select\"}" +ROUNDS="${ROUNDS:-6}" + +if [[ -t 1 && -z "${NO_COLOR:-}" ]]; then + export FLAGGEMS_PERF_COLOR=always +fi + +ROOT="runs/ab_repeat_${SWITCH}_$(date +%Y%m%d-%H%M%S)" +mkdir -p "$ROOT" +LOG="$ROOT/summary.log" + +{ + echo ">>> switch=$SWITCH rounds=$ROUNDS ops=$OPS" + for entry in $OPS; do + op="${entry%%:*}" + op_file="${entry#*:}" + [[ "$op_file" == "$entry" ]] && op_file="$op" + out="$ROOT/$op" + mkdir -p "$out" + + # 第 1 轮 A 侧 record，其余全部回放它，config 跨轮跨侧一致 + echo "########## $op | round 1 | A ($SWITCH=0, record) ##########" + if ! OP="$op" OP_FILE="$op_file" OUT_DIR="$out/r1_A" \ + env "$SWITCH=0" bash run_pytest.sh; then + echo "!!! FAILED $op round 1 A; skipping this operator" + continue + fi + REC="$out/r1_A" + + ok=1 + for ((k = 1; k <= ROUNDS; k++)); do + # 奇数轮先 A 后 B，偶数轮反过来 + if (( k % 2 == 1 )); then order="A B"; else order="B A"; fi + for side in $order; do + [[ $k -eq 1 && $side == A ]] && continue # 已经跑过 + val=$([[ $side == A ]] && echo 0 || echo 1) + echo "########## $op | round $k | $side ($SWITCH=$val, replay) ##########" + if ! OP="$op" OP_FILE="$op_file" OUT_DIR="$out/r${k}_${side}" \ + REPLAY_FROM="$REC" env "$SWITCH=$val" bash run_pytest.sh; then + echo "!!! FAILED $op round $k $side" + ok=0 + fi + done + done + + echo "########## $op | report ##########" + if (( ok )); then + python batch_summary.py --ab-rounds "$out" --label "$op" \ + || echo "!!! report failed: $op" + else + # 缺轮的算子照样出报告会让人误以为是完整的 N 轮 + echo "!!! $op 有失败轮次，不出报告；产物在 $out" + fi + done + echo "########## ALL DONE ##########" +} 2>&1 | tee "$LOG" + +sed -i -E $'s/\x1b\\[[0-9;]*[A-Za-z]//g' "$LOG" + +echo ">>> 总 log: $(pwd)/$LOG" +echo ">>> 逐算子重出报告: python batch_summary.py --ab-rounds $ROOT/ --label " +echo ">>> 解读要点: 逐轮区间宽说明判定不稳；单侧噪声大说明机器不稳；两者含义不同" diff --git a/batch_summary.py b/batch_summary.py index 5d2c190..46c0295 100644 --- a/batch_summary.py +++ b/batch_summary.py @@ -16,8 +16,10 @@ from __future__ import annotations import argparse import csv +import itertools import os import re +import statistics import sys from pathlib import Path from typing import Dict, List, Optional, Tuple @@ -244,6 +246,118 @@ def report_ab(before: Path, after: Path, label: str = "") -> List[Dict[str, str] return rows +def _lat_index(text: str) -> Dict[Tuple[str, str, int], float]: + """(dtype, size_detail, occurrence) -> gems latency, SUCCESS rows only. + + Occurrence disambiguates a shape a benchmark measures twice in one table, + which BLAS does at --level comprehensive. + """ + out: Dict[Tuple[str, str, int], float] = {} + seen: Dict[Tuple[str, str], int] = {} + for row in parse_run_log(text): + if row.get("row_status") != "SUCCESS": + continue + try: + lat = float(row.get("gems_lat", "")) + except ValueError: + continue + if lat <= 0: + continue + base = (row.get("dtype", ""), row.get("size_detail", "")) + i = seen.get(base, 0) + seen[base] = i + 1 + out[base + (i,)] = lat + return out + + +def ab_round_rows(root: Path) -> Tuple[List[Dict[str, str]], List[float], int]: + """Aggregate an N-round A/B laid out as /r_/run.log. + + Returns the per-shape rows, the per-round deltas, and the round count. + + A single pair cannot say whether its own verdict is stable, which is the + question that matters once a delta lands near zero: measured on an S5000, + topk eliminated 32 converts yet 17 of its 42 shapes changed the SIGN of + their delta between two equally valid runs. Two dispersions answer different + halves of that and both are reported: + + side spread max/min-1 within one side's samples -> how noisy the machine + was while measuring + round spread min~max of the per-round deltas -> whether the VERDICT holds, + which is what a single pair cannot show + """ + rounds: List[Tuple[Dict, Dict]] = [] + for k in itertools.count(1): + a, b = root / f"r{k}_A" / "run.log", root / f"r{k}_B" / "run.log" + if not (a.is_file() and b.is_file()): + break + rounds.append(( + _lat_index(a.read_text(encoding="utf-8", errors="replace")), + _lat_index(b.read_text(encoding="utf-8", errors="replace")), + )) + if not rounds: + return [], [], 0 + + keys = set(rounds[0][0]) & set(rounds[0][1]) + for a, b in rounds[1:]: + keys &= set(a) & set(b) + + per_round = [ + statistics.median((a[k] / b[k] - 1.0) * 100.0 for k in keys) + for a, b in rounds + ] if keys else [] + + rows: List[Dict[str, str]] = [] + for key in sorted(keys): + av = [a[key] for a, _ in rounds] + bv = [b[key] for _, b in rounds] + am, bm = statistics.median(av), statistics.median(bv) + deltas = [(a / b - 1.0) * 100.0 for a, b in zip(av, bv)] + rows.append({ + "dtype": key[0], "size_detail": key[1], + "before_lat": f"{am:.6f}", "after_lat": f"{bm:.6f}", + "speedup_percent": f"{(am / bm - 1.0) * 100.0:.2f}", + "delta_min": f"{min(deltas):.2f}", "delta_max": f"{max(deltas):.2f}", + "sign_stable": "yes" if all(d > 0 for d in deltas) + or all(d <= 0 for d in deltas) else "NO", + "before_spread": f"{(max(av) / min(av) - 1.0) * 100.0:.2f}", + "after_spread": f"{(max(bv) / min(bv) - 1.0) * 100.0:.2f}", + }) + return rows, per_round, len(rounds) + + +def report_ab_rounds(root: Path, label: str = "") -> List[Dict[str, str]]: + """Print the multi-round A/B: per-shape medians plus both dispersions.""" + rows, per_round, n = ab_round_rows(root) + name = label or root.name + if n < 2: + print(f"[ab-rounds] {name}: 需要至少 2 轮 /r_/run.log" + f"（找到 {n} 轮）") + return rows + if not rows: + print(f"[ab-rounds] {name}: {n} 轮，但没有全轮次都成功的配对行") + return rows + for r in rows: + flag = "" if r["sign_stable"] == "yes" else " 符号不稳" + print(f" {r['dtype']:<16} {float(r['speedup_percent']):+7.2f}% " + f"[{float(r['delta_min']):+.2f}%~{float(r['delta_max']):+.2f}%] " + f"{r['before_lat']} -> {r['after_lat']} " + f"{r['size_detail']}{flag}") + med = [float(r["speedup_percent"]) for r in rows] + spread = [float(r["before_spread"]) for r in rows] + \ + [float(r["after_spread"]) for r in rows] + unstable = sum(1 for r in rows if r["sign_stable"] != "yes") + print(f"[ab-rounds] {name}: {n} 轮 × {len(rows)} shape，" + f"speedup 中位 {statistics.median(med):+.2f}%，" + f"逐轮 {min(per_round):+.2f}% ~ {max(per_round):+.2f}%，" + f"单侧噪声中位 {statistics.median(spread):.2f}%") + if unstable: + # The verdict, not the machine, is what these rows fail to pin down. + print(f"[ab-rounds] {name}: {unstable}/{len(rows)} 个 shape 的 delta " + "在轮次间改变符号——这些行的方向不可结论，与计时口径无关") + return rows + + def main() -> None: p = argparse.ArgumentParser(description=__doc__.splitlines()[0]) p.add_argument("batch_dir", type=Path, nargs="?", @@ -252,8 +366,16 @@ def main() -> None: help="输出 CSV（默认 /summary.csv）") p.add_argument("--ab", nargs=2, type=Path, metavar=("BEFORE", "AFTER"), help="两份 run.log，按 shape 配对输出该算子的 speedup 区间") - p.add_argument("--label", default="", help="--ab 报告里的算子名") + p.add_argument("--ab-rounds", type=Path, default=None, metavar="ROOT", + help="多轮 A/B 目录，含 r_A/run.log 与 r_B/run.log；" + "逐 shape 取轮次中位，并报逐轮区间与符号稳定性") + p.add_argument("--label", default="", help="--ab / --ab-rounds 报告里的算子名") args = p.parse_args() + if args.ab_rounds: + if not args.ab_rounds.is_dir(): + raise SystemExit(f"目录不存在: {args.ab_rounds}") + report_ab_rounds(args.ab_rounds, args.label) + return if args.ab: for path in args.ab: if not path.is_file():