forked from zhoulin/zl_bench
feat: repeat an A/B over rounds and report whether the verdict holds
A single pair cannot say whether its own verdict is stable, and on this device that is not a theoretical gap. Measured on an S5000, topk eliminated 32 convert_layout ops yet 17 of its 42 shapes changed the SIGN of their delta between two equally valid runs. The cause is not the timer: graph capture and direct timing agree to 1.00x on that operator, and the verdict moved a median of 1.1pp between them, inside the 1.7pp noise floor a same-timer control established. The deltas are simply near zero, and one measurement cannot orient them. ab_repeat.sh runs an A/B over N rounds in independent processes. Round 1's A side records the autotune configs and every later run -- including A's own later rounds -- replays that one set, so configs are pinned across rounds and sides and only the switch under test and measurement noise remain. Round order alternates to cancel one-directional drift. An operator with a failed round gets no report, since a report missing rounds reads exactly like a complete one. batch_summary --ab-rounds aggregates <root>/r<k>_<A|B>/run.log and separates the two dispersions, which answer different questions and were previously conflated: the spread within one side's samples says how noisy the machine was, while the spread of the per-round deltas says whether the verdict holds. Shapes whose delta changes sign across rounds are counted and called out, because their direction is not a conclusion at any timer. The path is already validated on NVIDIA: six rounds narrowed router_gemm to -16.6% ~ -16.2% with 0.4% side noise, which is what turned a single pair into a statement. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
AharrypotterandClaude Opus 5 committed
1 parent
9665d36a7c
commit
aa147c938e
3 files changed
+247
-1
No files matched your search
@@ -0,0 +1,92 @@
|
||||
#!/bin/bash
|
||||
# 多轮 A/B：同一对比在 N 轮独立进程里各跑一次，逐 shape 取轮次中位。
|
||||
#
|
||||
# 为什么需要它：单对观测无法说明自己稳不稳。S5000 实测 topk 消除了 32 个
|
||||
# convert_layout，42 个 shape 里却有 17 个在两次同等有效的运行之间改变了 delta
|
||||
# 的符号——那不是计时口径的问题（图捕获与 direct 的延迟比是 1.00x），是收益本身
|
||||
# 落在零附近、单次测量给不出方向。多轮之后才谈得上"这个判定成立"。
|
||||
#
|
||||
# 口径：第 1 轮 A 侧 record autotune config，此后每一次运行（含 A 侧自己的后续
|
||||
# 轮次）都 REPLAY_FROM 那一份。config 因此跨轮跨侧全部锁死，剩下的变量只有被测
|
||||
# 开关与测量噪声。逐轮交替先后顺序，抵消单向漂移。
|
||||
#
|
||||
# 用法：
|
||||
# SWITCH=FLAGTREE_MUSA_RLC_ENHANCE OPS="topk masked_select" ROUNDS=6 \
|
||||
# ZL_BACKEND=musa FLAGGEMS_DIR=/path/to/FlagGems bash ab_repeat.sh
|
||||
#
|
||||
# OPS 元素为 OP 或 OP:OP_FILE（省略时 OP_FILE=OP）。
|
||||
# SWITCH 是被测环境变量，A 侧设 0、B 侧设 1。
|
||||
#
|
||||
# 套用前先确认对比轴有效：grep 该开关名于被测源码应有命中，否则两侧执行同一份
|
||||
# 代码，脚本照样跑完并报出一个"无差异"的假结论（同 ab_fold_test.sh 的告诫）。
|
||||
#
|
||||
# 产物：runs/ab_repeat_<开关>_<时间戳>/<op>/r<k>_<A|B>/，跑完对每个算子调用
|
||||
# batch_summary.py --ab-rounds 出报告。
|
||||
set -uo pipefail
|
||||
|
||||
cd "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
: "${SWITCH:?SWITCH is required, e.g. FLAGTREE_MUSA_RLC_ENHANCE}"
|
||||
OPS="${OPS:-}"
|
||||
: "${OPS:?OPS is required, e.g. \"topk masked_select\"}"
|
||||
ROUNDS="${ROUNDS:-6}"
|
||||
|
||||
if [[ -t 1 && -z "${NO_COLOR:-}" ]]; then
|
||||
export FLAGGEMS_PERF_COLOR=always
|
||||
fi
|
||||
|
||||
ROOT="runs/ab_repeat_${SWITCH}_$(date +%Y%m%d-%H%M%S)"
|
||||
mkdir -p "$ROOT"
|
||||
LOG="$ROOT/summary.log"
|
||||
|
||||
{
|
||||
echo ">>> switch=$SWITCH rounds=$ROUNDS ops=$OPS"
|
||||
for entry in $OPS; do
|
||||
op="${entry%%:*}"
|
||||
op_file="${entry#*:}"
|
||||
[[ "$op_file" == "$entry" ]] && op_file="$op"
|
||||
out="$ROOT/$op"
|
||||
mkdir -p "$out"
|
||||
|
||||
# 第 1 轮 A 侧 record，其余全部回放它，config 跨轮跨侧一致
|
||||
echo "########## $op | round 1 | A ($SWITCH=0, record) ##########"
|
||||
if ! OP="$op" OP_FILE="$op_file" OUT_DIR="$out/r1_A" \
|
||||
env "$SWITCH=0" bash run_pytest.sh; then
|
||||
echo "!!! FAILED $op round 1 A; skipping this operator"
|
||||
continue
|
||||
fi
|
||||
REC="$out/r1_A"
|
||||
|
||||
ok=1
|
||||
for ((k = 1; k <= ROUNDS; k++)); do
|
||||
# 奇数轮先 A 后 B，偶数轮反过来
|
||||
if (( k % 2 == 1 )); then order="A B"; else order="B A"; fi
|
||||
for side in $order; do
|
||||
[[ $k -eq 1 && $side == A ]] && continue # 已经跑过
|
||||
val=$([[ $side == A ]] && echo 0 || echo 1)
|
||||
echo "########## $op | round $k | $side ($SWITCH=$val, replay) ##########"
|
||||
if ! OP="$op" OP_FILE="$op_file" OUT_DIR="$out/r${k}_${side}" \
|
||||
REPLAY_FROM="$REC" env "$SWITCH=$val" bash run_pytest.sh; then
|
||||
echo "!!! FAILED $op round $k $side"
|
||||
ok=0
|
||||
fi
|
||||
done
|
||||
done
|
||||
|
||||
echo "########## $op | report ##########"
|
||||
if (( ok )); then
|
||||
python batch_summary.py --ab-rounds "$out" --label "$op" \
|
||||
|| echo "!!! report failed: $op"
|
||||
else
|
||||
# 缺轮的算子照样出报告会让人误以为是完整的 N 轮
|
||||
echo "!!! $op 有失败轮次，不出报告；产物在 $out"
|
||||
fi
|
||||
done
|
||||
echo "########## ALL DONE ##########"
|
||||
} 2>&1 | tee "$LOG"
|
||||
|
||||
sed -i -E $'s/\x1b\\[[0-9;]*[A-Za-z]//g' "$LOG"
|
||||
|
||||
echo ">>> 总 log: $(pwd)/$LOG"
|
||||
echo ">>> 逐算子重出报告: python batch_summary.py --ab-rounds $ROOT/<op> --label <op>"
|
||||
echo ">>> 解读要点: 逐轮区间宽说明判定不稳；单侧噪声大说明机器不稳；两者含义不同"
|
||||
Reference in new issue
Block a user