forked from zhoulin/zl_bench
feat: absorb the MThreads compat plugin and the A/B range report
Three pieces from the fork's feature/musa-benchmark-support branch that never reached main, rebased onto the backend abstraction rather than copied verbatim. _musa_benchmark_compat_plugin drops FlagGems benchmark skip guards written as `torch.cuda.is_available()` where the code under test dispatches through flag_gems.device. On MThreads those turn a working operator into "not measured": test_fft.py skipped entirely, and unskipped on an S5000 it produces 25 rows. The plugin pins each override to one file and one exact skip reason and only acts when flag_gems has dispatched to the mthreads vendor; two guards were probed on device and deliberately left in place, which is why they are named in its docstring rather than silently absent. It is wired through a new DeviceBackend.compat_plugins rather than a name test in the planner, so the policy stays where every other backend difference lives. Compat plugins are planned first because they rewrite collection-time markers and nothing may read the item list before them; a test pins that position and another pins that they do not leak to other backends. batch_summary gains --ab: the speedup column compares gems against torch, not one build against another, so an A/B needs the two run.logs paired by operator, dtype and size detail. Only rows that succeeded on both sides with a positive latency count, so an incomplete pairing cannot look like a complete range, and the range is annotated when either side carries a BENCHMARK_DIRECT_NO_CUDAGRAPH marker, since a fallback row was timed under a different regime. ab_rlc_test.sh is the same caliber as ab_fold_test.sh with the comparison axis moved to FLAGTREE_<VENDOR>_RLC_ENHANCE. Both are caliber examples, not regression scripts: if the switch under test no longer exists the script still runs every round and prints a full table showing no difference, so confirm the axis is live before reusing either. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
1 parent
c9f9b7885f
commit
29af59be9b
8 files changed
+258
-3
No files matched your search
@@ -0,0 +1,68 @@
|
||||
#!/bin/bash
|
||||
# A/B 对比:FlagTree RemoveLayoutConversions 增强关 vs 开,逐算子两轮。
|
||||
#
|
||||
# 口径:每个算子先跑关(record autotune config),再用 REPLAY_FROM 回放同一套
|
||||
# config 跑开,两侧只差 FLAGTREE_<VENDOR>_RLC_ENHANCE 这一个开关。产物沿用
|
||||
# run_pytest.sh 的目录结构,所以每个算子恰好留下两份 run.log:
|
||||
#
|
||||
# runs/<op>_<时间戳>/run.log 关(before)
|
||||
# runs/<op>_<时间戳>/run.log 开(after)
|
||||
#
|
||||
# 跑完用 batch_summary.py --ab <before> <after> 得到该算子的 speedup 区间。
|
||||
#
|
||||
# 用法:
|
||||
# OPS="softmax gelu_backward" bash ab_rlc_test.sh
|
||||
# OPS="sort:perf_sort topk" bash ab_rlc_test.sh # 冒号分隔 OP:OP_FILE
|
||||
#
|
||||
# 注意:RLC 开关名带厂商前缀,用 VENDOR 指定(默认 MUSA)。套用前先确认开关有效:
|
||||
# grep 该开关名于 FlagTree 源码应有命中,否则两侧跑的是同一份代码。
|
||||
set -uo pipefail
|
||||
|
||||
cd "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
VENDOR="${VENDOR:-MUSA}"
|
||||
ENHANCE_VAR="FLAGTREE_${VENDOR}_RLC_ENHANCE"
|
||||
|
||||
# 空格分隔的算子表。元素为 OP 或 OP:OP_FILE(省略时 OP_FILE=OP)。
|
||||
OPS="${OPS:-softmax}"
|
||||
|
||||
if [[ -t 1 && -z "${NO_COLOR:-}" ]]; then
|
||||
export FLAGGEMS_PERF_COLOR=always
|
||||
fi
|
||||
|
||||
LOG="runs/ab_rlc_$(date +%Y%m%d-%H%M%S).log"
|
||||
mkdir -p runs
|
||||
|
||||
{
|
||||
echo ">>> vendor=$VENDOR switch=$ENHANCE_VAR ops=$OPS"
|
||||
for entry in $OPS; do
|
||||
op="${entry%%:*}"
|
||||
op_file="${entry#*:}"
|
||||
[[ "$op_file" == "$entry" ]] && op_file="$op"
|
||||
|
||||
echo "########## $op | $ENHANCE_VAR=0 before (record) ##########"
|
||||
if ! OP="$op" OP_FILE="$op_file" env "$ENHANCE_VAR=0" bash run_pytest.sh; then
|
||||
echo "!!! FAILED before: $op"
|
||||
continue
|
||||
fi
|
||||
BEFORE=$(ls -dt "runs/${op}_"* | head -1)
|
||||
|
||||
echo "########## $op | $ENHANCE_VAR=1 after (replay $BEFORE) ##########"
|
||||
if ! OP="$op" OP_FILE="$op_file" REPLAY_FROM="$BEFORE" env "$ENHANCE_VAR=1" bash run_pytest.sh; then
|
||||
echo "!!! FAILED after: $op"
|
||||
continue
|
||||
fi
|
||||
AFTER=$(ls -dt "runs/${op}_"* | head -1)
|
||||
|
||||
echo "########## $op | summary ##########"
|
||||
python batch_summary.py --ab "$BEFORE/run.log" "$AFTER/run.log" --label "$op" \
|
||||
|| echo "!!! summary failed: $op"
|
||||
done
|
||||
echo "########## ALL DONE ##########"
|
||||
} 2>&1 | tee "$LOG"
|
||||
|
||||
# 去掉 ANSI 转义以便 grep/diff(终端输出保留颜色)
|
||||
sed -i -E $'s/\x1b\\[[0-9;]*[A-Za-z]//g' "$LOG"
|
||||
|
||||
echo ">>> 总 log: $(pwd)/$LOG"
|
||||
echo ">>> 每算子两份 run.log 在 runs/<op>_<时间戳>/ 下,按时间先后即 before / after"
|
||||
Reference in new issue
Block a user