feat: absorb the MThreads compat plugin and the A/B range report

Three pieces from the fork's feature/musa-benchmark-support branch that never
reached main, rebased onto the backend abstraction rather than copied verbatim.

_musa_benchmark_compat_plugin drops FlagGems benchmark skip guards written as
`torch.cuda.is_available()` where the code under test dispatches through
flag_gems.device. On MThreads those turn a working operator into "not measured":
test_fft.py skipped entirely, and unskipped on an S5000 it produces 25 rows. The
plugin pins each override to one file and one exact skip reason and only acts
when flag_gems has dispatched to the mthreads vendor; two guards were probed on
device and deliberately left in place, which is why they are named in its
docstring rather than silently absent.

It is wired through a new DeviceBackend.compat_plugins rather than a name test
in the planner, so the policy stays where every other backend difference lives.
Compat plugins are planned first because they rewrite collection-time markers
and nothing may read the item list before them; a test pins that position and
another pins that they do not leak to other backends.

batch_summary gains --ab: the speedup column compares gems against torch, not
one build against another, so an A/B needs the two run.logs paired by operator,
dtype and size detail. Only rows that succeeded on both sides with a positive
latency count, so an incomplete pairing cannot look like a complete range, and
the range is annotated when either side carries a BENCHMARK_DIRECT_NO_CUDAGRAPH
marker, since a fallback row was timed under a different regime.

ab_rlc_test.sh is the same caliber as ab_fold_test.sh with the comparison axis
moved to FLAGTREE_<VENDOR>_RLC_ENHANCE. Both are caliber examples, not
regression scripts: if the switch under test no longer exists the script still
runs every round and prints a full table showing no difference, so confirm the
axis is live before reusing either.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
AharrypotterandClaude Opus 5 committed 2026-09-10 10:31:59 +08:00
1 parent c9f9b7885f
commit 29af59be9b
8 files changed
+258 -3

No files matched your search

+68
View File
@@ -0,0 +1,68 @@
#!/bin/bash
# A/B 对比:FlagTree RemoveLayoutConversions 增强关 vs 开,逐算子两轮。
#
# 口径:每个算子先跑关(record autotune config),再用 REPLAY_FROM 回放同一套
# config 跑开,两侧只差 FLAGTREE_<VENDOR>_RLC_ENHANCE 这一个开关。产物沿用
# run_pytest.sh 的目录结构,所以每个算子恰好留下两份 run.log:
#
# runs/<op>_<时间戳>/run.log 关(before)
# runs/<op>_<时间戳>/run.log 开(after)
#
# 跑完用 batch_summary.py --ab <before> <after> 得到该算子的 speedup 区间。
#
# 用法:
# OPS="softmax gelu_backward" bash ab_rlc_test.sh
# OPS="sort:perf_sort topk" bash ab_rlc_test.sh # 冒号分隔 OP:OP_FILE
#
# 注意:RLC 开关名带厂商前缀,用 VENDOR 指定(默认 MUSA)。套用前先确认开关有效:
# grep 该开关名于 FlagTree 源码应有命中,否则两侧跑的是同一份代码。
set -uo pipefail
cd "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
VENDOR="${VENDOR:-MUSA}"
ENHANCE_VAR="FLAGTREE_${VENDOR}_RLC_ENHANCE"
# 空格分隔的算子表。元素为 OP 或 OP:OP_FILE(省略时 OP_FILE=OP)。
OPS="${OPS:-softmax}"
if [[ -t 1 && -z "${NO_COLOR:-}" ]]; then
export FLAGGEMS_PERF_COLOR=always
fi
LOG="runs/ab_rlc_$(date +%Y%m%d-%H%M%S).log"
mkdir -p runs
{
echo ">>> vendor=$VENDOR switch=$ENHANCE_VAR ops=$OPS"
for entry in $OPS; do
op="${entry%%:*}"
op_file="${entry#*:}"
[[ "$op_file" == "$entry" ]] && op_file="$op"
echo "########## $op | $ENHANCE_VAR=0 before (record) ##########"
if ! OP="$op" OP_FILE="$op_file" env "$ENHANCE_VAR=0" bash run_pytest.sh; then
echo "!!! FAILED before: $op"
continue
fi
BEFORE=$(ls -dt "runs/${op}_"* | head -1)
echo "########## $op | $ENHANCE_VAR=1 after (replay $BEFORE) ##########"
if ! OP="$op" OP_FILE="$op_file" REPLAY_FROM="$BEFORE" env "$ENHANCE_VAR=1" bash run_pytest.sh; then
echo "!!! FAILED after: $op"
continue
fi
AFTER=$(ls -dt "runs/${op}_"* | head -1)
echo "########## $op | summary ##########"
python batch_summary.py --ab "$BEFORE/run.log" "$AFTER/run.log" --label "$op" \
|| echo "!!! summary failed: $op"
done
echo "########## ALL DONE ##########"
} 2>&1 | tee "$LOG"
# 去掉 ANSI 转义以便 grep/diff(终端输出保留颜色)
sed -i -E $'s/\x1b\\[[0-9;]*[A-Za-z]//g' "$LOG"
echo ">>> 总 log: $(pwd)/$LOG"
echo ">>> 每算子两份 run.log 在 runs/<op>_<时间戳>/ 下,按时间先后即 before / after"