feat: absorb the MThreads compat plugin and the A/B range report

Three pieces from the fork's feature/musa-benchmark-support branch that never
reached main, rebased onto the backend abstraction rather than copied verbatim.

_musa_benchmark_compat_plugin drops FlagGems benchmark skip guards written as
`torch.cuda.is_available()` where the code under test dispatches through
flag_gems.device. On MThreads those turn a working operator into "not measured":
test_fft.py skipped entirely, and unskipped on an S5000 it produces 25 rows. The
plugin pins each override to one file and one exact skip reason and only acts
when flag_gems has dispatched to the mthreads vendor; two guards were probed on
device and deliberately left in place, which is why they are named in its
docstring rather than silently absent.

It is wired through a new DeviceBackend.compat_plugins rather than a name test
in the planner, so the policy stays where every other backend difference lives.
Compat plugins are planned first because they rewrite collection-time markers
and nothing may read the item list before them; a test pins that position and
another pins that they do not leak to other backends.

batch_summary gains --ab: the speedup column compares gems against torch, not
one build against another, so an A/B needs the two run.logs paired by operator,
dtype and size detail. Only rows that succeeded on both sides with a positive
latency count, so an incomplete pairing cannot look like a complete range, and
the range is annotated when either side carries a BENCHMARK_DIRECT_NO_CUDAGRAPH
marker, since a fallback row was timed under a different regime.

ab_rlc_test.sh is the same caliber as ab_fold_test.sh with the comparison axis
moved to FLAGTREE_<VENDOR>_RLC_ENHANCE. Both are caliber examples, not
regression scripts: if the switch under test no longer exists the script still
runs every round and prints a full table showing no difference, so confirm the
axis is live before reusing either.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
AharrypotterandClaude Opus 5 committed 2026-09-10 10:31:59 +08:00
1 parent c9f9b7885f
commit 29af59be9b
8 files changed
+258 -3

No files matched your search

+4 -1
View File
@@ -161,7 +161,7 @@ benchmark 目录,注册 `--level`/`--mode` 的 `conftest.py` 就没被加载
只校验节点的文件那一半;`::test_<fn>` 交给 pytest,函数不存在时它自己会清楚地报
`not found: ...`。
### ab_fold_test.sh:A/B 测试参考示例
### ab_fold_test.sh / ab_rlc_test.sh:A/B 测试参考示例
`ab_fold_test.sh` 是上述 A/B 流程的现成参考:以 FlagGems `FLAGGEMS_MXFP4_FOLDSCALE`(fold_scale 优化)开关为对比对象,对 4 个 DeepSeek-V4-Flash 真实 trace shape 集(`shape/DeepSeek-V4-Flash-p{1024,4096,32768,65536}d1024.yaml`,随仓库提供)各跑一对 FOLD=0(基线,record)→ FOLD=1(replay 同 config),共 8 轮:
@@ -170,6 +170,8 @@ bash ab_fold_test.sh
# 汇总 log:runs/ab_fold_<时间戳>.log(已去色);快速对账:grep -E '^#####|FAILED' <log>
```
`ab_rlc_test.sh` 是同一口径的另一个实例,对比轴换成 FlagTree 的 `FLAGTREE_<VENDOR>_RLC_ENHANCE`(`VENDOR` 默认 `MUSA`),算子表由 `OPS` 指定,跑完调 `batch_summary.py --ab <before> <after>` 出该算子的 speedup 区间。
可复用的是**口径**——同 config replay、cudagraph 计时、真实 trace shape 集、逐对成组——对比其它开关 / 编译器改动 / 算子,改循环变量与环境变量即可,口径部分不用动。
但**它是口径示例,不是可长期直接执行的回归脚本**:对比轴是 FlagGems 侧的算子开关,会随上游演进被改名、改语义或移除,本仓库不跟随同步更新(该脚本引用的开关目前就已被上游移除)。失效后果是**静默的**:脚本照样跑完全部轮次、照样输出完整对比表,只是两侧执行同一份代码,得出"无差异"的假结论。所以套用前先确认对比轴仍有效——`grep -rn '<开关名>' $FLAGGEMS_DIR/src` 要有命中,且两侧 latency 表确有差异。
@@ -268,6 +270,7 @@ python run_batch.py --ops-file my_ops.txt --replay-root runs/batch_xxx
| `_autotune_record_plugin` | record/replay Triton autotune 选择;`autotune_records/meta.json` 记录 backend,跨后端 replay 拒绝 | `ZL_ENABLE_AUTOTUNE=0` |
| `_timing_plugin` | 按后端选择计时路径:CUDA 可走 Graph(内部调用 `_cudagraph_plugin`),其余后端一开始就是上游 `do_bench`(`timer=direct`) | `ZL_ENABLE_CUDAGRAPH=0` 强制 direct |
| `_cudagraph_plugin` | 仅由 `_timing_plugin` 在 CUDA 上调用。`do_bench` → `do_bench_cudagraph`,capture 前按 warmup 预算显式预热;无法 capture 时回退并打 `BENCHMARK_DIRECT_NO_CUDAGRAPH` | 不要直接 `-p` 加到非 CUDA 运行 |
| `_musa_benchmark_compat_plugin` | 剥掉 FlagGems 上游若干 benchmark 上过时的 `torch.cuda.is_available()` 跳过门——被测代码本就经 `flag_gems.device` 分发,在 MThreads 上那些门把可用算子变成「未测量」。每条覆盖都钉死到一个文件与一条确切的 skip 理由,且仅在 `flag_gems.vendor_name == "mthreads"` 时生效 | 由 `backends/musa.py` 的 `compat_plugins` 声明,其他后端不加载 |
| `_pretty_report_plugin` | 结果表整理 + 着色:全表相同的输入折叠成表头下一行图例,每行 Size Detail 只留随行变化的部分(shape 保持 `torch.Size([...])` 原样,MoE 类算子单行从 ~400 字符缩到一屏内);SUCCESS 绿 / FAILED 红。列名与 `SUCCESS` 字样保持上游原文,`run.log` 的 grep/解析不受影响 | 需要与上游原始表格逐字对齐时去掉 |
| `_ir_meta_plugin` | 编译期记录每个变体的 constexpr/签名/特化;launch 钩子统计每个 (kernel, shape) 的实际使用;退出时按上述结构 dump ttgir。Triton knobs 缺失时 fail-soft | `ZL_ENABLE_IR=0` |
| `_mm_cluster_fix_plugin` | Hopper fp16 mm cluster kernel 越界崩溃的运行时规避 | 默认关;`ZL_ENABLE_MM_CLUSTER=1` 或 `ZL_EXTRA_PLUGINS` |