bench: let ab_rlc_test.sh decide which operators are worth six rounds

zl_bench already had the screening mechanics — ab_rlc_test.sh runs exactly one
config-pinned pair per operator (before records, after replays it) against the
vendor-prefixed RLC switch. I had started a second script for the same job
instead of extending this one; that is removed.

What the existing script could not tell you is whether the pair is worth
following with six rounds, which costs 13-44 minutes per operator against
about 0.8 for the pair. It now writes a verdict per operator to
<root>/screen.tsv, and three of the four are not "the switch does nothing":

  IR_CHANGED  the two sides' ttgir differ         -> measure it
  NO_IR_DIFF  byte-identical, configs pinned      -> skip it
  NO_KERNEL   rows but no .ttgir                  -> never reached Triton here
  NO_ROWS     no measurements                     -> usually the dtype

The criterion is the ttgir fingerprint, not the convert_layout count: equal
counts do not mean equal generated code, and what the A/B measures is whether
the generated code changed. Counts are still recorded — they say what kind of
change it was. The fingerprint comes from _ir_fingerprint.sh, shared with
ab_repeat.sh's round-1 check so the two cannot drift apart.

OPS entries take an optional per-operator dtype and default to the operator's
own set. Forcing one dtype across a list is how fft got written off as
unrunnable on MUSA: it only accepts complex64.

Output moves under runs/ab_rlc_<ts>/<op>_{before,after} with OUT_DIR set
explicitly, rather than recovering the directory with `ls -dt runs/<op>_*`.
That lookup is correct while one process runs operators in series, which is
the only way it has been used; it stops being correct the moment several cards
share runs/.
This commit is contained in:
Aharrypotter committed 2026-09-10 16:54:26 +08:00
1 parent 2fbfe1dd16
commit 4d69320139
3 files changed
+91 -130

No files matched your search

+10 -9
View File
@@ -140,37 +140,38 @@ ttgir 关键设计:
## 两步:先筛选,再实测
跑一个算子的六轮 A/B 要 13~44 分钟(S5000 实测:`rand` 15 shape 13.0 分钟,
`sort` 25 shape 43.4 分钟)。筛一个算子只要约 0.8 分钟(21 个算子一遍 15.0 分钟)。
`sort` 25 shape 43.4 分钟)。用 `ab_rlc_test.sh` 筛一对只要约 0.8 分钟。
**差 20~50 倍**,所以先筛后测:
```bash
# 第一步:筛掉不值得排队的算子
SWITCH=FLAGTREE_MUSA_RLC_ENHANCE ZL_BACKEND=musa FLAGGEMS_DIR=/path/to/FlagGems OPS="col2im:col2im topk:topk perf_fft:fft" bash screen.sh
# 第一步:一对定去留(A 侧 record,B 侧 REPLAY_FROM 它,config 锁死)
OPS="col2im:col2im topk:topk perf_fft:fft" VENDOR=MUSA bash ab_rlc_test.sh
# 第二步:只把 IR_CHANGED 的送进来
SWITCH=FLAGTREE_MUSA_RLC_ENHANCE ROUNDS=6 OPS="topk:topk" bash ab_repeat.sh
```
`screen.sh` 对每个算子给四种判定之一:
`ab_rlc_test.sh` 除了原有的 speedup 区间,还给每个算子写一行判定到
`<root>/screen.tsv`:
| 判定 | 含义 | 该怎么办 |
|---|---|---|
| `IR_CHANGED` | 两侧 ttgir 不同 | 排六轮 A/B |
| `NO_IR_DIFF` | 逐字节相同(config 已锁死) | 跳过。跑六轮只会测到噪声,并报成一个"无差异"结论 |
| `NO_KERNEL` | 跑出了行,但没编出 `.ttgir` | 该算子在本后端没走 Triton,编译器开关无从谈起。**不是"开关无作用"** |
| `NO_ROWS` | 没有测量行 | 先修可运行性:多半是 dtype 不被该算子接受,或该算子在本后端拿不到 config |
| `NO_ROWS` | 没有测量行 | 先修可运行性:多半是 dtype 不被该算子接受,或它在本后端拿不到 config |
三件容易做错的:
两件容易做错的:
- **筛选也是 A/B,口径一样严。** A 侧 record、B 侧 `REPLAY_FROM` 它。两侧各自
record 会各自 sweep 出不同 config,IR 自然不同,于是每个算子都判成
`IR_CHANGED`——筛了等于没筛。
- **判据是 ttgir 指纹,不是 `convert_layout` 计数。** cvt 数相同不等于生成代码
相同,而要测的是"生成代码变了没有"。cvt 数照样记,它解释的是变化的性质。
- **不要全局写死一个 `DTYPES`。** `OPS` 元素支持 `OP:OP_FILE:DTYPES`,留空表示用
该算子自己的全套。`fft` 只吃 complex64,被一刀切的 `DTYPES=float16` 判死过一次
——那是 harness 的问题,被记成了算子的问题。
同一份 `_ir_fingerprint.sh` 也被 `ab_repeat.sh` 用于第 1 轮后的对比轴有效性检查,
两处判据不会漂开。
## A/B 对比测试标准流程
```bash