bench: make screening a first-class step with the same caliber as the A/B

Six rounds of A/B cost 13-44 minutes per operator on S5000 (rand, 15 shapes:
13.0 min; sort, 25 shapes: 43.4 min). Screening one costs about 0.8 minutes —
21 operators in 15.0 minutes, which knocked 13 of them out of the expensive
queue and cut it from ~7 hours to ~2.7. That ratio is the whole argument for
having a separate first step.

Screening was being done by ad-hoc scripts living in the results tree, and
those got two things wrong that this codebase already knows better than:

Both sides ran `record`. Each swept autotune independently, so the two sides
compiled different configs and produced different IR for reasons that have
nothing to do with the switch — every operator would screen as "changed".
screen.sh pins configs exactly as round 1 of ab_repeat does: A records, B
replays A.

The criterion was the convert_layout count. Equal counts do not mean equal
generated code, and what the A/B measures is whether the generated code
changed. The ttgir fingerprint decides; the cvt counts are still recorded,
because they explain what kind of change it was.

Four verdicts, and three of them are not "the switch does nothing":
IR_CHANGED (measure it), NO_IR_DIFF (skip it), NO_KERNEL (the operator never
reached Triton on this backend), NO_ROWS (it did not run — usually the dtype).
The last two are runnability problems being mistaken for switch results, which
is how `fft` got written off: the screen forced DTYPES=float16 on every
operator and fft only accepts complex64. OPS entries now carry an optional
per-operator dtype and default to the operator's own set.

ir_fingerprint moves to _ir_fingerprint.sh so screening and the A/B cannot
drift apart on the criterion.
This commit is contained in:
Aharrypotter committed 2026-09-10 16:49:32 +08:00
1 parent 5236d0745f
commit 2fbfe1dd16
4 files changed
+160 -18

No files matched your search

+34
View File
@@ -137,6 +137,40 @@ ttgir 关键设计:
- **命名 = 组内有区分度的参数**:同一 (shape, kernel) 组内取值相同的 constexpr 不进文件名;多词参数缩写为首字母(`BLOCK_SIZE_M` → `BSM`,图例见 naming.md);同名冲突依次用 dtype(`__fp16`)、参数对齐特化(`__EMdiv16`)、hash 前缀消歧。
- **崩溃也能拿到 IR**:dump 挂在进程 atexit 上,CUDA crash 后仍会落盘已编译部分。
## 两步:先筛选,再实测
跑一个算子的六轮 A/B 要 13~44 分钟(S5000 实测:`rand` 15 shape 13.0 分钟,
`sort` 25 shape 43.4 分钟)。筛一个算子只要约 0.8 分钟(21 个算子一遍 15.0 分钟)。
**差 20~50 倍**,所以先筛后测:
```bash
# 第一步:筛掉不值得排队的算子
SWITCH=FLAGTREE_MUSA_RLC_ENHANCE ZL_BACKEND=musa FLAGGEMS_DIR=/path/to/FlagGems OPS="col2im:col2im topk:topk perf_fft:fft" bash screen.sh
# 第二步:只把 IR_CHANGED 的送进来
SWITCH=FLAGTREE_MUSA_RLC_ENHANCE ROUNDS=6 OPS="topk:topk" bash ab_repeat.sh
```
`screen.sh` 对每个算子给四种判定之一:
| 判定 | 含义 | 该怎么办 |
|---|---|---|
| `IR_CHANGED` | 两侧 ttgir 不同 | 排六轮 A/B |
| `NO_IR_DIFF` | 逐字节相同(config 已锁死) | 跳过。跑六轮只会测到噪声,并报成一个"无差异"结论 |
| `NO_KERNEL` | 跑出了行,但没编出 `.ttgir` | 该算子在本后端没走 Triton,编译器开关无从谈起。**不是"开关无作用"** |
| `NO_ROWS` | 没有测量行 | 先修可运行性:多半是 dtype 不被该算子接受,或该算子在本后端拿不到 config |
三件容易做错的:
- **筛选也是 A/B,口径一样严。** A 侧 record、B 侧 `REPLAY_FROM` 它。两侧各自
record 会各自 sweep 出不同 config,IR 自然不同,于是每个算子都判成
`IR_CHANGED`——筛了等于没筛。
- **判据是 ttgir 指纹,不是 `convert_layout` 计数。** cvt 数相同不等于生成代码
相同,而要测的是"生成代码变了没有"。cvt 数照样记,它解释的是变化的性质。
- **不要全局写死一个 `DTYPES`。** `OPS` 元素支持 `OP:OP_FILE:DTYPES`,留空表示用
该算子自己的全套。`fft` 只吃 complex64,被一刀切的 `DTYPES=float16` 判死过一次
——那是 harness 的问题,被记成了算子的问题。
## A/B 对比测试标准流程
```bash