forked from zhoulin/zl_bench
bench: make screening a first-class step with the same caliber as the A/B
Six rounds of A/B cost 13-44 minutes per operator on S5000 (rand, 15 shapes: 13.0 min; sort, 25 shapes: 43.4 min). Screening one costs about 0.8 minutes — 21 operators in 15.0 minutes, which knocked 13 of them out of the expensive queue and cut it from ~7 hours to ~2.7. That ratio is the whole argument for having a separate first step. Screening was being done by ad-hoc scripts living in the results tree, and those got two things wrong that this codebase already knows better than: Both sides ran `record`. Each swept autotune independently, so the two sides compiled different configs and produced different IR for reasons that have nothing to do with the switch — every operator would screen as "changed". screen.sh pins configs exactly as round 1 of ab_repeat does: A records, B replays A. The criterion was the convert_layout count. Equal counts do not mean equal generated code, and what the A/B measures is whether the generated code changed. The ttgir fingerprint decides; the cvt counts are still recorded, because they explain what kind of change it was. Four verdicts, and three of them are not "the switch does nothing": IR_CHANGED (measure it), NO_IR_DIFF (skip it), NO_KERNEL (the operator never reached Triton on this backend), NO_ROWS (it did not run — usually the dtype). The last two are runnability problems being mistaken for switch results, which is how `fft` got written off: the screen forced DTYPES=float16 on every operator and fft only accepts complex64. OPS entries now carry an optional per-operator dtype and default to the operator's own set. ir_fingerprint moves to _ir_fingerprint.sh so screening and the A/B cannot drift apart on the criterion.
This commit is contained in:
1 parent
5236d0745f
commit
2fbfe1dd16
4 files changed
+160
-18
No files matched your search
@@ -137,6 +137,40 @@ ttgir 关键设计:
|
||||
- **命名 = 组内有区分度的参数**:同一 (shape, kernel) 组内取值相同的 constexpr 不进文件名;多词参数缩写为首字母(`BLOCK_SIZE_M` → `BSM`,图例见 naming.md);同名冲突依次用 dtype(`__fp16`)、参数对齐特化(`__EMdiv16`)、hash 前缀消歧。
|
||||
- **崩溃也能拿到 IR**:dump 挂在进程 atexit 上,CUDA crash 后仍会落盘已编译部分。
|
||||
|
||||
## 两步:先筛选,再实测
|
||||
|
||||
跑一个算子的六轮 A/B 要 13~44 分钟(S5000 实测:`rand` 15 shape 13.0 分钟,
|
||||
`sort` 25 shape 43.4 分钟)。筛一个算子只要约 0.8 分钟(21 个算子一遍 15.0 分钟)。
|
||||
**差 20~50 倍**,所以先筛后测:
|
||||
|
||||
```bash
|
||||
# 第一步:筛掉不值得排队的算子
|
||||
SWITCH=FLAGTREE_MUSA_RLC_ENHANCE ZL_BACKEND=musa FLAGGEMS_DIR=/path/to/FlagGems OPS="col2im:col2im topk:topk perf_fft:fft" bash screen.sh
|
||||
|
||||
# 第二步:只把 IR_CHANGED 的送进来
|
||||
SWITCH=FLAGTREE_MUSA_RLC_ENHANCE ROUNDS=6 OPS="topk:topk" bash ab_repeat.sh
|
||||
```
|
||||
|
||||
`screen.sh` 对每个算子给四种判定之一:
|
||||
|
||||
| 判定 | 含义 | 该怎么办 |
|
||||
|---|---|---|
|
||||
| `IR_CHANGED` | 两侧 ttgir 不同 | 排六轮 A/B |
|
||||
| `NO_IR_DIFF` | 逐字节相同(config 已锁死) | 跳过。跑六轮只会测到噪声,并报成一个"无差异"结论 |
|
||||
| `NO_KERNEL` | 跑出了行,但没编出 `.ttgir` | 该算子在本后端没走 Triton,编译器开关无从谈起。**不是"开关无作用"** |
|
||||
| `NO_ROWS` | 没有测量行 | 先修可运行性:多半是 dtype 不被该算子接受,或该算子在本后端拿不到 config |
|
||||
|
||||
三件容易做错的:
|
||||
|
||||
- **筛选也是 A/B,口径一样严。** A 侧 record、B 侧 `REPLAY_FROM` 它。两侧各自
|
||||
record 会各自 sweep 出不同 config,IR 自然不同,于是每个算子都判成
|
||||
`IR_CHANGED`——筛了等于没筛。
|
||||
- **判据是 ttgir 指纹,不是 `convert_layout` 计数。** cvt 数相同不等于生成代码
|
||||
相同,而要测的是"生成代码变了没有"。cvt 数照样记,它解释的是变化的性质。
|
||||
- **不要全局写死一个 `DTYPES`。** `OPS` 元素支持 `OP:OP_FILE:DTYPES`,留空表示用
|
||||
该算子自己的全套。`fft` 只吃 complex64,被一刀切的 `DTYPES=float16` 判死过一次
|
||||
——那是 harness 的问题,被记成了算子的问题。
|
||||
|
||||
## A/B 对比测试标准流程
|
||||
|
||||
```bash
|
||||
|
||||
Reference in new issue
Block a user