forked from zhoulin/zl_bench
bench: let ab_rlc_test.sh decide which operators are worth six rounds
zl_bench already had the screening mechanics — ab_rlc_test.sh runs exactly one
config-pinned pair per operator (before records, after replays it) against the
vendor-prefixed RLC switch. I had started a second script for the same job
instead of extending this one; that is removed.
What the existing script could not tell you is whether the pair is worth
following with six rounds, which costs 13-44 minutes per operator against
about 0.8 for the pair. It now writes a verdict per operator to
<root>/screen.tsv, and three of the four are not "the switch does nothing":
IR_CHANGED the two sides' ttgir differ -> measure it
NO_IR_DIFF byte-identical, configs pinned -> skip it
NO_KERNEL rows but no .ttgir -> never reached Triton here
NO_ROWS no measurements -> usually the dtype
The criterion is the ttgir fingerprint, not the convert_layout count: equal
counts do not mean equal generated code, and what the A/B measures is whether
the generated code changed. Counts are still recorded — they say what kind of
change it was. The fingerprint comes from _ir_fingerprint.sh, shared with
ab_repeat.sh's round-1 check so the two cannot drift apart.
OPS entries take an optional per-operator dtype and default to the operator's
own set. Forcing one dtype across a list is how fft got written off as
unrunnable on MUSA: it only accepts complex64.
Output moves under runs/ab_rlc_<ts>/<op>_{before,after} with OUT_DIR set
explicitly, rather than recovering the directory with `ls -dt runs/<op>_*`.
That lookup is correct while one process runs operators in series, which is
the only way it has been used; it stops being correct the moment several cards
share runs/.
This commit is contained in:
1 parent
2fbfe1dd16
commit
4d69320139
3 files changed
+91
-130
No files matched your search
@@ -140,37 +140,38 @@ ttgir 关键设计:
|
||||
## 两步:先筛选,再实测
|
||||
|
||||
跑一个算子的六轮 A/B 要 13~44 分钟(S5000 实测:`rand` 15 shape 13.0 分钟,
|
||||
`sort` 25 shape 43.4 分钟)。筛一个算子只要约 0.8 分钟(21 个算子一遍 15.0 分钟)。
|
||||
`sort` 25 shape 43.4 分钟)。用 `ab_rlc_test.sh` 筛一对只要约 0.8 分钟。
|
||||
**差 20~50 倍**,所以先筛后测:
|
||||
|
||||
```bash
|
||||
# 第一步:筛掉不值得排队的算子
|
||||
SWITCH=FLAGTREE_MUSA_RLC_ENHANCE ZL_BACKEND=musa FLAGGEMS_DIR=/path/to/FlagGems OPS="col2im:col2im topk:topk perf_fft:fft" bash screen.sh
|
||||
# 第一步:一对定去留(A 侧 record,B 侧 REPLAY_FROM 它,config 锁死)
|
||||
OPS="col2im:col2im topk:topk perf_fft:fft" VENDOR=MUSA bash ab_rlc_test.sh
|
||||
|
||||
# 第二步:只把 IR_CHANGED 的送进来
|
||||
SWITCH=FLAGTREE_MUSA_RLC_ENHANCE ROUNDS=6 OPS="topk:topk" bash ab_repeat.sh
|
||||
```
|
||||
|
||||
`screen.sh` 对每个算子给四种判定之一:
|
||||
`ab_rlc_test.sh` 除了原有的 speedup 区间,还给每个算子写一行判定到
|
||||
`<root>/screen.tsv`:
|
||||
|
||||
| 判定 | 含义 | 该怎么办 |
|
||||
|---|---|---|
|
||||
| `IR_CHANGED` | 两侧 ttgir 不同 | 排六轮 A/B |
|
||||
| `NO_IR_DIFF` | 逐字节相同(config 已锁死) | 跳过。跑六轮只会测到噪声,并报成一个"无差异"结论 |
|
||||
| `NO_KERNEL` | 跑出了行,但没编出 `.ttgir` | 该算子在本后端没走 Triton,编译器开关无从谈起。**不是"开关无作用"** |
|
||||
| `NO_ROWS` | 没有测量行 | 先修可运行性:多半是 dtype 不被该算子接受,或该算子在本后端拿不到 config |
|
||||
| `NO_ROWS` | 没有测量行 | 先修可运行性:多半是 dtype 不被该算子接受,或它在本后端拿不到 config |
|
||||
|
||||
三件容易做错的:
|
||||
两件容易做错的:
|
||||
|
||||
- **筛选也是 A/B,口径一样严。** A 侧 record、B 侧 `REPLAY_FROM` 它。两侧各自
|
||||
record 会各自 sweep 出不同 config,IR 自然不同,于是每个算子都判成
|
||||
`IR_CHANGED`——筛了等于没筛。
|
||||
- **判据是 ttgir 指纹,不是 `convert_layout` 计数。** cvt 数相同不等于生成代码
|
||||
相同,而要测的是"生成代码变了没有"。cvt 数照样记,它解释的是变化的性质。
|
||||
- **不要全局写死一个 `DTYPES`。** `OPS` 元素支持 `OP:OP_FILE:DTYPES`,留空表示用
|
||||
该算子自己的全套。`fft` 只吃 complex64,被一刀切的 `DTYPES=float16` 判死过一次
|
||||
——那是 harness 的问题,被记成了算子的问题。
|
||||
|
||||
同一份 `_ir_fingerprint.sh` 也被 `ab_repeat.sh` 用于第 1 轮后的对比轴有效性检查,
|
||||
两处判据不会漂开。
|
||||
|
||||
## A/B 对比测试标准流程
|
||||
|
||||
```bash
|
||||
|
||||
Reference in new issue
Block a user