bench: orchestrate a multi-card campaign without touching the measurement path

campaign.sh fans a screened operator list across cards, one ab_repeat.sh per
card, serial within a card. It does no measuring: ab_repeat.sh is the path that
produced the four operator verdicts already on record, and it stays byte for
byte as it was.

Each card gets its own AB_ROOT. Eight ab_repeat.sh started in the same second
would otherwise compute the same timestamped directory name and two cards would
write into one.

ZL_ISOLATE_TUNER_CACHE=1 is forced, since eight processes on one sqlite is the
thing to avoid. The cost is that pinned configs differ from a shared-cache
batch, so absolute latencies cannot be quoted across the two; the caliber guard
refuses such comparisons on its own, which is why this is a note and not a
warning someone has to remember.

partition.py's shell output gained the per-card weight, because EXTRA_OPS has
to land on the lightest card and counting operators gets that wrong — one heavy
operator outweighs three light ones. It writes through stdout.buffer: Python
turns "\n" into CRLF on Windows, and the trailing \r then rides into the last
field and breaks the numeric comparison that picks the card.
This commit is contained in:
Aharrypotter committed 2026-09-10 19:43:37 +08:00
1 parent 03fdcbab28
commit 2786eb9886
5 files changed
+162 -3

No files matched your search

+23
View File
@@ -137,6 +137,29 @@ ttgir 关键设计:
- **命名 = 组内有区分度的参数**:同一 (shape, kernel) 组内取值相同的 constexpr 不进文件名;多词参数缩写为首字母(`BLOCK_SIZE_M` → `BSM`,图例见 naming.md);同名冲突依次用 dtype(`__fp16`)、参数对齐特化(`__EMdiv16`)、hash 前缀消歧。
- **崩溃也能拿到 IR**:dump 挂在进程 atexit 上,CUDA crash 后仍会落盘已编译部分。
## 多卡跑一批:campaign.sh
`ab_repeat.sh` 是单卡串行的。一批算子要并行时用 `campaign.sh` 编排——它只分活,
测量仍然全部由 `ab_repeat.sh` 做,那个脚本一行不动。
```bash
SWITCH=FLAGTREE_MUSA_RLC_ENHANCE DEVICES=0,1,2,3,4,5,6,7 SCREEN=runs/ab_rlc_XXX/screen.tsv ZL_BACKEND=musa FLAGGEMS_DIR=/path/to/FlagGems DRY_RUN=1 bash campaign.sh
```
`DRY_RUN=1` 先看方案,确认了再去掉它起跑。
- **只取筛选表里 `IR_CHANGED` 的算子**——第一步的判定在这里生效
- **按实测耗时贪心分片**(`partition.py`),不是 hash 取模。S5000 上 16 个算子实测:
hash 分片关键路径 120 分钟、有一张卡空转;贪心 70 分钟。**下界由最重的单个算子
决定**(`topk` 一个就 70.6 分钟),加卡不解决,分片只保证别的卡不空等
- **每卡独立 `AB_ROOT`**——八个 `ab_repeat.sh` 同一秒启动,默认的时间戳目录名会撞
- **强制 `ZL_ISOLATE_TUNER_CACHE=1`**。8 个进程写同一个 sqlite 是要避开的事。代价是
pin 的 config 与共享缓存批次不同,**两批的绝对延迟不可互引**——口径守卫会自己拒绝
- `EXTRA_OPS` 追加筛选表里没有的算子(如 benchmark 刚修好的),放到**负载最小**的卡上
- 跑完合并各卡 `status.tsv`(首列是卡号),并统计非零退出的格与三个标记的总数
设备可见性变量名从 backends 层取,不写死 `MUSA_VISIBLE_DEVICES`。
## 两步:先筛选,再实测
跑一个算子的六轮 A/B 要 13~44 分钟(S5000 实测:`rand` 15 shape 13.0 分钟,