bench: make screening a first-class step with the same caliber as the A/B

Six rounds of A/B cost 13-44 minutes per operator on S5000 (rand, 15 shapes:
13.0 min; sort, 25 shapes: 43.4 min). Screening one costs about 0.8 minutes —
21 operators in 15.0 minutes, which knocked 13 of them out of the expensive
queue and cut it from ~7 hours to ~2.7. That ratio is the whole argument for
having a separate first step.

Screening was being done by ad-hoc scripts living in the results tree, and
those got two things wrong that this codebase already knows better than:

Both sides ran `record`. Each swept autotune independently, so the two sides
compiled different configs and produced different IR for reasons that have
nothing to do with the switch — every operator would screen as "changed".
screen.sh pins configs exactly as round 1 of ab_repeat does: A records, B
replays A.

The criterion was the convert_layout count. Equal counts do not mean equal
generated code, and what the A/B measures is whether the generated code
changed. The ttgir fingerprint decides; the cvt counts are still recorded,
because they explain what kind of change it was.

Four verdicts, and three of them are not "the switch does nothing":
IR_CHANGED (measure it), NO_IR_DIFF (skip it), NO_KERNEL (the operator never
reached Triton on this backend), NO_ROWS (it did not run — usually the dtype).
The last two are runnability problems being mistaken for switch results, which
is how `fft` got written off: the screen forced DTYPES=float16 on every
operator and fft only accepts complex64. OPS entries now carry an optional
per-operator dtype and default to the operator's own set.

ir_fingerprint moves to _ir_fingerprint.sh so screening and the A/B cannot
drift apart on the criterion.
This commit is contained in:
Aharrypotter committed 2026-09-10 16:49:32 +08:00
1 parent 5236d0745f
commit 2fbfe1dd16
4 files changed
+160 -18

No files matched your search

+1 -18
View File
@@ -30,24 +30,7 @@ cd "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
SWITCH_ALLOW_NO_IR_DIFF="${SWITCH_ALLOW_NO_IR_DIFF:-0}"
# 一个 run 生成代码的指纹:只取 .ttgir 的相对路径 + 内容。
# 路径进指纹是因为"变体集合变了"与"某个变体内容变了"同样算改变,而只 cat 内容
# 会把前者洗掉。只认 .ttgir 则是因为 dump 目录里还有 index.tsv 和 naming.md:
# 一个 kernel 都没编出来的 run 只剩这两个文件、两侧完全相同,会把"什么都没生成"
# 报成"开关没改变生成代码"——router_gemm 在 MUSA 上就是这样。
# 零 kernel 返回非零,交给调用方当作"没有可比的 IR",而不是"一致"。
ir_fingerprint() {
local dir="$1/ttgir"
[[ -d "$dir" ]] || return 1
local n
n=$(find "$dir" -name '*.ttgir' -type f 2>/dev/null | wc -l)
(( n > 0 )) || return 1
local out
out=$( { find "$dir" -name '*.ttgir' -type f | LC_ALL=C sort | while IFS= read -r f; do
printf '%s\n' "${f#"$dir"/}"; cat "$f"
done; } | sha256sum | cut -d' ' -f1 )
[[ -n "$out" ]] && printf '%s' "$out"
}
source "$(dirname "${BASH_SOURCE[0]}")/_ir_fingerprint.sh"
: "${SWITCH:?SWITCH is required, e.g. FLAGTREE_MUSA_RLC_ENHANCE}"
OPS="${OPS:-}"