forked from zhoulin/zl_bench
bench: make screening a first-class step with the same caliber as the A/B
Six rounds of A/B cost 13-44 minutes per operator on S5000 (rand, 15 shapes: 13.0 min; sort, 25 shapes: 43.4 min). Screening one costs about 0.8 minutes — 21 operators in 15.0 minutes, which knocked 13 of them out of the expensive queue and cut it from ~7 hours to ~2.7. That ratio is the whole argument for having a separate first step. Screening was being done by ad-hoc scripts living in the results tree, and those got two things wrong that this codebase already knows better than: Both sides ran `record`. Each swept autotune independently, so the two sides compiled different configs and produced different IR for reasons that have nothing to do with the switch — every operator would screen as "changed". screen.sh pins configs exactly as round 1 of ab_repeat does: A records, B replays A. The criterion was the convert_layout count. Equal counts do not mean equal generated code, and what the A/B measures is whether the generated code changed. The ttgir fingerprint decides; the cvt counts are still recorded, because they explain what kind of change it was. Four verdicts, and three of them are not "the switch does nothing": IR_CHANGED (measure it), NO_IR_DIFF (skip it), NO_KERNEL (the operator never reached Triton on this backend), NO_ROWS (it did not run — usually the dtype). The last two are runnability problems being mistaken for switch results, which is how `fft` got written off: the screen forced DTYPES=float16 on every operator and fft only accepts complex64. OPS entries now carry an optional per-operator dtype and default to the operator's own set. ir_fingerprint moves to _ir_fingerprint.sh so screening and the A/B cannot drift apart on the criterion.
This commit is contained in:
1 parent
5236d0745f
commit
2fbfe1dd16
4 files changed
+160
-18
No files matched your search
+1
-18
@@ -30,24 +30,7 @@ cd "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
SWITCH_ALLOW_NO_IR_DIFF="${SWITCH_ALLOW_NO_IR_DIFF:-0}"
|
||||
|
||||
# 一个 run 生成代码的指纹:只取 .ttgir 的相对路径 + 内容。
|
||||
# 路径进指纹是因为"变体集合变了"与"某个变体内容变了"同样算改变,而只 cat 内容
|
||||
# 会把前者洗掉。只认 .ttgir 则是因为 dump 目录里还有 index.tsv 和 naming.md:
|
||||
# 一个 kernel 都没编出来的 run 只剩这两个文件、两侧完全相同,会把"什么都没生成"
|
||||
# 报成"开关没改变生成代码"——router_gemm 在 MUSA 上就是这样。
|
||||
# 零 kernel 返回非零,交给调用方当作"没有可比的 IR",而不是"一致"。
|
||||
ir_fingerprint() {
|
||||
local dir="$1/ttgir"
|
||||
[[ -d "$dir" ]] || return 1
|
||||
local n
|
||||
n=$(find "$dir" -name '*.ttgir' -type f 2>/dev/null | wc -l)
|
||||
(( n > 0 )) || return 1
|
||||
local out
|
||||
out=$( { find "$dir" -name '*.ttgir' -type f | LC_ALL=C sort | while IFS= read -r f; do
|
||||
printf '%s\n' "${f#"$dir"/}"; cat "$f"
|
||||
done; } | sha256sum | cut -d' ' -f1 )
|
||||
[[ -n "$out" ]] && printf '%s' "$out"
|
||||
}
|
||||
source "$(dirname "${BASH_SOURCE[0]}")/_ir_fingerprint.sh"
|
||||
|
||||
: "${SWITCH:?SWITCH is required, e.g. FLAGTREE_MUSA_RLC_ENHANCE}"
|
||||
OPS="${OPS:-}"
|
||||
|
||||
Reference in new issue
Block a user