forked from zhoulin/zl_bench
bench: make screening a first-class step with the same caliber as the A/B
Six rounds of A/B cost 13-44 minutes per operator on S5000 (rand, 15 shapes: 13.0 min; sort, 25 shapes: 43.4 min). Screening one costs about 0.8 minutes — 21 operators in 15.0 minutes, which knocked 13 of them out of the expensive queue and cut it from ~7 hours to ~2.7. That ratio is the whole argument for having a separate first step. Screening was being done by ad-hoc scripts living in the results tree, and those got two things wrong that this codebase already knows better than: Both sides ran `record`. Each swept autotune independently, so the two sides compiled different configs and produced different IR for reasons that have nothing to do with the switch — every operator would screen as "changed". screen.sh pins configs exactly as round 1 of ab_repeat does: A records, B replays A. The criterion was the convert_layout count. Equal counts do not mean equal generated code, and what the A/B measures is whether the generated code changed. The ttgir fingerprint decides; the cvt counts are still recorded, because they explain what kind of change it was. Four verdicts, and three of them are not "the switch does nothing": IR_CHANGED (measure it), NO_IR_DIFF (skip it), NO_KERNEL (the operator never reached Triton on this backend), NO_ROWS (it did not run — usually the dtype). The last two are runnability problems being mistaken for switch results, which is how `fft` got written off: the screen forced DTYPES=float16 on every operator and fft only accepts complex64. OPS entries now carry an optional per-operator dtype and default to the operator's own set. ir_fingerprint moves to _ir_fingerprint.sh so screening and the A/B cannot drift apart on the criterion.
This commit is contained in:
Aharrypotter committed
1 parent
5236d0745f
commit
2fbfe1dd16
4 files changed
+160
-18
No files matched your search
@@ -137,6 +137,40 @@ ttgir 关键设计：
|
||||
- **命名 = 组内有区分度的参数**：同一 (shape, kernel) 组内取值相同的 constexpr 不进文件名；多词参数缩写为首字母（`BLOCK_SIZE_M` → `BSM`，图例见 naming.md）；同名冲突依次用 dtype（`__fp16`）、参数对齐特化（`__EMdiv16`）、hash 前缀消歧。
|
||||
- **崩溃也能拿到 IR**：dump 挂在进程 atexit 上，CUDA crash 后仍会落盘已编译部分。
|
||||
|
||||
## 两步：先筛选，再实测
|
||||
|
||||
跑一个算子的六轮 A/B 要 13~44 分钟（S5000 实测：`rand` 15 shape 13.0 分钟，
|
||||
`sort` 25 shape 43.4 分钟）。筛一个算子只要约 0.8 分钟（21 个算子一遍 15.0 分钟）。
|
||||
**差 20~50 倍**，所以先筛后测：
|
||||
|
||||
```bash
|
||||
# 第一步：筛掉不值得排队的算子
|
||||
SWITCH=FLAGTREE_MUSA_RLC_ENHANCE ZL_BACKEND=musa FLAGGEMS_DIR=/path/to/FlagGems OPS="col2im:col2im topk:topk perf_fft:fft" bash screen.sh
|
||||
|
||||
# 第二步：只把 IR_CHANGED 的送进来
|
||||
SWITCH=FLAGTREE_MUSA_RLC_ENHANCE ROUNDS=6 OPS="topk:topk" bash ab_repeat.sh
|
||||
```
|
||||
|
||||
`screen.sh` 对每个算子给四种判定之一：
|
||||
|
||||
| 判定 | 含义 | 该怎么办 |
|
||||
|---|---|---|
|
||||
| `IR_CHANGED` | 两侧 ttgir 不同 | 排六轮 A/B |
|
||||
| `NO_IR_DIFF` | 逐字节相同（config 已锁死） | 跳过。跑六轮只会测到噪声，并报成一个"无差异"结论 |
|
||||
| `NO_KERNEL` | 跑出了行，但没编出 `.ttgir` | 该算子在本后端没走 Triton，编译器开关无从谈起。**不是"开关无作用"** |
|
||||
| `NO_ROWS` | 没有测量行 | 先修可运行性：多半是 dtype 不被该算子接受，或该算子在本后端拿不到 config |
|
||||
|
||||
三件容易做错的：
|
||||
|
||||
- **筛选也是 A/B，口径一样严。** A 侧 record、B 侧 `REPLAY_FROM` 它。两侧各自
|
||||
record 会各自 sweep 出不同 config，IR 自然不同，于是每个算子都判成
|
||||
`IR_CHANGED`——筛了等于没筛。
|
||||
- **判据是 ttgir 指纹，不是 `convert_layout` 计数。** cvt 数相同不等于生成代码
|
||||
相同，而要测的是"生成代码变了没有"。cvt 数照样记，它解释的是变化的性质。
|
||||
- **不要全局写死一个 `DTYPES`。** `OPS` 元素支持 `OP:OP_FILE:DTYPES`，留空表示用
|
||||
该算子自己的全套。`fft` 只吃 complex64，被一刀切的 `DTYPES=float16` 判死过一次
|
||||
——那是 harness 的问题，被记成了算子的问题。
|
||||
|
||||
## A/B 对比测试标准流程
|
||||
|
||||
```bash
|
||||
|
||||
@@ -0,0 +1,22 @@
|
||||
#!/bin/bash
|
||||
# 由 ab_repeat.sh 与 screen.sh 共用；两处必须用同一个判据，否则筛选放行的算子
|
||||
# 到了 A/B 那步又被拦下，或者反过来。
|
||||
|
||||
# 一个 run 生成代码的指纹：只取 .ttgir 的相对路径 + 内容。
|
||||
# 路径进指纹是因为"变体集合变了"与"某个变体内容变了"同样算改变，而只 cat 内容
|
||||
# 会把前者洗掉。只认 .ttgir 则是因为 dump 目录里还有 index.tsv 和 naming.md：
|
||||
# 一个 kernel 都没编出来的 run 只剩这两个文件、两侧完全相同，会把"什么都没生成"
|
||||
# 报成"开关没改变生成代码"——router_gemm 在 MUSA 上就是这样。
|
||||
# 零 kernel 返回非零，交给调用方当作"没有可比的 IR"，而不是"一致"。
|
||||
ir_fingerprint() {
|
||||
local dir="$1/ttgir"
|
||||
[[ -d "$dir" ]] || return 1
|
||||
local n
|
||||
n=$(find "$dir" -name '*.ttgir' -type f 2>/dev/null | wc -l)
|
||||
(( n > 0 )) || return 1
|
||||
local out
|
||||
out=$( { find "$dir" -name '*.ttgir' -type f | LC_ALL=C sort | while IFS= read -r f; do
|
||||
printf '%s\n' "${f#"$dir"/}"; cat "$f"
|
||||
done; } | sha256sum | cut -d' ' -f1 )
|
||||
[[ -n "$out" ]] && printf '%s' "$out"
|
||||
}
|
||||
+1
-18
@@ -30,24 +30,7 @@ cd "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
SWITCH_ALLOW_NO_IR_DIFF="${SWITCH_ALLOW_NO_IR_DIFF:-0}"
|
||||
|
||||
# 一个 run 生成代码的指纹：只取 .ttgir 的相对路径 + 内容。
|
||||
# 路径进指纹是因为"变体集合变了"与"某个变体内容变了"同样算改变，而只 cat 内容
|
||||
# 会把前者洗掉。只认 .ttgir 则是因为 dump 目录里还有 index.tsv 和 naming.md：
|
||||
# 一个 kernel 都没编出来的 run 只剩这两个文件、两侧完全相同，会把"什么都没生成"
|
||||
# 报成"开关没改变生成代码"——router_gemm 在 MUSA 上就是这样。
|
||||
# 零 kernel 返回非零，交给调用方当作"没有可比的 IR"，而不是"一致"。
|
||||
ir_fingerprint() {
|
||||
local dir="$1/ttgir"
|
||||
[[ -d "$dir" ]] || return 1
|
||||
local n
|
||||
n=$(find "$dir" -name '*.ttgir' -type f 2>/dev/null | wc -l)
|
||||
(( n > 0 )) || return 1
|
||||
local out
|
||||
out=$( { find "$dir" -name '*.ttgir' -type f | LC_ALL=C sort | while IFS= read -r f; do
|
||||
printf '%s\n' "${f#"$dir"/}"; cat "$f"
|
||||
done; } | sha256sum | cut -d' ' -f1 )
|
||||
[[ -n "$out" ]] && printf '%s' "$out"
|
||||
}
|
||||
source "$(dirname "${BASH_SOURCE[0]}")/_ir_fingerprint.sh"
|
||||
|
||||
: "${SWITCH:?SWITCH is required, e.g. FLAGTREE_MUSA_RLC_ENHANCE}"
|
||||
OPS="${OPS:-}"
|
||||
|
||||
@@ -0,0 +1,103 @@
|
||||
#!/bin/bash
|
||||
# 第一步：筛掉不值得跑 A/B 的算子。
|
||||
#
|
||||
# 为什么要单独一步：S5000 实测，筛选一个算子约 0.8 分钟，六轮 A/B 一个算子
|
||||
# 13~44 分钟——差 20~50 倍。21 个算子筛一遍 15 分钟，挡掉 13 个，把昂贵队列
|
||||
# 从 7 小时压到 2.7 小时。筛选的全部价值就在这个比例上。
|
||||
#
|
||||
# 口径与 A/B 完全一致，因为筛选也是一次 A/B：A 侧 record，B 侧 REPLAY_FROM 它。
|
||||
# 两侧各自 record 是不行的——各自 sweep 会选出不同 config，生成的 IR 自然不同，
|
||||
# 于是每个算子都被判成"开关改变了生成代码"，筛选等于没筛。这条本仓踩过。
|
||||
#
|
||||
# 判据是 ttgir 指纹，不是 convert_layout 计数。cvt 数相同不等于生成代码相同，
|
||||
# 而 A/B 要测的是"生成代码变了没有"。cvt 数照样记下来，它解释的是变化的性质。
|
||||
#
|
||||
# 用法：
|
||||
# SWITCH=FLAGTREE_MUSA_RLC_ENHANCE OPS="col2im:col2im topk:topk perf_fft:fft" \
|
||||
# ZL_BACKEND=musa FLAGGEMS_DIR=/path/to/FlagGems bash screen.sh
|
||||
#
|
||||
# OPS 元素为 OP[:OP_FILE[:DTYPES]]，省略 OP_FILE 时等于 OP。
|
||||
# DTYPES 留空表示用该算子自己的全套——**不要全局写死一个 dtype**：
|
||||
# fft 只吃 complex64，被一刀切的 DTYPES=float16 直接判死过一次。
|
||||
#
|
||||
# 产物：runs/screen_<开关>_<时间戳>/，含 screen.tsv 与逐算子 run 目录。
|
||||
set -uo pipefail
|
||||
|
||||
cd "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
source "./_ir_fingerprint.sh"
|
||||
|
||||
: "${SWITCH:?SWITCH is required, e.g. FLAGTREE_MUSA_RLC_ENHANCE}"
|
||||
OPS="${OPS:-}"
|
||||
: "${OPS:?OPS is required, e.g. \"topk:topk perf_fft:fft\"}"
|
||||
CELL_TIMEOUT="${CELL_TIMEOUT:-2400}"
|
||||
|
||||
ROOT="runs/screen_${SWITCH}_$(date +%Y%m%d-%H%M%S)"
|
||||
mkdir -p "$ROOT"
|
||||
TSV="$ROOT/screen.tsv"
|
||||
LOG="$ROOT/screen.log"
|
||||
printf 'op\top_file\tdtypes\trows_a\trows_b\tkernels_a\tkernels_b\tcvt_a\tcvt_b\tverdict\n' > "$TSV"
|
||||
|
||||
count_cvt() {
|
||||
find "$1/ttgir" -name '*.ttgir' -type f -exec cat {} + 2>/dev/null | grep -c convert_layout || true
|
||||
}
|
||||
count_kernels() { find "$1/ttgir" -name '*.ttgir' -type f 2>/dev/null | wc -l; }
|
||||
count_rows() { grep -c '^SUCCESS' "$1/run.log" 2>/dev/null || true; }
|
||||
|
||||
{
|
||||
echo ">>> switch=$SWITCH ops=$OPS"
|
||||
for entry in $OPS; do
|
||||
op="${entry%%:*}"; rest="${entry#*:}"
|
||||
if [[ "$rest" == "$entry" ]]; then op_file="$op"; dtypes=""
|
||||
else
|
||||
op_file="${rest%%:*}"; d="${rest#*:}"
|
||||
[[ "$d" == "$rest" ]] && dtypes="" || dtypes="$d"
|
||||
fi
|
||||
|
||||
echo "########## $op | A ($SWITCH=0, record) ##########"
|
||||
OP="$op" OP_FILE="$op_file" OUT_DIR="$ROOT/${op}_A" DTYPES="$dtypes" \
|
||||
timeout -k 30 "$CELL_TIMEOUT" env "$SWITCH=0" bash run_pytest.sh
|
||||
rc_a=$?
|
||||
|
||||
echo "########## $op | B ($SWITCH=1, replay A) ##########"
|
||||
OP="$op" OP_FILE="$op_file" OUT_DIR="$ROOT/${op}_B" DTYPES="$dtypes" \
|
||||
REPLAY_FROM="$ROOT/${op}_A" \
|
||||
timeout -k 30 "$CELL_TIMEOUT" env "$SWITCH=1" bash run_pytest.sh
|
||||
rc_b=$?
|
||||
|
||||
ra=$(count_rows "$ROOT/${op}_A"); rb=$(count_rows "$ROOT/${op}_B")
|
||||
ka=$(count_kernels "$ROOT/${op}_A"); kb=$(count_kernels "$ROOT/${op}_B")
|
||||
ca=$(count_cvt "$ROOT/${op}_A"); cb=$(count_cvt "$ROOT/${op}_B")
|
||||
fa=$(ir_fingerprint "$ROOT/${op}_A") || fa=""
|
||||
fb=$(ir_fingerprint "$ROOT/${op}_B") || fb=""
|
||||
|
||||
if (( rc_a != 0 || rc_b != 0 )) || [[ "${ra:-0}" == 0 || "${rb:-0}" == 0 ]]; then
|
||||
verdict="NO_ROWS"
|
||||
note="跑不出测量行（rc=$rc_a/$rc_b）。先看 console，多半是 dtype 不被该算子接受，或该算子在本后端拿不到 config"
|
||||
elif [[ -z "$fa" || -z "$fb" ]]; then
|
||||
verdict="NO_KERNEL"
|
||||
note="没编出 .ttgir：该算子在本后端没走 Triton。编译器开关对它无从谈起——这不是'开关无作用'"
|
||||
elif [[ "$fa" == "$fb" ]]; then
|
||||
verdict="NO_IR_DIFF"
|
||||
note="两侧 ttgir 逐字节相同（config 已锁死）：开关没改变该算子的生成代码，跑 A/B 只会测到噪声"
|
||||
else
|
||||
verdict="IR_CHANGED"
|
||||
note="生成代码有变（cvt ${ca}→${cb}，指纹 ${fa:0:8}/${fb:0:8}）：值得排六轮 A/B"
|
||||
fi
|
||||
printf '%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n' \
|
||||
"$op" "$op_file" "${dtypes:-<all>}" "${ra:-0}" "${rb:-0}" \
|
||||
"$ka" "$kb" "${ca:-0}" "${cb:-0}" "$verdict" >> "$TSV"
|
||||
echo ">>> $op: $verdict — $note"
|
||||
rm -rf "$ROOT/${op}_A/.triton_cache" "$ROOT/${op}_B/.triton_cache"
|
||||
done
|
||||
|
||||
echo "########## 汇总 ##########"
|
||||
awk -F'\t' 'NR>1{print " " $10 "\t" $1 "\t(cvt " $8 "→" $9 ", kernels " $6 "/" $7 ", rows " $4 "/" $5 ")"}' "$TSV" | sort
|
||||
echo
|
||||
echo ">>> 只把 IR_CHANGED 的算子送进 ab_repeat.sh。"
|
||||
echo ">>> NO_IR_DIFF 跑六轮只会测出噪声并报成一个'无差异'结论。"
|
||||
echo ">>> NO_KERNEL / NO_ROWS 要先修可运行性，不要当成开关的结论。"
|
||||
} 2>&1 | tee "$LOG"
|
||||
|
||||
sed -i -E $'s/\x1b\\[[0-9;]*[A-Za-z]//g' "$LOG"
|
||||
echo ">>> 结果表: $(pwd)/$TSV"
|
||||
echo ">>> 总 log: $(pwd)/$LOG"
|
||||
Reference in new issue
Block a user