forked from zhoulin/zl_bench
Aharrypotter
2786eb9886
campaign.sh fans a screened operator list across cards, one ab_repeat.sh per card, serial within a card. It does no measuring: ab_repeat.sh is the path that produced the four operator verdicts already on record, and it stays byte for byte as it was. Each card gets its own AB_ROOT. Eight ab_repeat.sh started in the same second would otherwise compute the same timestamped directory name and two cards would write into one. ZL_ISOLATE_TUNER_CACHE=1 is forced, since eight processes on one sqlite is the thing to avoid. The cost is that pinned configs differ from a shared-cache batch, so absolute latencies cannot be quoted across the two; the caliber guard refuses such comparisons on its own, which is why this is a note and not a warning someone has to remember. partition.py's shell output gained the per-card weight, because EXTRA_OPS has to land on the lightest card and counting operators gets that wrong — one heavy operator outweighs three light ones. It writes through stdout.buffer: Python turns "\n" into CRLF on Windows, and the trailing \r then rides into the last field and breaks the numeric comparison that picks the card.
199 lines
9.9 KiB
Bash
199 lines
9.9 KiB
Bash
#!/bin/bash
|
||
# A/B 对比：FlagTree RemoveLayoutConversions 增强关 vs 开，逐算子两轮。
|
||
#
|
||
# 口径：每个算子先跑关（record autotune config），再用 REPLAY_FROM 回放同一套
|
||
# config 跑开，两侧只差 FLAGTREE_<VENDOR>_RLC_ENHANCE 这一个开关。产物沿用
|
||
# run_pytest.sh 的目录结构，所以每个算子恰好留下两份 run.log：
|
||
#
|
||
# runs/<op>_<时间戳>/run.log 关（before）
|
||
# runs/<op>_<时间戳>/run.log 开（after）
|
||
#
|
||
# 跑完用 batch_summary.py --ab <before> <after> 得到该算子的 speedup 区间。
|
||
#
|
||
# 这一轮同时充当 ab_repeat.sh 的筛选前置：一对的代价约 0.8 分钟，六轮 A/B 是
|
||
# 13~44 分钟（S5000 实测），差 20~50 倍。所以先用它过一遍，只把 IR_CHANGED 的
|
||
# 算子送进 ab_repeat.sh。判定写在 <root>/screen.tsv，四种之一：
|
||
#
|
||
# IR_CHANGED 两侧 ttgir 不同 -> 排六轮
|
||
# NO_IR_DIFF 逐字节相同（config 已锁死）-> 跳过，跑六轮只会测到噪声
|
||
# NO_TRITON 跑出行但没编出 .ttgir -> 该算子在本后端没走 Triton，
|
||
# 编译器开关无从谈起。不是"开关无作用"
|
||
# NO_ROWS 没有测量行 -> 先修可运行性，多半是 dtype
|
||
#
|
||
# 判据是 ttgir 指纹而不是 convert_layout 计数：cvt 数相同不等于生成代码相同。
|
||
# cvt 数照样记，它解释的是变化的性质。
|
||
#
|
||
# 用法：
|
||
# OPS="softmax gelu_backward" bash ab_rlc_test.sh
|
||
# OPS="sort:perf_sort topk" bash ab_rlc_test.sh # OP:OP_FILE
|
||
# OPS="perf_fft:fft:complex64 topk:topk" bash ab_rlc_test.sh # 再加 DTYPES
|
||
#
|
||
# DTYPES 留空表示用该算子自己的全套。**不要全局写死一个 dtype**：fft 只吃
|
||
# complex64，被一刀切的 DTYPES=float16 判死过一次——那是 harness 的问题，
|
||
# 却被记成了算子的问题。
|
||
#
|
||
# 注意：RLC 开关名带厂商前缀，用 VENDOR 指定（默认 MUSA）。开关名拼错时两侧跑
|
||
# 同一份代码，此时判定会是 NO_IR_DIFF——先查拼写，再下"开关无作用"的结论。
|
||
set -uo pipefail
|
||
|
||
cd "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||
source "./_ir_fingerprint.sh"
|
||
|
||
# RESCORE=<已有的 root>：判据改了之后，用当前的 ir_fingerprint 对盘上已有的 ttgir
|
||
# 重新打分并改写该 root 的 screen.tsv，不重跑任何测量。
|
||
#
|
||
# 需要它是因为判据确实改过：_ir_fingerprint.sh 起初把 index.tsv 也算进哈希，后来又
|
||
# 发现要抹掉开关自己盖的 ttg.rlc-* 属性。同一批 24 个算子，结论从 23 个 IR_CHANGED
|
||
# 变成 16 个。重打分是秒级的，重跑那批筛选要 79 分钟。
|
||
#
|
||
# 原表存为 screen.tsv.before-rescore，判定有变化的算子逐个打出来。
|
||
RESCORE="${RESCORE:-}"
|
||
if [[ -n "$RESCORE" ]]; then
|
||
[[ -d "$RESCORE" ]] || { echo "!!! 目录不存在: $RESCORE" >&2; exit 2; }
|
||
TSV="$RESCORE/screen.tsv"
|
||
[[ -f "$TSV" ]] || { echo "!!! 没有 screen.tsv: $TSV" >&2; exit 2; }
|
||
cp "$TSV" "$TSV.before-rescore"
|
||
|
||
hdr=$(head -1 "$TSV.before-rescore")
|
||
# 列名 -> 序号，不写死列序
|
||
idx() { echo "$hdr" | tr '\t' '\n' | grep -nxF "$1" | cut -d: -f1; }
|
||
c_op=$(idx op); c_rb=$(idx rows_before); c_ra=$(idx rows_after); c_v=$(idx verdict)
|
||
[[ -n "$c_op" && -n "$c_v" ]] || { echo "!!! screen.tsv 缺 op/verdict 列" >&2; exit 2; }
|
||
|
||
printf '%s\n' "$hdr" > "$TSV"
|
||
changed=0
|
||
while IFS= read -r line; do
|
||
op=$(printf '%s' "$line" | cut -f"$c_op")
|
||
old=$(printf '%s' "$line" | cut -f"$c_v")
|
||
rb=$(printf '%s' "$line" | cut -f"${c_rb:-0}"); ra=$(printf '%s' "$line" | cut -f"${c_ra:-0}")
|
||
fa=$(ir_fingerprint "$RESCORE/${op}_before") || fa=""
|
||
fb=$(ir_fingerprint "$RESCORE/${op}_after") || fb=""
|
||
if [[ "${rb:-0}" == 0 || "${ra:-0}" == 0 ]]; then v=NO_ROWS
|
||
elif [[ -z "$fa" || -z "$fb" ]]; then v=NO_TRITON
|
||
elif [[ "$fa" == "$fb" ]]; then v=NO_IR_DIFF
|
||
else v=IR_CHANGED; fi
|
||
printf '%s\n' "$line" | awk -v FS='\t' -v OFS='\t' -v i="$c_v" -v v="$v" '{$i=v; print}' >> "$TSV"
|
||
if [[ "$old" != "$v" ]]; then
|
||
printf ' %-22s %-12s -> %s\n' "$op" "$old" "$v" >&2
|
||
changed=$((changed+1))
|
||
fi
|
||
done < <(tail -n +2 "$TSV.before-rescore")
|
||
|
||
echo ">>> 重打分完成，$changed 个算子的判定变了；原表存为 $TSV.before-rescore"
|
||
awk -v i="$c_v" -F'\t' 'NR>1{c[$i]++} END{for (k in c) printf(" %-12s %d\n", k, c[k])}' "$TSV"
|
||
exit 0
|
||
fi
|
||
|
||
VENDOR="${VENDOR:-MUSA}"
|
||
ENHANCE_VAR="FLAGTREE_${VENDOR}_RLC_ENHANCE"
|
||
|
||
# 空格分隔的算子表。元素为 OP 或 OP:OP_FILE（省略时 OP_FILE=OP）。
|
||
OPS="${OPS:-softmax}"
|
||
|
||
# 筛选是第一次接触一个算子的地方，最容易撞上挂死的那个。没有上限时一个算子
|
||
# 能把整批停在那里，而筛选的全部意义就是快。
|
||
CELL_TIMEOUT="${CELL_TIMEOUT:-2400}"
|
||
|
||
if [[ -t 1 && -z "${NO_COLOR:-}" ]]; then
|
||
export FLAGGEMS_PERF_COLOR=always
|
||
fi
|
||
|
||
# 显式 OUT_DIR，不再用 `ls -dt runs/<op>_*` 猜"最新那个就是我刚跑的"。串行下
|
||
# 那个假设成立，多卡并行共用 runs/ 就不成立了。
|
||
ROOT="runs/ab_rlc_$(date +%Y%m%d-%H%M%S)"
|
||
mkdir -p "$ROOT"
|
||
LOG="$ROOT/summary.log"
|
||
TSV="$ROOT/screen.tsv"
|
||
printf 'op op_file dtypes rows_before rows_after kernels_before kernels_after cvt_before cvt_after elapsed_before elapsed_after verdict backend device timer bench_level shape_mode tuner_cache direct_repeat\n' > "$TSV"
|
||
|
||
count_cvt() { find "$1/ttgir" -name '*.ttgir' -type f -exec cat {} + 2>/dev/null | grep -c convert_layout || true; }
|
||
count_kernels() { find "$1/ttgir" -name '*.ttgir' -type f 2>/dev/null | wc -l; }
|
||
count_rows() { grep -c '^SUCCESS' "$1/run.log" 2>/dev/null || true; }
|
||
|
||
{
|
||
echo ">>> vendor=$VENDOR switch=$ENHANCE_VAR ops=$OPS"
|
||
for entry in $OPS; do
|
||
op="${entry%%:*}"; rest="${entry#*:}"
|
||
if [[ "$rest" == "$entry" ]]; then op_file="$op"; dtypes=""
|
||
else
|
||
op_file="${rest%%:*}"; d="${rest#*:}"
|
||
[[ "$d" == "$rest" ]] && dtypes="" || dtypes="$d"
|
||
fi
|
||
BEFORE="$ROOT/${op}_before"; AFTER="$ROOT/${op}_after"
|
||
|
||
echo "########## $op | $ENHANCE_VAR=0 before (record) ##########"
|
||
t0=$(date +%s)
|
||
OP="$op" OP_FILE="$op_file" OUT_DIR="$BEFORE" DTYPES="$dtypes" \
|
||
timeout -k 30 "$CELL_TIMEOUT" env "$ENHANCE_VAR=0" bash run_pytest.sh
|
||
rc_before=$?; el_before=$(( $(date +%s) - t0 ))
|
||
|
||
echo "########## $op | $ENHANCE_VAR=1 after (replay $BEFORE) ##########"
|
||
t0=$(date +%s)
|
||
OP="$op" OP_FILE="$op_file" OUT_DIR="$AFTER" DTYPES="$dtypes" \
|
||
REPLAY_FROM="$BEFORE" timeout -k 30 "$CELL_TIMEOUT" env "$ENHANCE_VAR=1" bash run_pytest.sh
|
||
rc_after=$?; el_after=$(( $(date +%s) - t0 ))
|
||
for rc in "$rc_before" "$rc_after"; do
|
||
(( rc == 124 || rc == 137 )) && echo "!!! CELL_TIMEOUT ${CELL_TIMEOUT}s op=$op"
|
||
done
|
||
|
||
rb=$(count_rows "$BEFORE"); ra=$(count_rows "$AFTER")
|
||
kb=$(count_kernels "$BEFORE"); ka=$(count_kernels "$AFTER")
|
||
cb=$(count_cvt "$BEFORE"); ca=$(count_cvt "$AFTER")
|
||
fb=$(ir_fingerprint "$BEFORE") || fb=""
|
||
fa=$(ir_fingerprint "$AFTER") || fa=""
|
||
|
||
if (( rc_before != 0 || rc_after != 0 )) || [[ "${rb:-0}" == 0 || "${ra:-0}" == 0 ]]; then
|
||
verdict=NO_ROWS
|
||
if (( rc_before == 124 || rc_after == 124 || rc_before == 137 || rc_after == 137 )); then
|
||
note="超过 CELL_TIMEOUT=${CELL_TIMEOUT}s 被截断（rc=$rc_before/$rc_after）。这是耗时问题，不是对开关的判定——先看它为什么这么慢"
|
||
else
|
||
note="跑不出测量行（rc=$rc_before/$rc_after）。看 run.log——多半是 dtype 不被该算子接受，或它在本后端拿不到 config"
|
||
fi
|
||
elif [[ -z "$fb" || -z "$fa" ]]; then
|
||
verdict=NO_TRITON
|
||
note="跑出了行但没编出 .ttgir：本次运行没有任何东西经过 Triton。算子本身可能完全正常（多半跑的是 aten 或厂商库），只是被测的编译器 pass 没有作用对象。不是'开关无作用'，也不是'没有 kernel'"
|
||
elif [[ "$fb" == "$fa" ]]; then
|
||
verdict=NO_IR_DIFF
|
||
note="两侧 ttgir 逐字节相同（config 已锁死）：开关没改变生成代码。先查开关名拼写，再下结论"
|
||
else
|
||
verdict=IR_CHANGED
|
||
note="生成代码有变（cvt ${cb}→${ca}，指纹 ${fb:0:8}/${fa:0:8}）：值得排六轮 A/B"
|
||
fi
|
||
# 口径从 batch_summary.py 取，那里是 CALIBER_FIELDS 的唯一定义。两份不同
|
||
# --level 的 screen.tsv 放一起比时，这几列是唯一能拦住的东西。
|
||
cal=$(python batch_summary.py --caliber "$BEFORE" 2>/dev/null)
|
||
printf '%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n' \
|
||
"$op" "$op_file" "${dtypes:-<all>}" "${rb:-0}" "${ra:-0}" \
|
||
"$kb" "$ka" "${cb:-0}" "${ca:-0}" "$el_before" "$el_after" "$verdict" \
|
||
"$cal" >> "$TSV"
|
||
echo ">>> $op: $verdict — $note"
|
||
|
||
# 判定说这一对不可比，就别在下一行给出一个看起来像结论的区间
|
||
case "$verdict" in
|
||
NO_ROWS)
|
||
echo "!!! $op 没有可对比的测量，不出 speedup 区间"; continue ;;
|
||
NO_TRITON)
|
||
echo "!!! $op 没有经过 Triton，speedup 区间与被测开关无关，不出"; continue ;;
|
||
NO_IR_DIFF)
|
||
echo ">>> $op 两侧生成代码相同：下面这个区间是本机对该算子的噪声底，"
|
||
echo ">>> 不是开关的收益。它离零多远，就是该算子单次测量的可信下限。" ;;
|
||
esac
|
||
echo "########## $op | summary ##########"
|
||
python batch_summary.py --ab "$BEFORE/run.log" "$AFTER/run.log" --label "$op" \
|
||
|| echo "!!! summary failed: $op"
|
||
done
|
||
|
||
echo "########## 筛选汇总 ##########"
|
||
awk -F'\t' 'NR>1{printf " %-12s %-22s cvt %s->%s kernels %s/%s rows %s/%s %ss\n", \
|
||
$12,$1,$8,$9,$6,$7,$4,$5,$10+$11}' "$TSV" | sort
|
||
echo ">>> 只把 IR_CHANGED 的算子送进 ab_repeat.sh。"
|
||
echo "########## ALL DONE ##########"
|
||
} 2>&1 | tee "$LOG"
|
||
|
||
# 去掉 ANSI 转义以便 grep/diff（终端输出保留颜色）
|
||
sed -i -E $'s/\x1b\\[[0-9;]*[A-Za-z]//g' "$LOG"
|
||
|
||
echo ">>> 总 log: $(pwd)/$LOG"
|
||
echo ">>> 筛选表: $(pwd)/$TSV"
|
||
echo ">>> 每算子两份 run.log 在 $ROOT/<op>_{before,after}/ 下"
|