forked from zhoulin/zl_bench
Aharrypotter
03fdcbab28
Two pieces the parallel campaign needs. partition.py assigns screened operators to cards longest-first. run_batch.py's existing hash partition ignores weight: on this batch it puts the critical path at 120 minutes and leaves a card idle, where greedy gets 70. The floor is set by the single heaviest operator — topk alone is 70.6 minutes against an ideal share of 45.4 — so partitioning cannot beat it, only keep the other cards from waiting. Weights come from screen.tsv's elapsed_before + elapsed_after, which is one pair, i.e. 2 cells; N rounds is 2N cells, so the estimate is N times the weight. Getting that wrong reported topk as 11.8 minutes. RESCORE=<root> re-judges an existing screening root against the current fingerprint and rewrites its screen.tsv, keeping the original alongside. The criterion has already changed twice — index.tsv used to be hashed, and the switch's own ttg.rlc-* annotations had to be stripped — and the second change moved 24 operators from 23 IR_CHANGED to 16. Rescoring the artifacts already on disk takes seconds; re-running that screen takes 79 minutes.
199 lines
9.9 KiB
Bash
199 lines
9.9 KiB
Bash
#!/bin/bash
|
||
# A/B 对比：FlagTree RemoveLayoutConversions 增强关 vs 开，逐算子两轮。
|
||
#
|
||
# 口径：每个算子先跑关（record autotune config），再用 REPLAY_FROM 回放同一套
|
||
# config 跑开，两侧只差 FLAGTREE_<VENDOR>_RLC_ENHANCE 这一个开关。产物沿用
|
||
# run_pytest.sh 的目录结构，所以每个算子恰好留下两份 run.log：
|
||
#
|
||
# runs/<op>_<时间戳>/run.log 关（before）
|
||
# runs/<op>_<时间戳>/run.log 开（after）
|
||
#
|
||
# 跑完用 batch_summary.py --ab <before> <after> 得到该算子的 speedup 区间。
|
||
#
|
||
# 这一轮同时充当 ab_repeat.sh 的筛选前置：一对的代价约 0.8 分钟，六轮 A/B 是
|
||
# 13~44 分钟（S5000 实测），差 20~50 倍。所以先用它过一遍，只把 IR_CHANGED 的
|
||
# 算子送进 ab_repeat.sh。判定写在 <root>/screen.tsv，四种之一：
|
||
#
|
||
# IR_CHANGED 两侧 ttgir 不同 -> 排六轮
|
||
# NO_IR_DIFF 逐字节相同（config 已锁死）-> 跳过，跑六轮只会测到噪声
|
||
# NO_TRITON 跑出行但没编出 .ttgir -> 该算子在本后端没走 Triton，
|
||
# 编译器开关无从谈起。不是"开关无作用"
|
||
# NO_ROWS 没有测量行 -> 先修可运行性，多半是 dtype
|
||
#
|
||
# 判据是 ttgir 指纹而不是 convert_layout 计数：cvt 数相同不等于生成代码相同。
|
||
# cvt 数照样记，它解释的是变化的性质。
|
||
#
|
||
# 用法：
|
||
# OPS="softmax gelu_backward" bash ab_rlc_test.sh
|
||
# OPS="sort:perf_sort topk" bash ab_rlc_test.sh # OP:OP_FILE
|
||
# OPS="perf_fft:fft:complex64 topk:topk" bash ab_rlc_test.sh # 再加 DTYPES
|
||
#
|
||
# DTYPES 留空表示用该算子自己的全套。**不要全局写死一个 dtype**：fft 只吃
|
||
# complex64，被一刀切的 DTYPES=float16 判死过一次——那是 harness 的问题，
|
||
# 却被记成了算子的问题。
|
||
#
|
||
# 注意：RLC 开关名带厂商前缀，用 VENDOR 指定（默认 MUSA）。开关名拼错时两侧跑
|
||
# 同一份代码，此时判定会是 NO_IR_DIFF——先查拼写，再下"开关无作用"的结论。
|
||
set -uo pipefail
|
||
|
||
cd "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||
source "./_ir_fingerprint.sh"
|
||
|
||
# RESCORE=<已有的 root>：判据改了之后，用当前的 ir_fingerprint 对盘上已有的 ttgir
|
||
# 重新打分并改写该 root 的 screen.tsv，不重跑任何测量。
|
||
#
|
||
# 需要它是因为判据确实改过：_ir_fingerprint.sh 起初把 index.tsv 也算进哈希，后来又
|
||
# 发现要抹掉开关自己盖的 ttg.rlc-* 属性。同一批 24 个算子，结论从 23 个 IR_CHANGED
|
||
# 变成 16 个。重打分是秒级的，重跑那批筛选要 79 分钟。
|
||
#
|
||
# 原表存为 screen.tsv.before-rescore，判定有变化的算子逐个打出来。
|
||
RESCORE="${RESCORE:-}"
|
||
if [[ -n "$RESCORE" ]]; then
|
||
[[ -d "$RESCORE" ]] || { echo "!!! 目录不存在: $RESCORE" >&2; exit 2; }
|
||
TSV="$RESCORE/screen.tsv"
|
||
[[ -f "$TSV" ]] || { echo "!!! 没有 screen.tsv: $TSV" >&2; exit 2; }
|
||
cp "$TSV" "$TSV.before-rescore"
|
||
|
||
hdr=$(head -1 "$TSV.before-rescore")
|
||
# 列名 -> 序号，不写死列序
|
||
idx() { echo "$hdr" | tr '\t' '\n' | grep -nxF "$1" | cut -d: -f1; }
|
||
c_op=$(idx op); c_rb=$(idx rows_before); c_ra=$(idx rows_after); c_v=$(idx verdict)
|
||
[[ -n "$c_op" && -n "$c_v" ]] || { echo "!!! screen.tsv 缺 op/verdict 列" >&2; exit 2; }
|
||
|
||
printf '%s\n' "$hdr" > "$TSV"
|
||
changed=0
|
||
while IFS= read -r line; do
|
||
op=$(printf '%s' "$line" | cut -f"$c_op")
|
||
old=$(printf '%s' "$line" | cut -f"$c_v")
|
||
rb=$(printf '%s' "$line" | cut -f"${c_rb:-0}"); ra=$(printf '%s' "$line" | cut -f"${c_ra:-0}")
|
||
fa=$(ir_fingerprint "$RESCORE/${op}_before") || fa=""
|
||
fb=$(ir_fingerprint "$RESCORE/${op}_after") || fb=""
|
||
if [[ "${rb:-0}" == 0 || "${ra:-0}" == 0 ]]; then v=NO_ROWS
|
||
elif [[ -z "$fa" || -z "$fb" ]]; then v=NO_TRITON
|
||
elif [[ "$fa" == "$fb" ]]; then v=NO_IR_DIFF
|
||
else v=IR_CHANGED; fi
|
||
printf '%s\n' "$line" | awk -v FS='\t' -v OFS='\t' -v i="$c_v" -v v="$v" '{$i=v; print}' >> "$TSV"
|
||
if [[ "$old" != "$v" ]]; then
|
||
printf ' %-22s %-12s -> %s\n' "$op" "$old" "$v" >&2
|
||
changed=$((changed+1))
|
||
fi
|
||
done < <(tail -n +2 "$TSV.before-rescore")
|
||
|
||
echo ">>> 重打分完成，$changed 个算子的判定变了；原表存为 $TSV.before-rescore"
|
||
awk -F'\t' 'NR>1{c[$NF]++} END{for (k in c) printf(" %-12s %d\n", k, c[k])}' "$TSV"
|
||
exit 0
|
||
fi
|
||
|
||
VENDOR="${VENDOR:-MUSA}"
|
||
ENHANCE_VAR="FLAGTREE_${VENDOR}_RLC_ENHANCE"
|
||
|
||
# 空格分隔的算子表。元素为 OP 或 OP:OP_FILE（省略时 OP_FILE=OP）。
|
||
OPS="${OPS:-softmax}"
|
||
|
||
# 筛选是第一次接触一个算子的地方，最容易撞上挂死的那个。没有上限时一个算子
|
||
# 能把整批停在那里，而筛选的全部意义就是快。
|
||
CELL_TIMEOUT="${CELL_TIMEOUT:-2400}"
|
||
|
||
if [[ -t 1 && -z "${NO_COLOR:-}" ]]; then
|
||
export FLAGGEMS_PERF_COLOR=always
|
||
fi
|
||
|
||
# 显式 OUT_DIR，不再用 `ls -dt runs/<op>_*` 猜"最新那个就是我刚跑的"。串行下
|
||
# 那个假设成立，多卡并行共用 runs/ 就不成立了。
|
||
ROOT="runs/ab_rlc_$(date +%Y%m%d-%H%M%S)"
|
||
mkdir -p "$ROOT"
|
||
LOG="$ROOT/summary.log"
|
||
TSV="$ROOT/screen.tsv"
|
||
printf 'op op_file dtypes rows_before rows_after kernels_before kernels_after cvt_before cvt_after elapsed_before elapsed_after verdict backend device timer bench_level shape_mode tuner_cache direct_repeat\n' > "$TSV"
|
||
|
||
count_cvt() { find "$1/ttgir" -name '*.ttgir' -type f -exec cat {} + 2>/dev/null | grep -c convert_layout || true; }
|
||
count_kernels() { find "$1/ttgir" -name '*.ttgir' -type f 2>/dev/null | wc -l; }
|
||
count_rows() { grep -c '^SUCCESS' "$1/run.log" 2>/dev/null || true; }
|
||
|
||
{
|
||
echo ">>> vendor=$VENDOR switch=$ENHANCE_VAR ops=$OPS"
|
||
for entry in $OPS; do
|
||
op="${entry%%:*}"; rest="${entry#*:}"
|
||
if [[ "$rest" == "$entry" ]]; then op_file="$op"; dtypes=""
|
||
else
|
||
op_file="${rest%%:*}"; d="${rest#*:}"
|
||
[[ "$d" == "$rest" ]] && dtypes="" || dtypes="$d"
|
||
fi
|
||
BEFORE="$ROOT/${op}_before"; AFTER="$ROOT/${op}_after"
|
||
|
||
echo "########## $op | $ENHANCE_VAR=0 before (record) ##########"
|
||
t0=$(date +%s)
|
||
OP="$op" OP_FILE="$op_file" OUT_DIR="$BEFORE" DTYPES="$dtypes" \
|
||
timeout -k 30 "$CELL_TIMEOUT" env "$ENHANCE_VAR=0" bash run_pytest.sh
|
||
rc_before=$?; el_before=$(( $(date +%s) - t0 ))
|
||
|
||
echo "########## $op | $ENHANCE_VAR=1 after (replay $BEFORE) ##########"
|
||
t0=$(date +%s)
|
||
OP="$op" OP_FILE="$op_file" OUT_DIR="$AFTER" DTYPES="$dtypes" \
|
||
REPLAY_FROM="$BEFORE" timeout -k 30 "$CELL_TIMEOUT" env "$ENHANCE_VAR=1" bash run_pytest.sh
|
||
rc_after=$?; el_after=$(( $(date +%s) - t0 ))
|
||
for rc in "$rc_before" "$rc_after"; do
|
||
(( rc == 124 || rc == 137 )) && echo "!!! CELL_TIMEOUT ${CELL_TIMEOUT}s op=$op"
|
||
done
|
||
|
||
rb=$(count_rows "$BEFORE"); ra=$(count_rows "$AFTER")
|
||
kb=$(count_kernels "$BEFORE"); ka=$(count_kernels "$AFTER")
|
||
cb=$(count_cvt "$BEFORE"); ca=$(count_cvt "$AFTER")
|
||
fb=$(ir_fingerprint "$BEFORE") || fb=""
|
||
fa=$(ir_fingerprint "$AFTER") || fa=""
|
||
|
||
if (( rc_before != 0 || rc_after != 0 )) || [[ "${rb:-0}" == 0 || "${ra:-0}" == 0 ]]; then
|
||
verdict=NO_ROWS
|
||
if (( rc_before == 124 || rc_after == 124 || rc_before == 137 || rc_after == 137 )); then
|
||
note="超过 CELL_TIMEOUT=${CELL_TIMEOUT}s 被截断（rc=$rc_before/$rc_after）。这是耗时问题，不是对开关的判定——先看它为什么这么慢"
|
||
else
|
||
note="跑不出测量行（rc=$rc_before/$rc_after）。看 run.log——多半是 dtype 不被该算子接受，或它在本后端拿不到 config"
|
||
fi
|
||
elif [[ -z "$fb" || -z "$fa" ]]; then
|
||
verdict=NO_TRITON
|
||
note="跑出了行但没编出 .ttgir：本次运行没有任何东西经过 Triton。算子本身可能完全正常（多半跑的是 aten 或厂商库），只是被测的编译器 pass 没有作用对象。不是'开关无作用'，也不是'没有 kernel'"
|
||
elif [[ "$fb" == "$fa" ]]; then
|
||
verdict=NO_IR_DIFF
|
||
note="两侧 ttgir 逐字节相同（config 已锁死）：开关没改变生成代码。先查开关名拼写，再下结论"
|
||
else
|
||
verdict=IR_CHANGED
|
||
note="生成代码有变（cvt ${cb}→${ca}，指纹 ${fb:0:8}/${fa:0:8}）：值得排六轮 A/B"
|
||
fi
|
||
# 口径从 batch_summary.py 取，那里是 CALIBER_FIELDS 的唯一定义。两份不同
|
||
# --level 的 screen.tsv 放一起比时，这几列是唯一能拦住的东西。
|
||
cal=$(python batch_summary.py --caliber "$BEFORE" 2>/dev/null)
|
||
printf '%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n' \
|
||
"$op" "$op_file" "${dtypes:-<all>}" "${rb:-0}" "${ra:-0}" \
|
||
"$kb" "$ka" "${cb:-0}" "${ca:-0}" "$el_before" "$el_after" "$verdict" \
|
||
"$cal" >> "$TSV"
|
||
echo ">>> $op: $verdict — $note"
|
||
|
||
# 判定说这一对不可比，就别在下一行给出一个看起来像结论的区间
|
||
case "$verdict" in
|
||
NO_ROWS)
|
||
echo "!!! $op 没有可对比的测量，不出 speedup 区间"; continue ;;
|
||
NO_TRITON)
|
||
echo "!!! $op 没有经过 Triton，speedup 区间与被测开关无关，不出"; continue ;;
|
||
NO_IR_DIFF)
|
||
echo ">>> $op 两侧生成代码相同：下面这个区间是本机对该算子的噪声底，"
|
||
echo ">>> 不是开关的收益。它离零多远，就是该算子单次测量的可信下限。" ;;
|
||
esac
|
||
echo "########## $op | summary ##########"
|
||
python batch_summary.py --ab "$BEFORE/run.log" "$AFTER/run.log" --label "$op" \
|
||
|| echo "!!! summary failed: $op"
|
||
done
|
||
|
||
echo "########## 筛选汇总 ##########"
|
||
awk -F'\t' 'NR>1{printf " %-12s %-22s cvt %s->%s kernels %s/%s rows %s/%s %ss\n", \
|
||
$12,$1,$8,$9,$6,$7,$4,$5,$10+$11}' "$TSV" | sort
|
||
echo ">>> 只把 IR_CHANGED 的算子送进 ab_repeat.sh。"
|
||
echo "########## ALL DONE ##########"
|
||
} 2>&1 | tee "$LOG"
|
||
|
||
# 去掉 ANSI 转义以便 grep/diff（终端输出保留颜色）
|
||
sed -i -E $'s/\x1b\\[[0-9;]*[A-Za-z]//g' "$LOG"
|
||
|
||
echo ">>> 总 log: $(pwd)/$LOG"
|
||
echo ">>> 筛选表: $(pwd)/$TSV"
|
||
echo ">>> 每算子两份 run.log 在 $ROOT/<op>_{before,after}/ 下"
|