forked from zhoulin/zl_bench
bench: let the verdict govern the report, and put the caliber in the table
A pair judged incomparable was still followed by a speedup range on the next line. A NO_TRITON pair's range says nothing about the switch — nothing reached the compiler — and a NO_IR_DIFF pair's range is the machine's noise floor for that operator, which is useful but is not a result. Printing either unlabelled produces exactly the thing this tooling exists to prevent: a table that looks like a finding. NO_ROWS and NO_TRITON now print no range; NO_IR_DIFF prints one and says what it is — how far it sits from zero is the confidence floor for a single measurement of that operator. screen.tsv gains the seven caliber columns, read back through `batch_summary.py --caliber`, so CALIBER_FIELDS stays the single definition and the screening table does not restate it. Two screen.tsv files taken at different --level are otherwise indistinguishable side by side, and those columns are the only thing that would stop the comparison.
This commit is contained in:
1 parent
a11863399a
commit
2e456bdc74
3 files changed
+38
-7
No files matched your search
@@ -172,6 +172,20 @@ SWITCH=FLAGTREE_MUSA_RLC_ENHANCE ROUNDS=6 OPS="topk:topk" bash ab_repeat.sh
|
||||
同一份 `_ir_fingerprint.sh` 也被 `ab_repeat.sh` 用于第 1 轮后的对比轴有效性检查,
|
||||
两处判据不会漂开。
|
||||
|
||||
判定还管住报告:`NO_ROWS` 与 `NO_TRITON` 不出 speedup 区间(前者没有可比的测量,
|
||||
后者与被测开关无关);`NO_IR_DIFF` 照出,但明说那是**本机对该算子的噪声底**,不是
|
||||
开关的收益——它离零多远,就是该算子单次测量的可信下限。判定刚说完这一对不可比,
|
||||
下一行就给个 `+0.3%`,正是这套工具一路在防的"看起来像结论的表"。
|
||||
|
||||
`screen.tsv` 末七列是口径(`backend device timer bench_level shape_mode
|
||||
tuner_cache direct_repeat`),由 `python batch_summary.py --caliber <run_dir>` 取——
|
||||
`CALIBER_FIELDS` 只有那一处定义,筛选表不重列一遍。两份不同 `--level` 的
|
||||
`screen.tsv` 放一起比时,这几列是唯一能拦住的东西。
|
||||
|
||||
每对有 `CELL_TIMEOUT`(默认 2400 秒)兜底,超时打 `!!! CELL_TIMEOUT` 并按耗时问题
|
||||
报告,不混进 dtype 那条解释里。`elapsed_before/after` 也进表:两步流程的全部依据
|
||||
就是"一对 0.8 分钟对六轮 13~44 分钟"这个比例,那这一步该记下自己那一半。
|
||||
|
||||
## A/B 对比测试标准流程
|
||||
|
||||
```bash
|
||||
|
||||
+17
-7
@@ -59,7 +59,7 @@ ROOT="runs/ab_rlc_$(date +%Y%m%d-%H%M%S)"
|
||||
mkdir -p "$ROOT"
|
||||
LOG="$ROOT/summary.log"
|
||||
TSV="$ROOT/screen.tsv"
|
||||
printf 'op op_file dtypes rows_before rows_after kernels_before kernels_after cvt_before cvt_after elapsed_before elapsed_after verdict\n' > "$TSV"
|
||||
printf 'op op_file dtypes rows_before rows_after kernels_before kernels_after cvt_before cvt_after elapsed_before elapsed_after verdict backend device timer bench_level shape_mode tuner_cache direct_repeat\n' > "$TSV"
|
||||
|
||||
count_cvt() { find "$1/ttgir" -name '*.ttgir' -type f -exec cat {} + 2>/dev/null | grep -c convert_layout || true; }
|
||||
count_kernels() { find "$1/ttgir" -name '*.ttgir' -type f 2>/dev/null | wc -l; }
|
||||
@@ -114,15 +114,25 @@ count_rows() { grep -c '^SUCCESS' "$1/run.log" 2>/dev/null || true; }
|
||||
verdict=IR_CHANGED
|
||||
note="生成代码有变(cvt ${cb}→${ca},指纹 ${fb:0:8}/${fa:0:8}):值得排六轮 A/B"
|
||||
fi
|
||||
printf '%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n' \
|
||||
# 口径从 batch_summary.py 取,那里是 CALIBER_FIELDS 的唯一定义。两份不同
|
||||
# --level 的 screen.tsv 放一起比时,这几列是唯一能拦住的东西。
|
||||
cal=$(python batch_summary.py --caliber "$BEFORE" 2>/dev/null)
|
||||
printf '%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n' \
|
||||
"$op" "$op_file" "${dtypes:-<all>}" "${rb:-0}" "${ra:-0}" \
|
||||
"$kb" "$ka" "${cb:-0}" "${ca:-0}" "$el_before" "$el_after" "$verdict" >> "$TSV"
|
||||
"$kb" "$ka" "${cb:-0}" "${ca:-0}" "$el_before" "$el_after" "$verdict" \
|
||||
"$cal" >> "$TSV"
|
||||
echo ">>> $op: $verdict — $note"
|
||||
|
||||
if [[ "$verdict" == NO_ROWS ]]; then
|
||||
echo "!!! $op 没有可对比的测量,不出 speedup 区间"
|
||||
continue
|
||||
fi
|
||||
# 判定说这一对不可比,就别在下一行给出一个看起来像结论的区间
|
||||
case "$verdict" in
|
||||
NO_ROWS)
|
||||
echo "!!! $op 没有可对比的测量,不出 speedup 区间"; continue ;;
|
||||
NO_TRITON)
|
||||
echo "!!! $op 没有经过 Triton,speedup 区间与被测开关无关,不出"; continue ;;
|
||||
NO_IR_DIFF)
|
||||
echo ">>> $op 两侧生成代码相同:下面这个区间是本机对该算子的噪声底,"
|
||||
echo ">>> 不是开关的收益。它离零多远,就是该算子单次测量的可信下限。" ;;
|
||||
esac
|
||||
echo "########## $op | summary ##########"
|
||||
python batch_summary.py --ab "$BEFORE/run.log" "$AFTER/run.log" --label "$op" \
|
||||
|| echo "!!! summary failed: $op"
|
||||
|
||||
@@ -485,10 +485,17 @@ def main() -> None:
|
||||
help="多轮 A/B 目录,含 r<k>_A/run.log 与 r<k>_B/run.log;"
|
||||
"逐 shape 取轮次中位,并报逐轮区间与符号稳定性")
|
||||
p.add_argument("--label", default="", help="--ab / --ab-rounds 报告里的算子名")
|
||||
p.add_argument("--caliber", type=Path, default=None, metavar="RUN_DIR",
|
||||
help="打印该 run 的口径字段(TSV 一行),供筛选表复用")
|
||||
p.add_argument("--allow-caliber-mismatch", action="store_true",
|
||||
help="口径不一致时仍出报告(默认拒绝)。只在确认两次运行"
|
||||
"确实可比时使用")
|
||||
args = p.parse_args()
|
||||
if args.caliber is not None:
|
||||
# 口径的定义只有 CALIBER_FIELDS 一处;筛选表等复用它,不要在别处重列
|
||||
cal = caliber_of(args.caliber) or {}
|
||||
print(" ".join(str(cal.get(k, "")) for k in CALIBER_FIELDS))
|
||||
return
|
||||
if args.ab_rounds:
|
||||
if not args.ab_rounds.is_dir():
|
||||
raise SystemExit(f"目录不存在: {args.ab_rounds}")
|
||||
|
||||
Reference in new issue
Block a user