zhoulin
1e1d612031
For shape-heavy ops the autotune sweep dominates wall time, not the measurement. _parallel_warmup_plugin shards the sweep across N GPUs and hands the merged configs to the normal single-GPU serial measurement, so timing stays comparable. Activated by PARALLEL_WARMUP_GPUS=N (silent no-op when unset, so it can stay in the plugin list); output layout, REPLAY_FROM usage and the latency table are unchanged. Measurement is deliberately not parallelized: N processes saturating one box couple through the power/thermal budget, so per-card latency gets dragged by its neighbours by an amount that does not reproduce. Config keys whose winner differs across shards are counted and reported as a WARNING -- that count is how much to trust the run. Fixes found while auditing: - Shards no longer overwrite CUDA_VISIBLE_DEVICES with a bare shard index, which a caller who had selected idle cards (e.g. 6,7) would see re-interpreted as absolute ids 0,1 -- silently benchmarking on the busy cards they were avoiding. - Interrupting the sweep no longer leaves N subprocesses holding GPUs; children are terminated and reaped before the exception propagates (BaseException, since KeyboardInterrupt is not an Exception). - run_pytest.sh claims its output dir with a bare mkdir and retreats to a -2/-3 suffix on collision. The second-resolution timestamp meant two concurrent runs shared one directory and overwrote each other. - Replay eviction failures now emit a distinct compile_*_no_evict marker. LibTuner.cache is a sqlite-backed ConfigCache with no __delitem__, so the bad config could not be dropped and the retry re-read it, while the log still claimed a clean fallback to live autotune. Also trims comment density in both shell scripts and reworks the README: promotes the parallel and cudagraph sections out from under the A/B flow, groups the env table by purpose, documents that two record-mode runs are not comparable, and marks ab_fold_test.sh as a caliber example whose switch upstream has already removed.
41 lines
1.8 KiB
Bash
Executable File
41 lines
1.8 KiB
Bash
Executable File
#!/bin/bash
|
||
# A/B 对比口径示例：4 个 trace shape 集 × 开关 0/1，共 8 轮。
|
||
#
|
||
# 口径：每个 shape 集先跑基线（record autotune config），再用 REPLAY_FROM 回放
|
||
# 同一套 config 跑对照侧，两侧只差被测开关。对比其它开关/算子时改循环变量即可。
|
||
#
|
||
# 注意：对比轴 FLAGGEMS_MXFP4_FOLDSCALE 属于 FlagGems，已被上游移除。开关不存在
|
||
# 时本脚本仍会跑完并输出完整表格，但两侧执行同一份代码——套用前先确认对比轴有效
|
||
# （grep 该开关名于 $FLAGGEMS_DIR/src 应有命中）。详见 README。
|
||
#
|
||
# 输出：runs/ab_fold_<时间戳>.log（汇总）+ 各轮 runs/<op>_<时间戳>/
|
||
set -uo pipefail
|
||
|
||
cd "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||
|
||
# tee 管道会让子脚本检测不到 tty，故在此判断后下传（同 run_pytest.sh 的做法）
|
||
if [[ -t 1 && -z "${NO_COLOR:-}" ]]; then
|
||
export FLAGGEMS_PERF_COLOR=always
|
||
fi
|
||
|
||
LOG="runs/ab_fold_$(date +%Y%m%d-%H%M%S).log"
|
||
|
||
{
|
||
for y in shape/DeepSeek-V4-Flash-p{1024,4096,32768,65536}d1024.yaml; do
|
||
echo "########## $y | FOLD=0 baseline (record) ##########"
|
||
SHAPE_FILE="$y" USE_FLAGTUNE=0 FLAGGEMS_MXFP4_FOLDSCALE=0 bash run_pytest.sh \
|
||
|| { echo "!!! FAILED fold=0: $y"; continue; }
|
||
REC=$(ls -dt runs/fused_marlin_moe_mxfp4_* | head -1)
|
||
echo "########## $y | FOLD=1 (replay $REC) ##########"
|
||
SHAPE_FILE="$y" USE_FLAGTUNE=0 FLAGGEMS_MXFP4_FOLDSCALE=1 REPLAY_FROM="$REC" bash run_pytest.sh \
|
||
|| echo "!!! FAILED fold=1: $y"
|
||
done
|
||
echo "########## ALL DONE ##########"
|
||
} 2>&1 | tee "$LOG"
|
||
|
||
# 去掉 ANSI 转义以便 grep/diff（终端输出保留颜色）
|
||
sed -i -E $'s/\x1b\\[[0-9;]*[A-Za-z]//g' "$LOG"
|
||
|
||
echo ">>> 总 log: $(pwd)/$LOG"
|
||
echo ">>> 快速对账: grep -E '^#####|FAILED' $LOG"
|