Add opt-in multi-GPU sweep parallelism; fix GPU pinning and run dir races
For shape-heavy ops the autotune sweep dominates wall time, not the measurement. _parallel_warmup_plugin shards the sweep across N GPUs and hands the merged configs to the normal single-GPU serial measurement, so timing stays comparable. Activated by PARALLEL_WARMUP_GPUS=N (silent no-op when unset, so it can stay in the plugin list); output layout, REPLAY_FROM usage and the latency table are unchanged. Measurement is deliberately not parallelized: N processes saturating one box couple through the power/thermal budget, so per-card latency gets dragged by its neighbours by an amount that does not reproduce. Config keys whose winner differs across shards are counted and reported as a WARNING -- that count is how much to trust the run. Fixes found while auditing: - Shards no longer overwrite CUDA_VISIBLE_DEVICES with a bare shard index, which a caller who had selected idle cards (e.g. 6,7) would see re-interpreted as absolute ids 0,1 -- silently benchmarking on the busy cards they were avoiding. - Interrupting the sweep no longer leaves N subprocesses holding GPUs; children are terminated and reaped before the exception propagates (BaseException, since KeyboardInterrupt is not an Exception). - run_pytest.sh claims its output dir with a bare mkdir and retreats to a -2/-3 suffix on collision. The second-resolution timestamp meant two concurrent runs shared one directory and overwrote each other. - Replay eviction failures now emit a distinct compile_*_no_evict marker. LibTuner.cache is a sqlite-backed ConfigCache with no __delitem__, so the bad config could not be dropped and the retry re-read it, while the log still claimed a clean fallback to live autotune. Also trims comment density in both shell scripts and reworks the README: promotes the parallel and cudagraph sections out from under the A/B flow, groups the env table by purpose, documents that two record-mode runs are not comparable, and marks ab_fold_test.sh as a caliber example whose switch upstream has already removed.
This commit is contained in:
+12
-10
@@ -1,17 +1,19 @@
|
||||
#!/bin/bash
|
||||
# A/B 测试参考示例:FLAGGEMS_MXFP4_FOLDSCALE(fold_scale)开关验证
|
||||
# (4 个 DeepSeek-V4-Flash trace yaml × FOLD=0/1 共 8 轮)。
|
||||
# 口径:USE_FLAGTUNE=0;FOLD=0(基线)先跑并 record autotune config,FOLD=1 用
|
||||
# REPLAY_FROM 回放同一套 config(两侧同配置比 wall-time)。
|
||||
# 对比其它开关/算子时,改循环变量与环境变量即可复用同一口径。
|
||||
# 全部输出汇总到 runs/ab_fold_<时间戳>.log;各轮 runs/<op>_<时间戳>/ 产物照常存档。
|
||||
# 用法:bash ab_fold_test.sh
|
||||
# A/B 对比口径示例:4 个 trace shape 集 × 开关 0/1,共 8 轮。
|
||||
#
|
||||
# 口径:每个 shape 集先跑基线(record autotune config),再用 REPLAY_FROM 回放
|
||||
# 同一套 config 跑对照侧,两侧只差被测开关。对比其它开关/算子时改循环变量即可。
|
||||
#
|
||||
# 注意:对比轴 FLAGGEMS_MXFP4_FOLDSCALE 属于 FlagGems,已被上游移除。开关不存在
|
||||
# 时本脚本仍会跑完并输出完整表格,但两侧执行同一份代码——套用前先确认对比轴有效
|
||||
# (grep 该开关名于 $FLAGGEMS_DIR/src 应有命中)。详见 README。
|
||||
#
|
||||
# 输出:runs/ab_fold_<时间戳>.log(汇总)+ 各轮 runs/<op>_<时间戳>/
|
||||
set -uo pipefail
|
||||
|
||||
cd "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
# 终端颜色:下方 tee 管道会让 run_pytest.sh 侧检测不到 tty,在这里先判断并下传
|
||||
# (与 run_pytest.sh 自身的做法一致);总 log 最后统一去色。
|
||||
# tee 管道会让子脚本检测不到 tty,故在此判断后下传(同 run_pytest.sh 的做法)
|
||||
if [[ -t 1 && -z "${NO_COLOR:-}" ]]; then
|
||||
export FLAGGEMS_PERF_COLOR=always
|
||||
fi
|
||||
@@ -31,7 +33,7 @@ LOG="runs/ab_fold_$(date +%Y%m%d-%H%M%S).log"
|
||||
echo "########## ALL DONE ##########"
|
||||
} 2>&1 | tee "$LOG"
|
||||
|
||||
# 总 log 去掉 ANSI 转义,保证 grep/diff 面对纯文本(终端输出保留颜色)
|
||||
# 去掉 ANSI 转义以便 grep/diff(终端输出保留颜色)
|
||||
sed -i -E $'s/\x1b\\[[0-9;]*[A-Za-z]//g' "$LOG"
|
||||
|
||||
echo ">>> 总 log: $(pwd)/$LOG"
|
||||
|
||||
Reference in New Issue
Block a user