Add multi-op batch screening layer; fix cudagraph fallback regressions

run_batch.py drives run_pytest.sh per operator across GPUs (stable-hash
sharding, per-op subprocess isolation, process-group timeouts, retry with
deterministic-failure cutoff, two-level dtype fallback, .complete resume,
per-op REPLAY_FROM). batch_summary.py aggregates run.log tables into
summary.csv. ops/ holds the curated assets: dual-repo inventories rebuilt
via AST scan + pytest collect verification, shape sets migrated from the
old regression harness and merged with upstream core_shapes class-name
keys (upstream's set_shapes falls back op_name -> MRO class name ->
1-D DEFAULT_SHAPES, so replacing the shape file without class keys
crashes the BLAS family), and a dismiss list where all 76 entries carry
verified reasons. Validated end to end: 1036-op full screen with zero
failures.

Also fix two cudagraph plugin regressions: newer torch appends "enable
device-side assertions" to every CUDA error, so the loose fatal-error
marker disabled the documented do_bench fallback entirely; and an aborted
graph capture can leave the default CUDA RNG generator stuck in capturing
state, poisoning every later torch.randn - captures now run under a
throwaway RNG state. run_pytest.sh gains an optional DTYPES passthrough.
This commit is contained in:
2026-08-12 19:04:09 +00:00
parent 95895cea0e
commit da22885645
12 changed files with 14679 additions and 9 deletions
+11 -1
View File
@@ -42,6 +42,10 @@ YAML
# 调优空间:0=普通 autotune(默认,快速验证);1=FlagTune 扩展空间(首跑全量搜索、慢)
USE_FLAGTUNE="${USE_FLAGTUNE:-0}"
# 可选:限制 dtype 集(空格分隔,如 "bfloat16 float16")。空=上游默认 dtype 扫描。
# 算子不支持指定 dtype 时上游会报 "can't be supported by this op"(批量驱动据此降级重试)。
DTYPES="${DTYPES:-}"
# 空=record 模式,把本次选中的 config 记入 autotune_records/<op>.json
# 指向某次历史 run 目录则 replay 其记录,用于 A/B 两侧锁同一套 config。
REPLAY_FROM="${REPLAY_FROM:-}"
@@ -104,6 +108,12 @@ export PYTHONPATH="$SCRIPT_DIR${PYTHONPATH:+:$PYTHONPATH}"
export FLAGGEMS_PERF_CURRENT_OP="$OP"
export PYTHONUNBUFFERED=1 # 实时输出不缓冲
# DTYPES 非空时逐个转为上游 --dtypes 选项(action=append,每个 dtype 一次)
DTYPE_ARGS=()
for _dt in $DTYPES; do
DTYPE_ARGS+=(--dtypes "$_dt")
done
# record/replay 互斥,各由自己的环境变量激活
if [[ -n "$REPLAY_FROM" ]]; then
AUTOTUNE_ENV="FLAGGEMS_PERF_AUTOTUNE_REPLAY_DIR=$REPLAY_FROM/autotune_records"
@@ -139,7 +149,7 @@ status=0
env "$AUTOTUNE_ENV" \
USE_FLAGTUNE=$USE_FLAGTUNE python -u -m pytest -s "$TEST_FILE" \
"${PLUGINS[@]}" "${PYTEST_COLOR[@]}" \
--shape_file "$SHAPE_FILE" \
--shape_file "$SHAPE_FILE" ${DTYPE_ARGS[@]+"${DTYPE_ARGS[@]}"} \
--level core --mode kernel || status=$?
rm -rf "$CACHE_DIR"