Add multi-op batch screening layer; fix cudagraph fallback regressions
run_batch.py drives run_pytest.sh per operator across GPUs (stable-hash sharding, per-op subprocess isolation, process-group timeouts, retry with deterministic-failure cutoff, two-level dtype fallback, .complete resume, per-op REPLAY_FROM). batch_summary.py aggregates run.log tables into summary.csv. ops/ holds the curated assets: dual-repo inventories rebuilt via AST scan + pytest collect verification, shape sets migrated from the old regression harness and merged with upstream core_shapes class-name keys (upstream's set_shapes falls back op_name -> MRO class name -> 1-D DEFAULT_SHAPES, so replacing the shape file without class keys crashes the BLAS family), and a dismiss list where all 76 entries carry verified reasons. Validated end to end: 1036-op full screen with zero failures. Also fix two cudagraph plugin regressions: newer torch appends "enable device-side assertions" to every CUDA error, so the loose fatal-error marker disabled the documented do_bench fallback entirely; and an aborted graph capture can leave the default CUDA RNG generator stuck in capturing state, poisoning every later torch.randn - captures now run under a throwaway RNG state. run_pytest.sh gains an optional DTYPES passthrough.
This commit is contained in:
+11
-1
@@ -42,6 +42,10 @@ YAML
|
||||
# 调优空间:0=普通 autotune(默认,快速验证);1=FlagTune 扩展空间(首跑全量搜索、慢)
|
||||
USE_FLAGTUNE="${USE_FLAGTUNE:-0}"
|
||||
|
||||
# 可选:限制 dtype 集(空格分隔,如 "bfloat16 float16")。空=上游默认 dtype 扫描。
|
||||
# 算子不支持指定 dtype 时上游会报 "can't be supported by this op"(批量驱动据此降级重试)。
|
||||
DTYPES="${DTYPES:-}"
|
||||
|
||||
# 空=record 模式,把本次选中的 config 记入 autotune_records/<op>.json;
|
||||
# 指向某次历史 run 目录则 replay 其记录,用于 A/B 两侧锁同一套 config。
|
||||
REPLAY_FROM="${REPLAY_FROM:-}"
|
||||
@@ -104,6 +108,12 @@ export PYTHONPATH="$SCRIPT_DIR${PYTHONPATH:+:$PYTHONPATH}"
|
||||
export FLAGGEMS_PERF_CURRENT_OP="$OP"
|
||||
export PYTHONUNBUFFERED=1 # 实时输出不缓冲
|
||||
|
||||
# DTYPES 非空时逐个转为上游 --dtypes 选项(action=append,每个 dtype 一次)
|
||||
DTYPE_ARGS=()
|
||||
for _dt in $DTYPES; do
|
||||
DTYPE_ARGS+=(--dtypes "$_dt")
|
||||
done
|
||||
|
||||
# record/replay 互斥,各由自己的环境变量激活
|
||||
if [[ -n "$REPLAY_FROM" ]]; then
|
||||
AUTOTUNE_ENV="FLAGGEMS_PERF_AUTOTUNE_REPLAY_DIR=$REPLAY_FROM/autotune_records"
|
||||
@@ -139,7 +149,7 @@ status=0
|
||||
env "$AUTOTUNE_ENV" \
|
||||
USE_FLAGTUNE=$USE_FLAGTUNE python -u -m pytest -s "$TEST_FILE" \
|
||||
"${PLUGINS[@]}" "${PYTEST_COLOR[@]}" \
|
||||
--shape_file "$SHAPE_FILE" \
|
||||
--shape_file "$SHAPE_FILE" ${DTYPE_ARGS[@]+"${DTYPE_ARGS[@]}"} \
|
||||
--level core --mode kernel || status=$?
|
||||
|
||||
rm -rf "$CACHE_DIR"
|
||||
|
||||
Reference in New Issue
Block a user