Add multi-op batch screening layer; fix cudagraph fallback regressions
run_batch.py drives run_pytest.sh per operator across GPUs (stable-hash sharding, per-op subprocess isolation, process-group timeouts, retry with deterministic-failure cutoff, two-level dtype fallback, .complete resume, per-op REPLAY_FROM). batch_summary.py aggregates run.log tables into summary.csv. ops/ holds the curated assets: dual-repo inventories rebuilt via AST scan + pytest collect verification, shape sets migrated from the old regression harness and merged with upstream core_shapes class-name keys (upstream's set_shapes falls back op_name -> MRO class name -> 1-D DEFAULT_SHAPES, so replacing the shape file without class keys crashes the BLAS family), and a dismiss list where all 76 entries carry verified reasons. Validated end to end: 1036-op full screen with zero failures. Also fix two cudagraph plugin regressions: newer torch appends "enable device-side assertions" to every CUDA error, so the loose fatal-error marker disabled the documented do_bench fallback entirely; and an aborted graph capture can leave the default CUDA RNG generator stuck in capturing state, poisoning every later torch.randn - captures now run under a throwaway RNG state. run_pytest.sh gains an optional DTYPES passthrough.
This commit is contained in:
+120
@@ -0,0 +1,120 @@
|
||||
# 批量测试排除清单。每行一个算子：`op`（默认 main 仓）或 `repo:op`。
|
||||
# run_batch.py --all 会跳过这里列出的算子；带无条件 pytest.mark.skip 的算子由
|
||||
# inventory 的 marked_skip 列自动剔除，不必写在这里。
|
||||
# 只放"经本仓批量跑复核确认失败"的条目，注明原因与复核日期。
|
||||
#
|
||||
# 2026-08-12 复核记录：
|
||||
# - 旧回归清单 24 个候选全部重验：14 个新上游已修复（移出），3 个系本仓
|
||||
# _cudagraph_plugin 回退失效 bug（已修复，移出），7 个确认为上游/环境问题；
|
||||
# - 全量筛查（1103 算子）新增确认 67 个：43 个 torch jiterator 环境问题 +
|
||||
# 24 个上游 bug/环境限制。shape 语义类失败（28 个）已通过 shape yaml 合并
|
||||
# 上游 core_shapes 类名键修复，不在此列；
|
||||
# - test_FLA/ 补齐后新增确认 2 个（硬件限制 + 专用 harness），共 76 条。
|
||||
|
||||
# ==== 上游 kernel 编译错误 [2026-08-12]
|
||||
# CompilationError（关 cudagraph 后仍复现）
|
||||
hstack
|
||||
scaled_mm_benchmark
|
||||
scaled_mm_out_benchmark
|
||||
# tt.load result/ptr type 校验失败
|
||||
flash_mla_with_kvcache
|
||||
# PassManager::run failed（FlagTree triton 编译失败）
|
||||
flash_mla_sparse_fwd
|
||||
vllm:flash_mla_sparse_fwd
|
||||
# capture 失败回退后仍报 kernel 错误
|
||||
vllm:flash_mla
|
||||
vllm:flash_mla_with_kvcache
|
||||
linalg_lu_factor_ex
|
||||
|
||||
# ==== 上游 benchmark 代码 bug [2026-08-12]
|
||||
# baseline torch.trace 收到非矩阵输入（expected a matrix）
|
||||
trace
|
||||
# set_shapes() 签名与基类调用不符（TypeError）
|
||||
get_paged_mqa_logits_metadata
|
||||
# 输入解包 ValueError（expected 3, got 1）
|
||||
grouped_mm
|
||||
# 上游 benchmark assert 失败
|
||||
get_scheduler_metadata
|
||||
# is_backward 对未参与计算图的张量求梯度（allow_unused）
|
||||
embedding_backward
|
||||
margin_ranking_loss_backward
|
||||
# baseline 函数签名不符（unexpected keyword 'q' / 缺 constexpr 实参）
|
||||
fp8_fp4_paged_mqa_logits
|
||||
vllm:fp8_fp4_mqa_logits
|
||||
# baseline 返回 None（NoneType has no attribute 'device'）
|
||||
special_gammainc
|
||||
# normalized_shape=[]（上游默认 shape 语义错误）
|
||||
native_layer_norm
|
||||
# 上游断言 K>=16 与其默认 attention shape 冲突
|
||||
perf_scaled_dot_product_flash_attention_backward
|
||||
perf_scaled_dot_product_cudnn_attention_backward
|
||||
# Triton Error [CUDA]: invalid argument（上游默认 shape 下 kernel 启动参数非法）
|
||||
index_select_backward
|
||||
pairwise_distance
|
||||
# test_blas_perf_parallel 专用 harness（shape kind 语义与通用注入冲突；常规 mul 已覆盖）
|
||||
perf_mul
|
||||
# 需要 pytest-benchmark fixture（非 FlagGems Benchmark 体系）
|
||||
vllm:triton_unified_attention_perf
|
||||
# 自定义输出格式（非 FlagGems Benchmark 表）且预编译超 40min
|
||||
vllm:perf_chunk_gla
|
||||
|
||||
# ==== 硬件限制 [2026-08-12]
|
||||
# kernel 共享内存需求 245-335KB 超 H20 上限 227KB，autotune 全候选 OutOfResources
|
||||
vllm:chunk_gdn2
|
||||
|
||||
# ==== 环境限制（换环境后应重验）[2026-08-12]
|
||||
# vllm 0.20.2 无 vllm.utils.deep_gemm.fp8_mqa_logits
|
||||
fp8_mqa_logits
|
||||
# FA2 不支持 num_splits > 1（flash-attn 版本）
|
||||
flash_attn_varlen_opt_func
|
||||
# magma 显存分配失败（cannot allocate memory on GPU, info=-113）
|
||||
cholesky_solve
|
||||
cholesky_solve_out
|
||||
# cusolver INTERNAL_ERROR（Xgeev）
|
||||
linalg_eigvals
|
||||
# ---- torch jiterator/NVRTC 在本环境编译失败（fp32 也复现），
|
||||
# ---- sinc/bessel/special 族的 torch baseline 整体不可用：
|
||||
erfc
|
||||
erfc_
|
||||
erfinv
|
||||
erfinv_
|
||||
lcm
|
||||
lcm_
|
||||
lgamma
|
||||
lgamma_
|
||||
mvlgamma
|
||||
mvlgamma_
|
||||
polygamma
|
||||
polygamma_inplace
|
||||
polygamma_out
|
||||
sinc
|
||||
sinc_
|
||||
special_airy_ai
|
||||
special_airy_ai_out
|
||||
special_bessel_j0
|
||||
special_bessel_j1
|
||||
special_chebyshev_polynomial_u
|
||||
special_chebyshev_polynomial_v
|
||||
special_chebyshev_polynomial_w
|
||||
special_chebyshev_polynomial_w_out
|
||||
special_erfc
|
||||
special_erfcx
|
||||
special_erfinv
|
||||
special_erfinv_out
|
||||
special_gammaln
|
||||
special_gammaln_out
|
||||
special_hermite_polynomial_h
|
||||
special_i1e
|
||||
special_legendre_polynomial_p
|
||||
special_log1p_non_tensor
|
||||
special_modified_bessel_k0
|
||||
special_modified_bessel_k0_out
|
||||
special_modified_bessel_k1
|
||||
special_modified_bessel_k1_out
|
||||
special_scaled_modified_bessel_k1
|
||||
special_scaled_modified_bessel_k1_out
|
||||
special_shifted_chebyshev_polynomial_u
|
||||
special_shifted_chebyshev_polynomial_u_
|
||||
special_shifted_chebyshev_polynomial_v
|
||||
special_shifted_chebyshev_polynomial_w
|
||||
special_sinc
|
||||
Reference in New Issue
Block a user