forked from zhoulin/zl_bench
fix: never let the parallel sweep cover only part of the shape set
The sharded autotune sweep could publish a config set with holes, and a run using it reports itself as replaying while every uncovered key silently self-tunes. That is the exact failure the record/replay caliber exists to prevent, and it was reachable two ways. Shape lookup was op_name-only, but BLAS operators keep their shapes under the class name: core_shapes.yaml has BlasBenchmark, not mm. Every BLAS operator therefore resolved to zero shapes and the plugin disabled itself with a message that read like a normal skip. It now walks the benchmark class chain the way FlagGems' set_shapes does. The chain is recovered statically -- the class comes from the op_name= keyword of the constructor call in the test module, its bases from that module plus benchmark/base.py -- because the plugin runs at pytest_configure, before any Benchmark instance exists. Resolved keys are named in the log, and a total miss lists the candidates it tried. Two cases can still produce partial coverage, and both now refuse rather than shard a subset: - --level comprehensive, where set_shapes() merges set_more_shapes() on top of the yaml. That addition is arbitrary per-subclass Python and invisible here. Measured on mm: 5 of 29 shapes swept, 90 AUTOTUNE_REPLAY_FALLBACK reason=key_missing in the run that consumed the result. - a shard exiting non-zero, whose shapes were never swept. This used to append a WARNING to a success message and publish the merge anyway; observed with 3 of 5 shards dead. The whole sweep is now discarded, worker logs kept. Both fall back to the normal serial sweep, which costs wall time rather than the same-config premise. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
1 parent
92dae82c7f
commit
783c0655bc
2 files changed
+164
-17
No files matched your search
@@ -179,7 +179,17 @@ PARALLEL_WARMUP_GPUS=8 SHAPE_FILE=shape/DeepSeek-V4-Flash-p32768d1024.yaml bash
|
||||
WARNING 1 config key(s) got different winners across shards — parallel interference reached the sweep
|
||||
```
|
||||
|
||||
**冲突数就是这次加速的可信度指标**:0 可放心用;偏多说明 config 选择已被污染,出正式结论前不设该变量重跑一遍。其余行为(`REPLAY_FROM` 已设或可见卡不足 2 张时跳过、分片失败回退串行、轮转分片而非按块切、用子进程而非 xdist 的原因)见插件 docstring。分片绑卡遵守当前后端的可见性变量(CUDA 为 `CUDA_VISIBLE_DEVICES`)——用它选过卡时,分片只会落在你选的那几张上。
|
||||
**冲突数就是这次加速的可信度指标**:0 可放心用;偏多说明 config 选择已被污染,出正式结论前不设该变量重跑一遍。
|
||||
|
||||
插件会在三种情况下**主动放弃分片、退回串行 sweep**,日志各有独立措辞,都不是静默的:
|
||||
|
||||
- **`BENCH_LEVEL` 不是 `core`**。`comprehensive` 会让上游在 `set_shapes()` 里并入 `set_more_shapes()`,那部分是各子类自己的 Python(硬编码元组加 `models_shapes/*.yaml`),插件只看得到 yaml 里的 shape 列表。按子集分片会让未覆盖的 key 全部现场自调优,而这一轮却自称在 replay——`mm` 上实测是 29 个 shape 只扫到 5 个、90 次 `AUTOTUNE_REPLAY_FALLBACK reason=key_missing`。
|
||||
- **任一分片非零退出**。死掉的分片的 shape 从未被扫过,合并出来的 config 集是带洞的,publish 之后每个洞都会静默自调优。因此整轮 sweep 直接作废,worker 日志保留在临时目录里备查。
|
||||
- yaml 里查不到该算子的 shape(见下)、`REPLAY_FROM` 已设、或可见卡不足 2 张。
|
||||
|
||||
yaml key 的查找与上游对齐:先按 `op_name`,查不到再沿 Benchmark 类链逐级尝试类名(`mm` 因此命中 `BlasBenchmark`)。类链是静态解析测试模块里 `op_name=` 那次构造调用得到的,不 import FlagGems。日志会打出命中的 key;一个都没命中时会列出尝试过的候选。
|
||||
|
||||
其余行为(轮转分片而非按块切、用子进程而非 xdist 的原因)见插件 docstring。分片绑卡遵守当前后端的可见性变量(CUDA 为 `CUDA_VISIBLE_DEVICES`)——用它选过卡时,分片只会落在你选的那几张上。
|
||||
|
||||
## 批量多算子测试(run_batch.py)
|
||||
|
||||
|
||||
Reference in new issue
Block a user