feat: expose the upstream benchmark level as BENCH_LEVEL

run_pytest.sh hardcoded --level core, so the only shapes ever reachable were
each operator's core_shapes entry. FlagGems merges a subclass's
set_more_shapes() on top of that at --level comprehensive, and for BLAS that
addition is where the large GEMMs live: mm goes from 5 shapes to 29, taking in
four large-K tuples plus models_shapes/*.yaml, with N/K up to 152064. The
comprehensive path also yields each shape a second time (transposed, for BLAS),
so the case count doubles. None of that was reachable without editing the driver.

BENCH_LEVEL defaults to core, so existing runs are unchanged, and rejects
anything other than core/comprehensive rather than passing it through to pytest.
The chosen level is echoed in the run header, written into the archived
shapes.yaml, and recorded in run_identity.json, because an A/B whose two sides
used different levels is not comparable and is otherwise invisible afterwards.

Sweep cost scales with the shape count, so comprehensive on a BLAS operator
wants PARALLEL_WARMUP_GPUS.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
AharrypotterandClaude Opus 5 committed 2026-09-09 16:39:45 +08:00
1 parent c8ea586d0f
commit 92dae82c7f
3 files changed
+21 -6

No files matched your search

+2
View File
@@ -70,6 +70,7 @@ def main(argv=None) -> int:
p.add_argument("--device", default=os.environ.get("ZL_DEVICE", ""))
p.add_argument("--test-file", default=os.environ.get("TEST_FILE", ""))
p.add_argument("--shape-mode", default=os.environ.get("SHAPE_MODE", ""))
p.add_argument("--bench-level", default=os.environ.get("BENCH_LEVEL", "core"))
args = p.parse_args(argv)
init_identity(
args.out_dir,
@@ -81,6 +82,7 @@ def main(argv=None) -> int:
device=args.device,
test_file=args.test_file,
shape_mode=args.shape_mode,
bench_level=args.bench_level,
)
return 0