forked from zhoulin/zl_bench
feat: check the run's preconditions, and record which tuner cache it used
Two failure modes that cost this harness real results, both of them silent. A wrong path does not fail as a wrong path. pytest cannot collect a benchmark directory that is not there, so benchmark/conftest.py never registers --level and --mode, and the run dies with "unrecognized arguments: --level core --mode kernel" -- an error about the flags, several layers away from the missing directory. run_pytest.sh now checks the FlagGems directory, the benchmark file, SHAPE_FILE and REPLAY_FROM before starting pytest, prints "PREFLIGHT_FAILED check=<the named check>" and exits 3, distinct from pytest's own 1 and 4 so the batch layer can tell a broken environment from a broken operator. Only the file half of a node is checked; a missing ::test_<fn> already produces a clear "not found: ..." from pytest and needs no help. FlagGems keeps libtuner winners in a sqlite DB under $FLAGGEMS_CACHE_DIR (default ~/.flaggems), shared by every run on the machine. Whichever side of an A/B runs first therefore hands its winners to the second, which then compiles only those: measured on router_gemm, the second of two shared-cache runs compiled 10 variants instead of 234 and finished in 17s instead of 52s. That is how an mm A/B in this repo produced "945 variants -> 33, 136 converts -> 10" and read as a compiler result when it was cache inheritance; isolated, both runs compile 234. ZL_ISOLATE_TUNER_CACHE=1 gives a run its own cache directory, dropped afterwards with the Triton one since the chosen configs are already in autotune_records/. It defaults to 0, because cold tuning is expensive and the standard REPLAY_FROM A/B pins configs anyway; only record-against-record is exposed. The default is unchanged, but the caliber is no longer invisible: either way it goes into the run header and run_identity.json as tuner_cache, which is what was missing when the mm numbers could not be explained after the fact. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
1 parent
ac818291a6
commit
c9f9b7885f
3 files changed
+68
-3
No files matched your search
@@ -71,6 +71,10 @@ def main(argv=None) -> int:
|
||||
p.add_argument("--test-file", default=os.environ.get("TEST_FILE", ""))
|
||||
p.add_argument("--shape-mode", default=os.environ.get("SHAPE_MODE", ""))
|
||||
p.add_argument("--bench-level", default=os.environ.get("BENCH_LEVEL", "core"))
|
||||
# "isolated" or "shared(<path>)". Recorded because a shared libtuner cache
|
||||
# lets whichever side runs first hand its winners to the other, and after
|
||||
# the fact the two runs are otherwise indistinguishable.
|
||||
p.add_argument("--tuner-cache", default="")
|
||||
args = p.parse_args(argv)
|
||||
init_identity(
|
||||
args.out_dir,
|
||||
@@ -83,6 +87,7 @@ def main(argv=None) -> int:
|
||||
test_file=args.test_file,
|
||||
shape_mode=args.shape_mode,
|
||||
bench_level=args.bench_level,
|
||||
tuner_cache=args.tuner_cache,
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
Reference in new issue
Block a user