forked from zhoulin/zl_bench
c95474a432725e71cb2e19339060547ca5f90015
13
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  AharrypotterandClaude Opus 5
|
c95474a432 |
bench: refuse to report a comparison whose two sides have different calibers
Two runs on different calibers produce a report that looks exactly like a valid one — no step errors, the numbers just mean something else. This repo has published three figures that were an artifact of such a difference: router_gemm -8.2% (record vs record), mm "945 variants -> 33" (shared vs inherited libtuner cache), replication_pad3d +43% vs +124.5% (timer). batch_summary --ab / --ab-rounds now reads every participating run's run_identity.json and compares backend/device/timer/bench_level/shape_mode/ tuner_cache/direct_repeat before parsing a single latency. Mismatch, or a missing identity, prints !!! [caliber] and exits 4. autotune_mode is judged separately: record pinning a replay is the normal shape, both sides record is the exact trap above and is refused by name. --allow-caliber-mismatch releases it, still printing what it found. Recording on a *shared* tuner cache warns rather than refuses: the pinned config depends on what else ran on the machine, which leaves relative conclusions intact and absolute latencies unquotable. Refusing it would reject the default every A/B here currently uses. ab_repeat.sh closes the one axis that had no check: after round 1 it fingerprints both sides' ttgir (relative path + content, so a changed variant *set* counts too). Byte-identical means the switch changed no generated code for this op, and the remaining N-1 rounds would measure noise and report it as "no difference" — so the op is skipped with SWITCH_NO_IR_DIFF unless SWITCH_ALLOW_NO_IR_DIFF=1. README leads with a caliber table: each axis, its measured magnitude, what pins it, and which code checks it. The fourth column is code because documentation did not hold — the README already said "the B side must use REPLAY_FROM" when the record-vs-record number was published. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
AharrypotterandClaude Opus 5
|
aa147c938e |
feat: repeat an A/B over rounds and report whether the verdict holds
A single pair cannot say whether its own verdict is stable, and on this device that is not a theoretical gap. Measured on an S5000, topk eliminated 32 convert_layout ops yet 17 of its 42 shapes changed the SIGN of their delta between two equally valid runs. The cause is not the timer: graph capture and direct timing agree to 1.00x on that operator, and the verdict moved a median of 1.1pp between them, inside the 1.7pp noise floor a same-timer control established. The deltas are simply near zero, and one measurement cannot orient them. ab_repeat.sh runs an A/B over N rounds in independent processes. Round 1's A side records the autotune configs and every later run -- including A's own later rounds -- replays that one set, so configs are pinned across rounds and sides and only the switch under test and measurement noise remain. Round order alternates to cancel one-directional drift. An operator with a failed round gets no report, since a report missing rounds reads exactly like a complete one. batch_summary --ab-rounds aggregates <root>/r<k>_<A|B>/run.log and separates the two dispersions, which answer different questions and were previously conflated: the spread within one side's samples says how noisy the machine was, while the spread of the per-round deltas says whether the verdict holds. Shapes whose delta changes sign across rounds are counted and called out, because their direction is not a conclusion at any timer. The path is already validated on NVIDIA: six rounds narrowed router_gemm to -16.6% ~ -16.2% with 0.4% side noise, which is what turned a single pair into a statement. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
AharrypotterandClaude Opus 5
|
9665d36a7c |
feat: allow the direct timer to repeat and take the median
do_bench already returns a median over the iterations inside one call, which suppresses iteration-level jitter. It cannot see drift across a call: clocks ramping, another process arriving. Direct timing is where that shows, because it walks the full Python launch path once per iteration, so on small kernels the number carries launch overhead and CPU jitter that graph capture removes. ZL_DIRECT_REPEAT=N repeats the call and takes the median over calls, which is the layer the inner median misses. It defaults to 1, so nothing changes unless asked. The knob covers both ways a row ends up on the direct timer: a run that selected timer=direct outright, and the rows a captured run falls back on, whose marker now carries repeat=N since those rows were already a different caliber from the captured ones and can now also be a different sample count. Repetition is skipped when the caller asks for quantiles or return_mode "all", because the result is then a sequence and a median across calls is undefined for it. On MThreads this is not a refinement. A convert-reduction screen over 21 operators puts the largest reductions at sort (65->15), dropout (48->0) and rand (24->0), and all three are exactly the operators that cannot be graph-captured there -- RNG seeding and a capture-illegal stream op. The headline numbers on that device are produced by the noisiest timer, which until now had no repetition at all. _cudagraph_plugin imports triton at module scope, so the tests load it against a stub: the repeat logic is arithmetic and should stay testable without a GPU. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
AharrypotterandClaude Opus 5
|
d7cb7efc94 |
fix: have the campaign worker ask zl_bench for its plugin list
The worker carried a hand-copied plugin array, taken from zl_bench before backends/ existed, and it had drifted twice over. It still named _device_guard_plugin, which is now a shim for _device_setup_plugin. More consequentially it carried no timing plugin at all, so every measurement this campaign ever sealed was timer=direct while zl_bench's own driver had been capturing -- a difference the harness never reported, because the missing plugin cannot announce itself. It now calls plugin_plan.py with ZL_BACKEND=musa and uses whatever comes back, so the caliber follows the backend's declared capabilities instead of a snapshot of them. The plan is preflighted for non-emptiness, since an empty array would silently run pytest with no plugins at all: no shape injection, no autotune record, no IR dump, and a table that still looks well-formed. Two effects on this campaign. _timing_plugin now enters the list, and since MUSA declares graph capture the worker will capture where the operator allows it; operators that cannot are marked per row, as on any other backend. And _parallel_warmup_plugin joins, inert unless PARALLEL_WARMUP_GPUS is set. The sealed 32-workload run therefore sits on the far side of a caliber change and is not comparable with anything produced after this commit. Re-running it is a campaign decision, not a harness one. The worker test asserted the literal "-p _musa_benchmark_compat_plugin"; it now asserts the delegation and that no -p is spelled out locally. What the plan holds for musa is covered where it belongs, in zl_bench's test_backends. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
AharrypotterandClaude Opus 5
|
75405cc555 |
Merge the S5000 MUSA RLC product campaign
Brings in campaigns/musa_rlc_product/: the runner, its worker, manifest validation, the sealed 32-workload manifest, a topology re-accounting tool and their tests. Developed on a fork branch that never reached main, and it merges without conflict because it only adds a new directory; the one file it shares with main, _musa_benchmark_compat_plugin.py, is byte-identical to the copy already committed here. The campaign layer sits above zl_bench rather than inside it: freezing a workload set, classifying outcomes, and producing receipts, where zl_bench turns one operator into one reproducible measurement. Two of its ideas are worth naming because they are absent below it. Narrowing what a workload measures -- which correctness variant, which benchmark case -- requires a stated reason that manifest validation enforces, so an exclusion stays visible in review. And reaccount_musa_rlc_topology.py re-derives verdicts from sealed artifacts under a corrected accounting rule without re-running anything, while refusing to promote a workload on the new count, only to demote it. Its worker still hardcodes a plugin list copied from a pre-backends zl_bench, which is why the campaign has been timing with timer=direct. Switching it to plugin_plan.py is the next change and will alter its caliber, so the sealed run is not comparable across that boundary. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
AharrypotterandClaude Opus 5
|
29af59be9b |
feat: absorb the MThreads compat plugin and the A/B range report
Three pieces from the fork's feature/musa-benchmark-support branch that never reached main, rebased onto the backend abstraction rather than copied verbatim. _musa_benchmark_compat_plugin drops FlagGems benchmark skip guards written as `torch.cuda.is_available()` where the code under test dispatches through flag_gems.device. On MThreads those turn a working operator into "not measured": test_fft.py skipped entirely, and unskipped on an S5000 it produces 25 rows. The plugin pins each override to one file and one exact skip reason and only acts when flag_gems has dispatched to the mthreads vendor; two guards were probed on device and deliberately left in place, which is why they are named in its docstring rather than silently absent. It is wired through a new DeviceBackend.compat_plugins rather than a name test in the planner, so the policy stays where every other backend difference lives. Compat plugins are planned first because they rewrite collection-time markers and nothing may read the item list before them; a test pins that position and another pins that they do not leak to other backends. batch_summary gains --ab: the speedup column compares gems against torch, not one build against another, so an A/B needs the two run.logs paired by operator, dtype and size detail. Only rows that succeeded on both sides with a positive latency count, so an incomplete pairing cannot look like a complete range, and the range is annotated when either side carries a BENCHMARK_DIRECT_NO_CUDAGRAPH marker, since a fallback row was timed under a different regime. ab_rlc_test.sh is the same caliber as ab_fold_test.sh with the comparison axis moved to FLAGTREE_<VENDOR>_RLC_ENHANCE. Both are caliber examples, not regression scripts: if the switch under test no longer exists the script still runs every round and prints a full table showing no difference, so confirm the axis is live before reusing either. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
AharrypotterandClaude Opus 5
|
c9f9b7885f |
feat: check the run's preconditions, and record which tuner cache it used
Two failure modes that cost this harness real results, both of them silent. A wrong path does not fail as a wrong path. pytest cannot collect a benchmark directory that is not there, so benchmark/conftest.py never registers --level and --mode, and the run dies with "unrecognized arguments: --level core --mode kernel" -- an error about the flags, several layers away from the missing directory. run_pytest.sh now checks the FlagGems directory, the benchmark file, SHAPE_FILE and REPLAY_FROM before starting pytest, prints "PREFLIGHT_FAILED check=<the named check>" and exits 3, distinct from pytest's own 1 and 4 so the batch layer can tell a broken environment from a broken operator. Only the file half of a node is checked; a missing ::test_<fn> already produces a clear "not found: ..." from pytest and needs no help. FlagGems keeps libtuner winners in a sqlite DB under $FLAGGEMS_CACHE_DIR (default ~/.flaggems), shared by every run on the machine. Whichever side of an A/B runs first therefore hands its winners to the second, which then compiles only those: measured on router_gemm, the second of two shared-cache runs compiled 10 variants instead of 234 and finished in 17s instead of 52s. That is how an mm A/B in this repo produced "945 variants -> 33, 136 converts -> 10" and read as a compiler result when it was cache inheritance; isolated, both runs compile 234. ZL_ISOLATE_TUNER_CACHE=1 gives a run its own cache directory, dropped afterwards with the Triton one since the chosen configs are already in autotune_records/. It defaults to 0, because cold tuning is expensive and the standard REPLAY_FROM A/B pins configs anyway; only record-against-record is exposed. The default is unchanged, but the caliber is no longer invisible: either way it goes into the run header and run_identity.json as tuner_cache, which is what was missing when the mm numbers could not be explained after the fact. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
AharrypotterandClaude Opus 5
|
ac818291a6 |
feat: turn on graph capture for MUSA
supports_cuda_graph was False for MUSA, so every run there was pinned to
timer=direct. That was correct when it was written -- the capture path named
torch.cuda and could not have worked -- but it is not correct now, and the two
timers are not interchangeable: direct timing walks the full Python launch path
per iteration, so on small kernels it measures mostly launch overhead. Measured
on an S5000, capture against direct: batch_norm 16x16x8x48 0.0758ms -> 0.0089ms,
tanh and gelu at 4096x4096 about 3.3x, GEMMs unchanged at 1.00x. Leaving it off
discards that on every operator that can capture, and it also puts MUSA on a
different caliber from the CUDA numbers the harness is meant to compare against.
torch_musa supplies Stream, Event, synchronize, graph and MUSAGraph, so capture
works through the device interface. Verified on an S5000 over five operators,
one per path:
tanh captured, no marker
dropout 15/15 rows fall back: MUSAGeneratorImpl::current_seed is not
callable during capture
rand same
perf_sort 25/25 rows fall back: operation not permitted when stream is
capturing
unique2 denied up front; rc=0 with 10 rows
MUSA gets its own error wording; none of the CUDA markers match it. "MUSA
error: unknown error" during capture is classified as poisoning rather than
recoverable, because the fallback runs and the process then dies in teardown:
before the deny entry was keyed correctly, unique2 reproduced exactly that,
ending at rc=1 with no rows, but now under an explicit BENCHMARK_CAPTURE_POISONED
marker instead of a bare ExceptionGroup.
capture_deny_ops is keyed by FLAGGEMS_PERF_CURRENT_OP, which holds the pytest
FUNCTION name minus `test_` -- unique2, perf_sort -- not the benchmark file
name. Getting that wrong fails open silently, which is how the first attempt
still hit rc=1 with the entry written as "unique"; a test now pins the key
space. The plugin-plan test is renamed because its assertion that
_cudagraph_plugin is absent from the plan holds on CUDA too and says nothing
about capture support.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
AharrypotterandClaude Opus 5
|
6fa5f5d7a2 |
refactor: let any backend own graph capture and its failure modes
Two things kept graph-capture timing tied to NVIDIA, and both are mechanism rather than policy, so neither backend's behaviour changes here. The capture path named torch.cuda directly for stream, event, synchronize and the graph type. FlagTree's do_bench already resolves those through driver.active.get_device_interface(); its do_bench_cudagraph does not. On a non-CUDA device every capture therefore raised and every row silently took the fallback while the log still reported capture as enabled. The path now takes those calls from the interface, with _graph_class picking CUDAGraph or MUSAGraph. CUDA still calls upstream's do_bench_cudagraph verbatim, so its numbers do not move; the port only matters where upstream cannot run. Failure classification moves onto DeviceBackend, because the text it matches is vendor wording that has no meaning on another runtime, and gains a third verdict. A kernel refusing capture is ordinary and falls back. A kernel failing on the device is fatal and re-raises, since a fallback latency there would report a broken kernel as a measurement. New: the capture ATTEMPT can leave the context unusable, and falling back then measures inside a dead process -- the observed shape of that is a fallback marker followed by a teardown crash and zero rows. Those raise under BENCHMARK_CAPTURE_POISONED instead, naming the operator as capture-incompatible rather than broken. capture_deny_ops skips the attempt entirely for operators already known to do this, because for them the attempt is the damage and rediscovering it per shape costs the run. CUDA's marker list is carried over unchanged and no poisoning wording is guessed for it. The scratch-RNG guard around capture is skipped when an interface exposes no default_generators, since there is then no registration to protect. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
AharrypotterandClaude Opus 5
|
783c0655bc |
fix: never let the parallel sweep cover only part of the shape set
The sharded autotune sweep could publish a config set with holes, and a run using it reports itself as replaying while every uncovered key silently self-tunes. That is the exact failure the record/replay caliber exists to prevent, and it was reachable two ways. Shape lookup was op_name-only, but BLAS operators keep their shapes under the class name: core_shapes.yaml has BlasBenchmark, not mm. Every BLAS operator therefore resolved to zero shapes and the plugin disabled itself with a message that read like a normal skip. It now walks the benchmark class chain the way FlagGems' set_shapes does. The chain is recovered statically -- the class comes from the op_name= keyword of the constructor call in the test module, its bases from that module plus benchmark/base.py -- because the plugin runs at pytest_configure, before any Benchmark instance exists. Resolved keys are named in the log, and a total miss lists the candidates it tried. Two cases can still produce partial coverage, and both now refuse rather than shard a subset: - --level comprehensive, where set_shapes() merges set_more_shapes() on top of the yaml. That addition is arbitrary per-subclass Python and invisible here. Measured on mm: 5 of 29 shapes swept, 90 AUTOTUNE_REPLAY_FALLBACK reason=key_missing in the run that consumed the result. - a shard exiting non-zero, whose shapes were never swept. This used to append a WARNING to a success message and publish the merge anyway; observed with 3 of 5 shards dead. The whole sweep is now discarded, worker logs kept. Both fall back to the normal serial sweep, which costs wall time rather than the same-config premise. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
AharrypotterandClaude Opus 5
|
92dae82c7f |
feat: expose the upstream benchmark level as BENCH_LEVEL
run_pytest.sh hardcoded --level core, so the only shapes ever reachable were each operator's core_shapes entry. FlagGems merges a subclass's set_more_shapes() on top of that at --level comprehensive, and for BLAS that addition is where the large GEMMs live: mm goes from 5 shapes to 29, taking in four large-K tuples plus models_shapes/*.yaml, with N/K up to 152064. The comprehensive path also yields each shape a second time (transposed, for BLAS), so the case count doubles. None of that was reachable without editing the driver. BENCH_LEVEL defaults to core, so existing runs are unchanged, and rejects anything other than core/comprehensive rather than passing it through to pytest. The chosen level is echoed in the run header, written into the archived shapes.yaml, and recorded in run_identity.json, because an A/B whose two sides used different levels is not comparable and is otherwise invisible afterwards. Sweep cost scales with the shape count, so comprehensive on a BLAS operator wants PARALLEL_WARMUP_GPUS. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
AharrypotterandClaude Opus 5
|
c8ea586d0f |
feat: make the harness backend-agnostic
Everything about which accelerator a run targets was previously spelled CUDA: device discovery went through nvidia-smi, pinning through CUDA_VISIBLE_DEVICES, the vendor guard only recognised NVIDIA, and the -p plugin list was hardcoded in run_pytest.sh with no way to drop the CUDA-only members. Running the same operator corpus on another vendor's device therefore meant editing the drivers. The selection now hangs off one ZL_BACKEND variable (cuda/musa/rocm/ascend, with the usual aliases): - backends/ owns per-backend device enumeration, visibility env keys and capability flags, so nothing above it names a vendor; - plugin_plan.py derives the -p list from those capabilities instead of a hardcoded array, which is what lets a non-CUDA run omit the graph-capture and cluster-fix plugins rather than watch them fail; - _timing_plugin selects the timer from the same capabilities: CUDA keeps graph capture with its per-shape fallback, other backends stay on upstream do_bench and record timer=direct, because capture-then-fallback on a device without graph support only produces noise; - _device_setup_plugin replaces the NVIDIA-only vendor guard; - providers/flaggems.py resolves the pytest node, so a caller may pass TEST_NODE directly instead of the OP/OP_FILE pair, and SHAPE_MODE=upstream can defer to FlagGems' own core_shapes; - _run_identity.py writes run_identity.json next to each run, recording backend, device, timer, autotune mode and the resolved test node. Two runs whose calibers differ are otherwise indistinguishable after the fact. run_batch gains --backend and --devices (--gpus kept as an alias) and carries backend/device/timer into ops_status.csv. test_backends.py covers the registry, alias resolution, visibility env and plugin planning. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
AharrypotterandClaude Opus 5
|
35a928a516 |
chore: pin shipped text formats to LF
Everything in this repository runs on Linux accelerator hosts, and campaign receipts record the SHA-256 of the harness files that produced them. With Git's default autocrlf on Windows a checkout rewrites the shell drivers with CRLF, which breaks them outright on the target host and changes every recorded digest. The two inventory CSVs are renormalized here because their content is unchanged; the remaining files normalize as part of the commits that actually modify them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |