The worker carried a hand-copied plugin array, taken from zl_bench before
backends/ existed, and it had drifted twice over. It still named
_device_guard_plugin, which is now a shim for _device_setup_plugin. More
consequentially it carried no timing plugin at all, so every measurement this
campaign ever sealed was timer=direct while zl_bench's own driver had been
capturing -- a difference the harness never reported, because the missing plugin
cannot announce itself.
It now calls plugin_plan.py with ZL_BACKEND=musa and uses whatever comes back,
so the caliber follows the backend's declared capabilities instead of a snapshot
of them. The plan is preflighted for non-emptiness, since an empty array would
silently run pytest with no plugins at all: no shape injection, no autotune
record, no IR dump, and a table that still looks well-formed.
Two effects on this campaign. _timing_plugin now enters the list, and since
MUSA declares graph capture the worker will capture where the operator allows
it; operators that cannot are marked per row, as on any other backend. And
_parallel_warmup_plugin joins, inert unless PARALLEL_WARMUP_GPUS is set.
The sealed 32-workload run therefore sits on the far side of a caliber change
and is not comparable with anything produced after this commit. Re-running it is
a campaign decision, not a harness one.
The worker test asserted the literal "-p _musa_benchmark_compat_plugin"; it now
asserts the delegation and that no -p is spelled out locally. What the plan
holds for musa is covered where it belongs, in zl_bench's test_backends.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A sealed run already holds every TTGIR dump and its launch index, so a change
in how convert topology is counted can be applied to it directly instead of
spending accelerator time.
The tool re-reads each cell's dump, recomputes the scoped count, re-runs the
campaign classifier against the sealed correctness verdicts, and writes a
derived receipt and matrix outside the run root. It refuses to write inside a
sealed run and rejects a directory with no campaign receipt.
A derived verdict can only demote a topology claim. Promoting one to G or C
needs reversed-order timing, which a sealed run holds only for tasks that
already showed a reduction, so a reduction appearing for the first time under
scoping is reported as X for missing round-b evidence rather than guessed.
Run against the sealed 32-workload S5000 campaign, a floor of 1 changes exactly
one task: gelu_backward moves from C to B, and the only excluded files in the
whole campaign are its five once-launched randn_kernel dumps. A floor of 0
reproduces the sealed counts with no task changed, which is the equivalence
that makes the scoped number auditable rather than merely different.
A frozen 32-workload campaign that measures how FlagTree's RemoveLayoutConversions
pass behaves on a Moore Threads S5000, through FlagGems public dispatch and this
repository's existing plugins.
The manifest is the contract: source commits, the single physical device, the
three screen cells off/mask0/candidate, the native-event timer, and per-workload
dtypes, timeouts and canary membership. prepare validates it against
ops/inventory_main.csv and freezes an immutable task matrix; the runner executes
the matrix serially, and the worker drives one operator cell through pytest with
_ir_meta_plugin, _autotune_record_plugin and the rest.
Three properties are worth calling out because they decide whether a verdict
means anything.
Convert topology is scoped to the measured operator. A benchmark call builds its
inputs once and then runs the operator many times under the timer, so counting
ttg.convert_layout over every executed kernel credits the candidate with
converts removed from input construction that no timer can observe. Each dumped
TTGIR file is joined to its launch count from index.tsv and only variants
launched more than measured_launch_floor contribute; the unscoped total and the
excluded files stay in the receipt, and a cell that cannot be joined to its
index is classified X rather than trusted.
Correctness is scoped to the variant a workload measures. Running a whole test
file lets an unrelated variant decide the verdict, so a workload may pin node
ids or a -k selector, never both, and any narrowing must state its reason.
Targets are resolved before the campaign starts. The inventory can lag the
pinned FlagGems commit, and such a task previously failed only inside the
worker's preflight, which exited silently and left an empty log. The runner now
resolves every benchmark and correctness target against the real tree, a
workload may name a target the inventory does not know about if it says why,
and each preflight check names itself on failure.
AharrypotterandClaude Opus 5