bench: log every A/B cell as it finishes, and cap it with a timeout

The three markers the README requires be checked — AUTOTUNE_REPLAY_FALLBACK,
BENCHMARK_DIRECT_NO_CUDAGRAPH, BENCHMARK_CAPTURE_POISONED — were only
reachable by grepping run.log afterwards, which happens when someone
remembers to. The NV run that reproduced PR763 wrote a per-cell status row
instead; a 6-round campaign here had no equivalent, so a missing cell, a
stalled cell, or a marker count going from 0 to nonzero was invisible until
the batch ended.

run_cell now writes round/op/side/switch/rc/elapsed_s/success/failed plus the
three marker counts to <root>/status.tsv as each cell completes, and wraps the
cell in CELL_TIMEOUT (default 2400s, matching the NV run) so one hung cell
cannot hold the batch. rc 124/137 is called out by name.

grep -c prints 0 and exits 1 when it finds nothing, so each count swallows its
status; without that the bookkeeping itself would mark a clean cell failed.
This commit is contained in:
Aharrypotter committed 2026-09-10 15:55:36 +08:00
1 parent 92f3d150c7
commit d45806eacd
2 files changed
+42 -4

No files matched your search

+6
View File
@@ -231,6 +231,12 @@ SWITCH=FLAGTREE_MUSA_RLC_ENHANCE OPS="topk masked_select" ROUNDS=6 ZL_BACKEND=
`REPLAY_FROM` 那一份,config 因此跨轮跨侧全部锁死;逐轮交替先后顺序抵消单向漂移。
有轮次失败的算子不出报告——缺轮的报告会让人误以为是完整的 N 轮。
每跑完一格往 `<root>/status.tsv` 追一行:`round op side switch rc elapsed_s success
failed replay_fallback no_cudagraph capture_poisoned`。README 要求必查的三个标记就在
这张表里,跑的时候就能看见它们从 0 变成非 0,不用等结束再 grep——事后 grep 只在有人
想起来的时候才发生。每格另有 `CELL_TIMEOUT`(默认 2400 秒)兜底,超时记 `rc=124` 并打
`!!! CELL_TIMEOUT`,不会让一格卡死拖住整批。
第 1 轮 A/B 跑完后,脚本比两侧 `ttgir/` 的指纹(相对路径 + 内容)。**逐字节相同就
跳过这个算子**,并打 `!!! SWITCH_NO_IR_DIFF op=... switch=...`:这个开关没改变该算子的
生成代码,剩下 N-1 轮测的全是噪声,最后还会报成一个"无差异"的结论。先查开关名拼写