forked from zhoulin/zl_bench
bench: log every A/B cell as it finishes, and cap it with a timeout
The three markers the README requires be checked — AUTOTUNE_REPLAY_FALLBACK, BENCHMARK_DIRECT_NO_CUDAGRAPH, BENCHMARK_CAPTURE_POISONED — were only reachable by grepping run.log afterwards, which happens when someone remembers to. The NV run that reproduced PR763 wrote a per-cell status row instead; a 6-round campaign here had no equivalent, so a missing cell, a stalled cell, or a marker count going from 0 to nonzero was invisible until the batch ended. run_cell now writes round/op/side/switch/rc/elapsed_s/success/failed plus the three marker counts to <root>/status.tsv as each cell completes, and wraps the cell in CELL_TIMEOUT (default 2400s, matching the NV run) so one hung cell cannot hold the batch. rc 124/137 is called out by name. grep -c prints 0 and exits 1 when it finds nothing, so each count swallows its status; without that the bookkeeping itself would mark a clean cell failed.
This commit is contained in:
1 parent
92f3d150c7
commit
d45806eacd
2 files changed
+42
-4
No files matched your search
@@ -231,6 +231,12 @@ SWITCH=FLAGTREE_MUSA_RLC_ENHANCE OPS="topk masked_select" ROUNDS=6 ZL_BACKEND=
|
||||
`REPLAY_FROM` 那一份,config 因此跨轮跨侧全部锁死;逐轮交替先后顺序抵消单向漂移。
|
||||
有轮次失败的算子不出报告——缺轮的报告会让人误以为是完整的 N 轮。
|
||||
|
||||
每跑完一格往 `<root>/status.tsv` 追一行:`round op side switch rc elapsed_s success
|
||||
failed replay_fallback no_cudagraph capture_poisoned`。README 要求必查的三个标记就在
|
||||
这张表里,跑的时候就能看见它们从 0 变成非 0,不用等结束再 grep——事后 grep 只在有人
|
||||
想起来的时候才发生。每格另有 `CELL_TIMEOUT`(默认 2400 秒)兜底,超时记 `rc=124` 并打
|
||||
`!!! CELL_TIMEOUT`,不会让一格卡死拖住整批。
|
||||
|
||||
第 1 轮 A/B 跑完后,脚本比两侧 `ttgir/` 的指纹(相对路径 + 内容)。**逐字节相同就
|
||||
跳过这个算子**,并打 `!!! SWITCH_NO_IR_DIFF op=... switch=...`:这个开关没改变该算子的
|
||||
生成代码,剩下 N-1 轮测的全是噪声,最后还会报成一个"无差异"的结论。先查开关名拼写
|
||||
|
||||
Reference in new issue
Block a user