bench: record device clocks, temperature and power for the whole campaign

A 6-round A/B here reports side noise at the 0.01% level. Clock throttling and
thermal drift move latency by far more than that and leave no trace in
run.log, so the report comes out looking exactly as clean as a run on a stable
machine. The NV run that reproduced PR763 sampled nvidia-smi every 5s
alongside the campaign; the MUSA campaigns had no equivalent, and the parallel
plan makes it worse by heating the whole box.

telemetry.py samples through the backend, so cuda and musa emit the same
columns (index, sm_clock_mhz, mem_clock_mhz, temp_c, power_w, util_pct) and
two machines' campaigns can be read side by side. mthreads-gmi offers no CSV
query mode, so the MUSA path parses --json and strips the units glued onto
each value ("36C", "1888MHz", "477.68W"); its "Power Draw " key carries a
trailing space, matched by stripping rather than by literal.

The device comes from the backend's own visibility variable when not pinned. A
list of devices resolves to no -i at all, sampling every one, rather than
silently picking the first. Dropping the placeholder drops its flag too: a
bare trailing -i fails the sampler instead of widening it.

Telemetry is evidence, not a gate. A missing or unpermitted sampler exits
quietly with its reason in telemetry.err and the campaign runs unchanged.
This commit is contained in:
Aharrypotter committed 2026-09-10 16:00:01 +08:00
1 parent d45806eacd
commit ee5f70fa89
7 files changed
+298

No files matched your search

+13
View File
@@ -237,6 +237,19 @@ failed replay_fallback no_cudagraph capture_poisoned`。README 要求必查的
想起来的时候才发生。每格另有 `CELL_TIMEOUT`(默认 2400 秒)兜底,超时记 `rc=124` 并打
`!!! CELL_TIMEOUT`,不会让一格卡死拖住整批。
全程还采一份 `<root>/telemetry.tsv`:
```
timestamp index sm_clock_mhz mem_clock_mhz temp_c power_w util_pct
2026-09-10T07:41:12 7 1888 2500 36 477.68 87
```
列在 cuda 与 musa 上完全一致(`nvidia-smi --query-gpu` 与 `mthreads-gmi -q --json`
各自解析后归一),所以两台机器的 campaign 可以并排读。理由很直接:六轮 A/B 声称的是
0.01% 量级的单侧噪声,**降频和热漂移能移动比这大得多的延迟,而且在 run.log 里不留
任何痕迹**——报告照样干净。`TELEMETRY_INTERVAL`(默认 5 秒)设 0 关闭;采样器缺失或
没权限时 `telemetry.py` 安静退出并把原因写进 `telemetry.err`,不影响测试本身。
第 1 轮 A/B 跑完后,脚本比两侧 `ttgir/` 的指纹(相对路径 + 内容)。**逐字节相同就
跳过这个算子**,并打 `!!! SWITCH_NO_IR_DIFF op=... switch=...`:这个开关没改变该算子的
生成代码,剩下 N-1 轮测的全是噪声,最后还会报成一个"无差异"的结论。先查开关名拼写