refactor: let any backend own graph capture and its failure modes

Two things kept graph-capture timing tied to NVIDIA, and both are mechanism
rather than policy, so neither backend's behaviour changes here.

The capture path named torch.cuda directly for stream, event, synchronize and
the graph type. FlagTree's do_bench already resolves those through
driver.active.get_device_interface(); its do_bench_cudagraph does not. On a
non-CUDA device every capture therefore raised and every row silently took the
fallback while the log still reported capture as enabled. The path now takes
those calls from the interface, with _graph_class picking CUDAGraph or
MUSAGraph. CUDA still calls upstream's do_bench_cudagraph verbatim, so its
numbers do not move; the port only matters where upstream cannot run.

Failure classification moves onto DeviceBackend, because the text it matches is
vendor wording that has no meaning on another runtime, and gains a third
verdict. A kernel refusing capture is ordinary and falls back. A kernel failing
on the device is fatal and re-raises, since a fallback latency there would
report a broken kernel as a measurement. New: the capture ATTEMPT can leave the
context unusable, and falling back then measures inside a dead process -- the
observed shape of that is a fallback marker followed by a teardown crash and
zero rows. Those raise under BENCHMARK_CAPTURE_POISONED instead, naming the
operator as capture-incompatible rather than broken. capture_deny_ops skips the
attempt entirely for operators already known to do this, because for them the
attempt is the damage and rediscovering it per shape costs the run.

CUDA's marker list is carried over unchanged and no poisoning wording is
guessed for it. The scratch-RNG guard around capture is skipped when an
interface exposes no default_generators, since there is then no registration to
protect.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
AharrypotterandClaude Opus 5 committed 2026-09-09 17:56:32 +08:00
1 parent 783c0655bc
commit 6fa5f5d7a2
4 files changed
+258 -71

No files matched your search

+14
View File
@@ -12,3 +12,17 @@ class CudaBackend(DeviceBackend):
supports_mm_cluster_fix = True
supports_triton_hooks = True
list_command = (("nvidia-smi", "--list-gpus"), "GPU ")
# Unchanged from when this list lived in _cudagraph_plugin; CUDA numbers do
# not move because of the classification refactor.
fatal_error_markers = (
"illegal memory access",
"cudaerrorillegaladdress",
"out of memory",
"device-side assert triggered",
"an illegal instruction",
"misaligned address",
"uncorrectable ecc",
)
# No CUDA wording is known to poison the context without also matching one
# of the fatal markers above, so nothing is listed here rather than guessed.
poisoning_error_markers = ()