refactor: let any backend own graph capture and its failure modes

Two things kept graph-capture timing tied to NVIDIA, and both are mechanism
rather than policy, so neither backend's behaviour changes here.

The capture path named torch.cuda directly for stream, event, synchronize and
the graph type. FlagTree's do_bench already resolves those through
driver.active.get_device_interface(); its do_bench_cudagraph does not. On a
non-CUDA device every capture therefore raised and every row silently took the
fallback while the log still reported capture as enabled. The path now takes
those calls from the interface, with _graph_class picking CUDAGraph or
MUSAGraph. CUDA still calls upstream's do_bench_cudagraph verbatim, so its
numbers do not move; the port only matters where upstream cannot run.

Failure classification moves onto DeviceBackend, because the text it matches is
vendor wording that has no meaning on another runtime, and gains a third
verdict. A kernel refusing capture is ordinary and falls back. A kernel failing
on the device is fatal and re-raises, since a fallback latency there would
report a broken kernel as a measurement. New: the capture ATTEMPT can leave the
context unusable, and falling back then measures inside a dead process -- the
observed shape of that is a fallback marker followed by a teardown crash and
zero rows. Those raise under BENCHMARK_CAPTURE_POISONED instead, naming the
operator as capture-incompatible rather than broken. capture_deny_ops skips the
attempt entirely for operators already known to do this, because for them the
attempt is the damage and rediscovering it per shape costs the run.

CUDA's marker list is carried over unchanged and no poisoning wording is
guessed for it. The scratch-RNG guard around capture is skipped when an
interface exposes no default_generators, since there is then no registration to
protect.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
AharrypotterandClaude Opus 5 committed 2026-09-09 17:56:32 +08:00
1 parent 783c0655bc
commit 6fa5f5d7a2
4 files changed
+258 -71

No files matched your search

+38
View File
@@ -52,6 +52,44 @@ class DeviceBackend:
supports_cuda_graph: bool = False
supports_mm_cluster_fix: bool = False
supports_triton_hooks: bool = True
# --- graph-capture failure classification -----------------------------
# A kernel that refuses capture is normal and falls back. A device error is
# not, and the two are told apart by vendor error text, so the wording lives
# on the backend rather than in the timing plugin.
#
# fatal : the kernel itself failed on the device. Re-raise; recording a
# fallback latency here would report a broken kernel as a result.
# poisoning: the capture ATTEMPT left the context unusable. Falling back runs
# the measurement inside a dead process, which is how an operator
# ends up emitting a fallback marker and then still dying at
# teardown with zero rows. Re-raise too, but under its own marker,
# because the operator is capture-incompatible rather than broken:
# the actionable response is to re-run it with ZL_ENABLE_CUDAGRAPH=0.
fatal_error_markers: Tuple[str, ...] = ()
poisoning_error_markers: Tuple[str, ...] = ()
# Operators whose capture attempt is known to poison the context on this
# backend. Skipped up front so one of them cannot cost a whole run.
capture_deny_ops: Tuple[str, ...] = ()
def classify_capture_error(self, exc: BaseException) -> str:
"""-> "fatal" | "poisoned" | "recoverable".
Matched against the exception text itself, never a generic footer: torch
appends "Compile with `TORCH_USE_CUDA_DSA` to enable device-side
assertions." to every AcceleratorError, so a loose "device-side assert"
marker would classify every capture refusal as fatal and disable the
documented fallback path outright.
"""
msg = f"{type(exc).__name__}: {exc}".lower()
if any(m in msg for m in self.fatal_error_markers):
return "fatal"
if any(m in msg for m in self.poisoning_error_markers):
return "poisoned"
return "recoverable"
def denies_capture(self, op: str) -> bool:
return bool(op) and op in self.capture_deny_ops
# Optional listing command; (argv, line_prefix). prefix="" means count
# non-empty lines. Must be a real vendor CLI; do not invent flags.
list_command: Optional[Tuple[Sequence[str], str]] = None