forked from zhoulin/zl_bench
refactor: let any backend own graph capture and its failure modes
Two things kept graph-capture timing tied to NVIDIA, and both are mechanism rather than policy, so neither backend's behaviour changes here. The capture path named torch.cuda directly for stream, event, synchronize and the graph type. FlagTree's do_bench already resolves those through driver.active.get_device_interface(); its do_bench_cudagraph does not. On a non-CUDA device every capture therefore raised and every row silently took the fallback while the log still reported capture as enabled. The path now takes those calls from the interface, with _graph_class picking CUDAGraph or MUSAGraph. CUDA still calls upstream's do_bench_cudagraph verbatim, so its numbers do not move; the port only matters where upstream cannot run. Failure classification moves onto DeviceBackend, because the text it matches is vendor wording that has no meaning on another runtime, and gains a third verdict. A kernel refusing capture is ordinary and falls back. A kernel failing on the device is fatal and re-raises, since a fallback latency there would report a broken kernel as a measurement. New: the capture ATTEMPT can leave the context unusable, and falling back then measures inside a dead process -- the observed shape of that is a fallback marker followed by a teardown crash and zero rows. Those raise under BENCHMARK_CAPTURE_POISONED instead, naming the operator as capture-incompatible rather than broken. capture_deny_ops skips the attempt entirely for operators already known to do this, because for them the attempt is the damage and rediscovering it per shape costs the run. CUDA's marker list is carried over unchanged and no poisoning wording is guessed for it. The scratch-RNG guard around capture is skipped when an interface exposes no default_generators, since there is then no registration to protect. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
1 parent
783c0655bc
commit
6fa5f5d7a2
4 files changed
+258
-71
No files matched your search
@@ -52,6 +52,44 @@ class DeviceBackend:
|
||||
supports_cuda_graph: bool = False
|
||||
supports_mm_cluster_fix: bool = False
|
||||
supports_triton_hooks: bool = True
|
||||
|
||||
# --- graph-capture failure classification -----------------------------
|
||||
# A kernel that refuses capture is normal and falls back. A device error is
|
||||
# not, and the two are told apart by vendor error text, so the wording lives
|
||||
# on the backend rather than in the timing plugin.
|
||||
#
|
||||
# fatal : the kernel itself failed on the device. Re-raise; recording a
|
||||
# fallback latency here would report a broken kernel as a result.
|
||||
# poisoning: the capture ATTEMPT left the context unusable. Falling back runs
|
||||
# the measurement inside a dead process, which is how an operator
|
||||
# ends up emitting a fallback marker and then still dying at
|
||||
# teardown with zero rows. Re-raise too, but under its own marker,
|
||||
# because the operator is capture-incompatible rather than broken:
|
||||
# the actionable response is to re-run it with ZL_ENABLE_CUDAGRAPH=0.
|
||||
fatal_error_markers: Tuple[str, ...] = ()
|
||||
poisoning_error_markers: Tuple[str, ...] = ()
|
||||
# Operators whose capture attempt is known to poison the context on this
|
||||
# backend. Skipped up front so one of them cannot cost a whole run.
|
||||
capture_deny_ops: Tuple[str, ...] = ()
|
||||
|
||||
def classify_capture_error(self, exc: BaseException) -> str:
|
||||
"""-> "fatal" | "poisoned" | "recoverable".
|
||||
|
||||
Matched against the exception text itself, never a generic footer: torch
|
||||
appends "Compile with `TORCH_USE_CUDA_DSA` to enable device-side
|
||||
assertions." to every AcceleratorError, so a loose "device-side assert"
|
||||
marker would classify every capture refusal as fatal and disable the
|
||||
documented fallback path outright.
|
||||
"""
|
||||
msg = f"{type(exc).__name__}: {exc}".lower()
|
||||
if any(m in msg for m in self.fatal_error_markers):
|
||||
return "fatal"
|
||||
if any(m in msg for m in self.poisoning_error_markers):
|
||||
return "poisoned"
|
||||
return "recoverable"
|
||||
|
||||
def denies_capture(self, op: str) -> bool:
|
||||
return bool(op) and op in self.capture_deny_ops
|
||||
# Optional listing command; (argv, line_prefix). prefix="" means count
|
||||
# non-empty lines. Must be a real vendor CLI; do not invent flags.
|
||||
list_command: Optional[Tuple[Sequence[str], str]] = None
|
||||
|
||||
@@ -12,3 +12,17 @@ class CudaBackend(DeviceBackend):
|
||||
supports_mm_cluster_fix = True
|
||||
supports_triton_hooks = True
|
||||
list_command = (("nvidia-smi", "--list-gpus"), "GPU ")
|
||||
# Unchanged from when this list lived in _cudagraph_plugin; CUDA numbers do
|
||||
# not move because of the classification refactor.
|
||||
fatal_error_markers = (
|
||||
"illegal memory access",
|
||||
"cudaerrorillegaladdress",
|
||||
"out of memory",
|
||||
"device-side assert triggered",
|
||||
"an illegal instruction",
|
||||
"misaligned address",
|
||||
"uncorrectable ecc",
|
||||
)
|
||||
# No CUDA wording is known to poison the context without also matching one
|
||||
# of the fatal markers above, so nothing is listed here rather than guessed.
|
||||
poisoning_error_markers = ()
|
||||
Reference in new issue
Block a user