forked from zhoulin/zl_bench
refactor: let any backend own graph capture and its failure modes
Two things kept graph-capture timing tied to NVIDIA, and both are mechanism rather than policy, so neither backend's behaviour changes here. The capture path named torch.cuda directly for stream, event, synchronize and the graph type. FlagTree's do_bench already resolves those through driver.active.get_device_interface(); its do_bench_cudagraph does not. On a non-CUDA device every capture therefore raised and every row silently took the fallback while the log still reported capture as enabled. The path now takes those calls from the interface, with _graph_class picking CUDAGraph or MUSAGraph. CUDA still calls upstream's do_bench_cudagraph verbatim, so its numbers do not move; the port only matters where upstream cannot run. Failure classification moves onto DeviceBackend, because the text it matches is vendor wording that has no meaning on another runtime, and gains a third verdict. A kernel refusing capture is ordinary and falls back. A kernel failing on the device is fatal and re-raises, since a fallback latency there would report a broken kernel as a measurement. New: the capture ATTEMPT can leave the context unusable, and falling back then measures inside a dead process -- the observed shape of that is a fallback marker followed by a teardown crash and zero rows. Those raise under BENCHMARK_CAPTURE_POISONED instead, naming the operator as capture-incompatible rather than broken. capture_deny_ops skips the attempt entirely for operators already known to do this, because for them the attempt is the damage and rediscovering it per shape costs the run. CUDA's marker list is carried over unchanged and no poisoning wording is guessed for it. The scratch-RNG guard around capture is skipped when an interface exposes no default_generators, since there is then no registration to protect. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
1 parent
783c0655bc
commit
6fa5f5d7a2
4 files changed
+258
-71
No files matched your search
@@ -12,3 +12,17 @@ class CudaBackend(DeviceBackend):
|
||||
supports_mm_cluster_fix = True
|
||||
supports_triton_hooks = True
|
||||
list_command = (("nvidia-smi", "--list-gpus"), "GPU ")
|
||||
# Unchanged from when this list lived in _cudagraph_plugin; CUDA numbers do
|
||||
# not move because of the classification refactor.
|
||||
fatal_error_markers = (
|
||||
"illegal memory access",
|
||||
"cudaerrorillegaladdress",
|
||||
"out of memory",
|
||||
"device-side assert triggered",
|
||||
"an illegal instruction",
|
||||
"misaligned address",
|
||||
"uncorrectable ecc",
|
||||
)
|
||||
# No CUDA wording is known to poison the context without also matching one
|
||||
# of the fatal markers above, so nothing is listed here rather than guessed.
|
||||
poisoning_error_markers = ()
|
||||
Reference in new issue
Block a user