feat: make the harness backend-agnostic

Everything about which accelerator a run targets was previously spelled CUDA:
device discovery went through nvidia-smi, pinning through CUDA_VISIBLE_DEVICES,
the vendor guard only recognised NVIDIA, and the -p plugin list was hardcoded in
run_pytest.sh with no way to drop the CUDA-only members. Running the same
operator corpus on another vendor's device therefore meant editing the drivers.

The selection now hangs off one ZL_BACKEND variable (cuda/musa/rocm/ascend, with
the usual aliases):

- backends/ owns per-backend device enumeration, visibility env keys and
  capability flags, so nothing above it names a vendor;
- plugin_plan.py derives the -p list from those capabilities instead of a
  hardcoded array, which is what lets a non-CUDA run omit the graph-capture and
  cluster-fix plugins rather than watch them fail;
- _timing_plugin selects the timer from the same capabilities: CUDA keeps
  graph capture with its per-shape fallback, other backends stay on upstream
  do_bench and record timer=direct, because capture-then-fallback on a device
  without graph support only produces noise;
- _device_setup_plugin replaces the NVIDIA-only vendor guard;
- providers/flaggems.py resolves the pytest node, so a caller may pass TEST_NODE
  directly instead of the OP/OP_FILE pair, and SHAPE_MODE=upstream can defer to
  FlagGems' own core_shapes;
- _run_identity.py writes run_identity.json next to each run, recording backend,
  device, timer, autotune mode and the resolved test node. Two runs whose
  calibers differ are otherwise indistinguishable after the fact.

run_batch gains --backend and --devices (--gpus kept as an alias) and carries
backend/device/timer into ops_status.csv. test_backends.py covers the registry,
alias resolution, visibility env and plugin planning.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
AharrypotterandClaude Opus 5 committed 2026-09-09 16:39:14 +08:00
1 parent 35a928a516
commit c8ea586d0f
24 files changed
+1448 -159

No files matched your search

+5 -43
View File
@@ -1,45 +1,7 @@
"""Guard against FlagGems' device-detection hang (benchmark-side fix).
"""Backward-compatible alias of `_device_setup_plugin`.
FlagGems' `runtime.backend.device_finder` falls back to
`subprocess.run(nvidia-smi)` **without a timeout**. Under this conda env's
forked/inconsistent subprocess (`_posixsubprocess` symbol mismatch), that
child can hang indefinitely, leaving `import flag_gems` stuck in `wait4`
(seen as run_pytest hanging with no output, GPU at 0%).
This plugin runs at import time — before any test module imports flag_gems —
detects the vendor via torch (no subprocess), and sets GEMS_VENDOR so
device_finder takes its env fast-path (`_get_vendor_from_env`) and never
reaches the hang-prone subprocess probe. No change to the FlagGems repo.
Side effect: get_device_properties initializes the CUDA context very early in
the pytest process. Fine for the current single-process runs; revisit if
fork-based parallelism (e.g. pytest-xdist) is ever introduced.
Kept so an old `-p _device_guard_plugin` line still sets GEMS_VENDOR. New
plugin lists should load `_device_setup_plugin` instead.
"""
import os
import sys
_VENDOR_ENV_KEYS = ("GEMS_VENDOR", "FLAGGEMS_VENDOR", "GEMS_BACKEND", "FLAGGEMS_BACKEND")
def _guard_device_vendor():
# Respect an explicit choice if the user already set one.
if any(k in os.environ for k in _VENDOR_ENV_KEYS):
return
try:
import torch
if torch.cuda.is_available():
name = torch.cuda.get_device_properties(0).name.upper()
if "NVIDIA" in name:
os.environ["GEMS_VENDOR"] = "nvidia"
from _term_style import tag
print(f"{tag('[device-guard-plugin]')} set GEMS_VENDOR=nvidia "
"(skip flag_gems nvidia-smi subprocess probe, avoids import hang)",
file=sys.stderr, flush=True)
except Exception as e: # pragma: no cover
print(f"[device-guard-plugin] torch vendor probe failed ({e}); "
"leaving detection to flag_gems",
file=sys.stderr, flush=True)
_guard_device_vendor()
from _device_setup_plugin import * # noqa: F401,F403
from _device_setup_plugin import pytest_configure # noqa: F401