feat: make the harness backend-agnostic

Everything about which accelerator a run targets was previously spelled CUDA:
device discovery went through nvidia-smi, pinning through CUDA_VISIBLE_DEVICES,
the vendor guard only recognised NVIDIA, and the -p plugin list was hardcoded in
run_pytest.sh with no way to drop the CUDA-only members. Running the same
operator corpus on another vendor's device therefore meant editing the drivers.

The selection now hangs off one ZL_BACKEND variable (cuda/musa/rocm/ascend, with
the usual aliases):

- backends/ owns per-backend device enumeration, visibility env keys and
  capability flags, so nothing above it names a vendor;
- plugin_plan.py derives the -p list from those capabilities instead of a
  hardcoded array, which is what lets a non-CUDA run omit the graph-capture and
  cluster-fix plugins rather than watch them fail;
- _timing_plugin selects the timer from the same capabilities: CUDA keeps
  graph capture with its per-shape fallback, other backends stay on upstream
  do_bench and record timer=direct, because capture-then-fallback on a device
  without graph support only produces noise;
- _device_setup_plugin replaces the NVIDIA-only vendor guard;
- providers/flaggems.py resolves the pytest node, so a caller may pass TEST_NODE
  directly instead of the OP/OP_FILE pair, and SHAPE_MODE=upstream can defer to
  FlagGems' own core_shapes;
- _run_identity.py writes run_identity.json next to each run, recording backend,
  device, timer, autotune mode and the resolved test node. Two runs whose
  calibers differ are otherwise indistinguishable after the fact.

run_batch gains --backend and --devices (--gpus kept as an alias) and carries
backend/device/timer into ops_status.csv. test_backends.py covers the registry,
alias resolution, visibility env and plugin planning.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
AharrypotterandClaude Opus 5 committed 2026-09-09 16:39:14 +08:00
1 parent 35a928a516
commit c8ea586d0f
24 files changed
+1448 -159

No files matched your search

+18 -14
View File
@@ -60,8 +60,8 @@ def _requested_gpus() -> int:
if want < 2:
return 0
try:
import torch
have = torch.cuda.device_count()
from backends import get_backend
have = get_backend().probe_device_count()
except Exception:
return 0
return max(0, min(want, have))
@@ -70,17 +70,20 @@ def _requested_gpus() -> int:
def _visible_devices() -> List[str]:
"""The device ids a shard may be pinned to, in parent-visible order.
Must respect an inherited CUDA_VISIBLE_DEVICES: writing a bare shard index
into the child would re-interpret it as an absolute id, so a caller who
picked idle cards (CUDA_VISIBLE_DEVICES=5,6) would silently get cards 0,1 —
exactly the busy-GPU case the caller was avoiding.
Must respect inherited visibility env for the *active* backend. Writing a
bare shard index into the child would re-interpret it as an absolute id,
so a caller who picked idle cards (CUDA_VISIBLE_DEVICES=5,6) would silently
get cards 0,1 — exactly the busy-GPU case the caller was avoiding.
"""
raw = os.environ.get("CUDA_VISIBLE_DEVICES", "").strip()
if raw:
return [d.strip() for d in raw.split(",") if d.strip()]
from backends import get_backend
backend = get_backend()
for key in backend.visibility_env_keys:
raw = os.environ.get(key, "").strip()
if raw:
return [d.strip() for d in raw.split(",") if d.strip()]
try:
import torch
return [str(i) for i in range(torch.cuda.device_count())]
n = backend.probe_device_count()
return [str(i) for i in range(n)] if n else []
except Exception:
return []
@@ -111,8 +114,8 @@ def _spawn(shards: List[List[Any]], op: str, extra: Dict[str, Any],
) -> Tuple[List[Path], List[int]]:
"""One subprocess per shard, each pinned to its own GPU.
Subprocesses rather than pytest-xdist: _device_guard_plugin initializes a
CUDA context at import time, which does not survive fork-based parallelism.
Subprocesses rather than pytest-xdist: `_device_setup_plugin` may initialize
a device context at import time, which does not survive fork-based parallelism.
"""
import yaml
procs: List[Tuple[int, subprocess.Popen, Any]] = []
@@ -143,7 +146,8 @@ def _spawn(shards: List[List[Any]], op: str, extra: Dict[str, Any],
child_argv += ["--shape_file", str(w / "shapes.yaml")]
env = dict(os.environ)
env["CUDA_VISIBLE_DEVICES"] = devices[i]
from backends import get_backend
env = get_backend().apply_visibility(env, devices[i])
env[_WORKER_ENV] = "1" # stops recursion
env[_RECORD_DIR_ENV] = str(rec) # shard records its own picks
env.pop(_REPLAY_DIR_ENV, None) # a shard must sweep