forked from zhoulin/zl_bench
feat: make the harness backend-agnostic
Everything about which accelerator a run targets was previously spelled CUDA: device discovery went through nvidia-smi, pinning through CUDA_VISIBLE_DEVICES, the vendor guard only recognised NVIDIA, and the -p plugin list was hardcoded in run_pytest.sh with no way to drop the CUDA-only members. Running the same operator corpus on another vendor's device therefore meant editing the drivers. The selection now hangs off one ZL_BACKEND variable (cuda/musa/rocm/ascend, with the usual aliases): - backends/ owns per-backend device enumeration, visibility env keys and capability flags, so nothing above it names a vendor; - plugin_plan.py derives the -p list from those capabilities instead of a hardcoded array, which is what lets a non-CUDA run omit the graph-capture and cluster-fix plugins rather than watch them fail; - _timing_plugin selects the timer from the same capabilities: CUDA keeps graph capture with its per-shape fallback, other backends stay on upstream do_bench and record timer=direct, because capture-then-fallback on a device without graph support only produces noise; - _device_setup_plugin replaces the NVIDIA-only vendor guard; - providers/flaggems.py resolves the pytest node, so a caller may pass TEST_NODE directly instead of the OP/OP_FILE pair, and SHAPE_MODE=upstream can defer to FlagGems' own core_shapes; - _run_identity.py writes run_identity.json next to each run, recording backend, device, timer, autotune mode and the resolved test node. Two runs whose calibers differ are otherwise indistinguishable after the fact. run_batch gains --backend and --devices (--gpus kept as an alias) and carries backend/device/timer into ops_status.csv. test_backends.py covers the registry, alias resolution, visibility env and plugin planning. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
1 parent
35a928a516
commit
c8ea586d0f
24 files changed
+1448
-159
No files matched your search
+18
-14
@@ -60,8 +60,8 @@ def _requested_gpus() -> int:
|
||||
if want < 2:
|
||||
return 0
|
||||
try:
|
||||
import torch
|
||||
have = torch.cuda.device_count()
|
||||
from backends import get_backend
|
||||
have = get_backend().probe_device_count()
|
||||
except Exception:
|
||||
return 0
|
||||
return max(0, min(want, have))
|
||||
@@ -70,17 +70,20 @@ def _requested_gpus() -> int:
|
||||
def _visible_devices() -> List[str]:
|
||||
"""The device ids a shard may be pinned to, in parent-visible order.
|
||||
|
||||
Must respect an inherited CUDA_VISIBLE_DEVICES: writing a bare shard index
|
||||
into the child would re-interpret it as an absolute id, so a caller who
|
||||
picked idle cards (CUDA_VISIBLE_DEVICES=5,6) would silently get cards 0,1 —
|
||||
exactly the busy-GPU case the caller was avoiding.
|
||||
Must respect inherited visibility env for the *active* backend. Writing a
|
||||
bare shard index into the child would re-interpret it as an absolute id,
|
||||
so a caller who picked idle cards (CUDA_VISIBLE_DEVICES=5,6) would silently
|
||||
get cards 0,1 — exactly the busy-GPU case the caller was avoiding.
|
||||
"""
|
||||
raw = os.environ.get("CUDA_VISIBLE_DEVICES", "").strip()
|
||||
if raw:
|
||||
return [d.strip() for d in raw.split(",") if d.strip()]
|
||||
from backends import get_backend
|
||||
backend = get_backend()
|
||||
for key in backend.visibility_env_keys:
|
||||
raw = os.environ.get(key, "").strip()
|
||||
if raw:
|
||||
return [d.strip() for d in raw.split(",") if d.strip()]
|
||||
try:
|
||||
import torch
|
||||
return [str(i) for i in range(torch.cuda.device_count())]
|
||||
n = backend.probe_device_count()
|
||||
return [str(i) for i in range(n)] if n else []
|
||||
except Exception:
|
||||
return []
|
||||
|
||||
@@ -111,8 +114,8 @@ def _spawn(shards: List[List[Any]], op: str, extra: Dict[str, Any],
|
||||
) -> Tuple[List[Path], List[int]]:
|
||||
"""One subprocess per shard, each pinned to its own GPU.
|
||||
|
||||
Subprocesses rather than pytest-xdist: _device_guard_plugin initializes a
|
||||
CUDA context at import time, which does not survive fork-based parallelism.
|
||||
Subprocesses rather than pytest-xdist: `_device_setup_plugin` may initialize
|
||||
a device context at import time, which does not survive fork-based parallelism.
|
||||
"""
|
||||
import yaml
|
||||
procs: List[Tuple[int, subprocess.Popen, Any]] = []
|
||||
@@ -143,7 +146,8 @@ def _spawn(shards: List[List[Any]], op: str, extra: Dict[str, Any],
|
||||
child_argv += ["--shape_file", str(w / "shapes.yaml")]
|
||||
|
||||
env = dict(os.environ)
|
||||
env["CUDA_VISIBLE_DEVICES"] = devices[i]
|
||||
from backends import get_backend
|
||||
env = get_backend().apply_visibility(env, devices[i])
|
||||
env[_WORKER_ENV] = "1" # stops recursion
|
||||
env[_RECORD_DIR_ENV] = str(rec) # shard records its own picks
|
||||
env.pop(_REPLAY_DIR_ENV, None) # a shard must sweep
|
||||
|
||||
Reference in new issue
Block a user