feat: check the run's preconditions, and record which tuner cache it used

Two failure modes that cost this harness real results, both of them silent.

A wrong path does not fail as a wrong path. pytest cannot collect a benchmark
directory that is not there, so benchmark/conftest.py never registers --level
and --mode, and the run dies with "unrecognized arguments: --level core --mode
kernel" -- an error about the flags, several layers away from the missing
directory. run_pytest.sh now checks the FlagGems directory, the benchmark file,
SHAPE_FILE and REPLAY_FROM before starting pytest, prints
"PREFLIGHT_FAILED check=<the named check>" and exits 3, distinct from pytest's
own 1 and 4 so the batch layer can tell a broken environment from a broken
operator. Only the file half of a node is checked; a missing ::test_<fn> already
produces a clear "not found: ..." from pytest and needs no help.

FlagGems keeps libtuner winners in a sqlite DB under $FLAGGEMS_CACHE_DIR
(default ~/.flaggems), shared by every run on the machine. Whichever side of an
A/B runs first therefore hands its winners to the second, which then compiles
only those: measured on router_gemm, the second of two shared-cache runs
compiled 10 variants instead of 234 and finished in 17s instead of 52s. That is
how an mm A/B in this repo produced "945 variants -> 33, 136 converts -> 10" and
read as a compiler result when it was cache inheritance; isolated, both runs
compile 234.

ZL_ISOLATE_TUNER_CACHE=1 gives a run its own cache directory, dropped afterwards
with the Triton one since the chosen configs are already in autotune_records/.
It defaults to 0, because cold tuning is expensive and the standard REPLAY_FROM
A/B pins configs anyway; only record-against-record is exposed. The default is
unchanged, but the caliber is no longer invisible: either way it goes into the
run header and run_identity.json as tuner_cache, which is what was missing when
the mm numbers could not be explained after the fact.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

README.md

@@ -66,6 +66,7 @@ softmax:
 | `REPLAY_FROM` | 空（record 模式） | 指向某次历史 run 目录，replay 其 autotune 选择。做 A/B 时 B 侧必须设（见「A/B 对比测试标准流程」） |
 | `USE_FLAGTUNE` | `0` | `0` 走普通 autotune（跳过搜索、快速验证）；`1` 用 FlagTune 扩展调优空间（首跑全量搜索、慢）。A/B 两侧须取同值 |
 | `BENCH_LEVEL` | `core` | 透传上游 `--level`。`comprehensive` 相对 `core` 做两件事：① 在 `set_shapes()` 里并入子类 `set_more_shapes()`（`mm` 因此从 5 个 shape 变 29 个，含 N/K 到 152064 的大 K 与 `models_shapes/*.yaml` 的模型 shape）；② 在 `get_input_iter()` 里对每个 shape 再产出一次变体（BLAS 是转置侧），用例数翻倍。注意 ① 只在 `SHAPE_MODE=upstream` 下可见——传了 `SHAPE_FILE` 时 `_shape_inject_plugin` 会在上游合并之后再覆盖一次、把 shape 集拉回 yaml 内容（BLAS 那类 arity 不一致的情形除外，插件会退让）；② 不经插件，任何 shape 来源下都生效。sweep 成本同步放大，shape 多时配合 `PARALLEL_WARMUP_GPUS`。A/B 两侧须取同值 |
+| `ZL_ISOLATE_TUNER_CACHE` | `0` | FlagGems 的 libtuner winner 存在 `$FLAGGEMS_CACHE_DIR/config_cache` 的 sqlite 里（默认 `~/.flaggems`），跨 run 长存。`1` 给本次 run 一个私有缓存目录，两侧各自冷调优（README「各自最优」口径），跑完随 Triton 缓存一起删——选中的 config 已在 `autotune_records/` 里，DB 本身无留存价值。共享缓存的风险是单向的：先跑的一侧写入 winner，后跑的一侧直接命中、只编译那几个胜出变体。实测 `router_gemm` 共享模式下第二次运行编译变体从 234 掉到 10、耗时 52s→17s，隔离模式下两次都是 234 个变体。走 `REPLAY_FROM` 的标准 A/B 不受影响（config 本就锁死），record 对 record 才会中招。取值无论如何都写进 `run_identity.json` 的 `tuner_cache` |
 | `DTYPES` | 空（上游默认 dtype 扫描） | 空格分隔的 dtype 白名单（如 `"bfloat16 float16"`），逐个转为上游 `--dtypes`。算子不支持时上游报 `can't be supported by this op` |
 | `PARALLEL_WARMUP_GPUS` | 空（串行） | 设为 `N`（≥2）时把 autotune sweep 分片到 N 卡并行，测量仍单卡串行；不设或 `<2` 则完全串行（见「加速：多卡并行 sweep」） |
 | `ZL_ENABLE_CUDAGRAPH` | `auto` | `auto`/`1` 仅在 CUDA 上走 Graph 计时；非 CUDA 即使设 `1` 也拒绝并保持 `timer=direct` |
@@ -148,6 +149,18 @@ libtuner 的持久缓存何时失效：FlagGems kernel 源码改动、tune_confi
 
 同一层缓存也决定了 `_autotune_record_plugin` 的实现方式：record 不能靠"cache 新增了哪个 key"来判断本次选中的 config（缓存一热就走 cached 分支、不写新 key，键集差集恒为空），改为 `run()` 之后直接读回本次调用的 `self.cache[key]`；replay 同理不能加"key 不在 cache 里才注入"的前置条件，否则注入被跳过、该 run 表面在 replay 实际在自调优。想确认 record 真的覆盖到目标 kernel，查 `runs/<run>/autotune_records/<op>.json` 里有无对应 kernel 条目——漏记时该文件照样生成，只是少了 libtuner 那几个。
 
+### 前置检查
+
+`run_pytest.sh` 在启动 pytest 之前校验 FlagGems 目录、benchmark 文件、`SHAPE_FILE`
+与 `REPLAY_FROM`，失败时打 `!!! PREFLIGHT_FAILED check=<具名检查>` 并以 3 退出。
+
+这些检查存在的理由是**静默失败会指向错误的方向**：路径不存在时 pytest 收集不到
+benchmark 目录，注册 `--level`/`--mode` 的 `conftest.py` 就没被加载，报错变成
+`unrecognized arguments: --level core --mode kernel`，看起来像参数或版本问题。
+
+只校验节点的文件那一半；`::test_<fn>` 交给 pytest，函数不存在时它自己会清楚地报
+`not found: ...`。
+
 ### ab_fold_test.sh：A/B 测试参考示例
 
 `ab_fold_test.sh` 是上述 A/B 流程的现成参考：以 FlagGems `FLAGGEMS_MXFP4_FOLDSCALE`（fold_scale 优化）开关为对比对象，对 4 个 DeepSeek-V4-Flash 真实 trace shape 集（`shape/DeepSeek-V4-Flash-p{1024,4096,32768,65536}d1024.yaml`，随仓库提供）各跑一对 FOLD=0（基线，record）→ FOLD=1（replay 同 config），共 8 轮：

_run_identity.py

@@ -71,6 +71,10 @@ def main(argv=None) -> int:
     p.add_argument("--test-file", default=os.environ.get("TEST_FILE", ""))
     p.add_argument("--shape-mode", default=os.environ.get("SHAPE_MODE", ""))
     p.add_argument("--bench-level", default=os.environ.get("BENCH_LEVEL", "core"))
+    # "isolated" or "shared(<path>)". Recorded because a shared libtuner cache
+    # lets whichever side runs first hand its winners to the other, and after
+    # the fact the two runs are otherwise indistinguishable.
+    p.add_argument("--tuner-cache", default="")
     args = p.parse_args(argv)
     init_identity(
         args.out_dir,
@@ -83,6 +87,7 @@ def main(argv=None) -> int:
         test_file=args.test_file,
         shape_mode=args.shape_mode,
         bench_level=args.bench_level,
+        tuner_cache=args.tuner_cache,
     )
     return 0
 

run_pytest.sh

@@ -75,8 +75,30 @@ case "$BENCH_LEVEL" in
   *) echo "!!! BENCH_LEVEL 只能是 core 或 comprehensive（当前：$BENCH_LEVEL）" >&2; exit 2 ;;
 esac
 
+# FlagGems 的 libtuner winner 存在 $FLAGGEMS_CACHE_DIR/config_cache 的 sqlite 里，
+# 默认 ~/.flaggems，跨 run 长存。1=本次 run 用独立缓存目录，两侧各自冷调优
+# （README「各自最优」口径）；0=沿用共享缓存（默认，快）。
+#
+# 共享缓存的风险是单向的：先跑的一侧把 winner 写进去，后跑的一侧直接命中，
+# 于是它只编译那几个胜出变体。实测 mm 上后跑侧的 TTGIR 变体从 945 塌到 33、
+# cvt 从 136 变 10，看起来像被测改动的效果，其实只是继承了对侧的 config。
+# 走 REPLAY_FROM 的标准 A/B 不受此影响（config 本来就锁死），record 对 record
+# 才会中招。无论取哪个值都会写进 run_identity.json，口径不再隐形。
+ZL_ISOLATE_TUNER_CACHE="${ZL_ISOLATE_TUNER_CACHE:-0}"
+
 # ============================== 执行逻辑 ==============================
 
+# 前置检查。静默失败最贵：路径不存在时 pytest 收集不到 benchmark 目录，注册
+# --level/--mode 的 conftest 就没加载，报错变成 "unrecognized arguments"，指向
+# 完全错误的方向。每项检查自报名字，失败即停。
+preflight() {
+  local what="$1"; shift
+  if ! "$@"; then
+    echo "!!! PREFLIGHT_FAILED check=$what" >&2
+    exit 3
+  fi
+}
+
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 export PYTHONPATH="$SCRIPT_DIR${PYTHONPATH:+:$PYTHONPATH}"
 eval "$(python "$SCRIPT_DIR/providers/flaggems.py" --emit-shell \
@@ -88,6 +110,17 @@ eval "$(python "$SCRIPT_DIR/providers/flaggems.py" --emit-shell \
   --shape-mode "$SHAPE_MODE")"
 : "${TEST_FILE:=$FLAGGEMS_DIR/benchmark/test_${OP_FILE}.py::test_${OP}}"
 
+preflight "FlagGems directory exists: $FLAGGEMS_DIR" test -d "$FLAGGEMS_DIR"
+# The node's file half; the ::test_<fn> half is left to pytest, which reports a
+# missing function clearly ("not found: ...") while a missing FILE does not.
+preflight "benchmark file exists: ${TEST_FILE%%::*}" test -f "${TEST_FILE%%::*}"
+if [[ -n "$SHAPE_FILE" ]]; then
+  preflight "shape file exists: $SHAPE_FILE" test -f "$SHAPE_FILE"
+fi
+if [[ -n "$REPLAY_FROM" ]]; then
+  preflight "replay run directory exists: $REPLAY_FROM" test -d "$REPLAY_FROM"
+fi
+
 # 产物目录，可用 OUT_DIR 指定。时间戳只到秒，并发启动会撞名，故用不带 -p 的
 # mkdir 抢占（已存在即失败），撞上就退让到 -2、-3……
 if [[ -n "${OUT_DIR:-}" ]]; then
@@ -162,6 +195,18 @@ else
   SHAPE_ARGS=(--shape_file "$SHAPE_FILE")
 fi
 
+# libtuner 缓存：隔离时用本 run 的私有目录，跑完随 Triton 缓存一起删（选中的
+# config 已经在 autotune_records/ 里，DB 本身没有留存价值）。
+TUNER_CACHE_ARGS=()
+if [[ "$ZL_ISOLATE_TUNER_CACHE" == "1" ]]; then
+  TUNER_CACHE_DIR="$OUT_DIR/.flaggems_cache"
+  rm -rf "$TUNER_CACHE_DIR"; mkdir -p "$TUNER_CACHE_DIR"
+  TUNER_CACHE_ARGS=(FLAGGEMS_CACHE_DIR="$TUNER_CACHE_DIR")
+  TUNER_DESC=isolated
+else
+  TUNER_DESC="shared(${FLAGGEMS_CACHE_DIR:-~/.flaggems})"
+fi
+
 python "$SCRIPT_DIR/_run_identity.py" "$OUT_DIR" \
   --backend "$ZL_BACKEND" \
   --op "$OP" \
@@ -171,11 +216,12 @@ python "$SCRIPT_DIR/_run_identity.py" "$OUT_DIR" \
   --device "${ZL_DEVICE:-}" \
   --test-file "$TEST_FILE" \
   --shape-mode "$SHAPE_MODE" \
-  --bench-level "$BENCH_LEVEL"
+  --bench-level "$BENCH_LEVEL" \
+  --tuner-cache "$TUNER_DESC"
 
 status=0
 {
-  echo "${C_BOLD}>>> op=$OP  backend=$ZL_BACKEND  mode=$MODE_DESC  USE_FLAGTUNE=$USE_FLAGTUNE  shape=$SHAPE_MODE  level=$BENCH_LEVEL${C_RESET}"
+  echo "${C_BOLD}>>> op=$OP  backend=$ZL_BACKEND  mode=$MODE_DESC  USE_FLAGTUNE=$USE_FLAGTUNE  shape=$SHAPE_MODE  level=$BENCH_LEVEL  tuner=$TUNER_DESC${C_RESET}"
   echo "${C_DIM}>>> node=$TEST_FILE${C_RESET}"
   echo "${C_DIM}>>> out=$OUT_DIR${C_RESET}"
 
@@ -186,13 +232,14 @@ status=0
   status=0
   TRITON_CACHE_DIR="$CACHE_DIR" \
   FLAGGEMS_PERF_TTGIR_DUMP_DIR="$OUT_DIR/ttgir" \
-  env "$AUTOTUNE_ENV" \
+  env "$AUTOTUNE_ENV" ${TUNER_CACHE_ARGS[@]+"${TUNER_CACHE_ARGS[@]}"} \
   USE_FLAGTUNE=$USE_FLAGTUNE python -u -m pytest -s "$TEST_FILE" \
     "${PLUGINS[@]}" "${PYTEST_COLOR[@]}" \
     ${SHAPE_ARGS[@]+"${SHAPE_ARGS[@]}"} ${DTYPE_ARGS[@]+"${DTYPE_ARGS[@]}"} \
     --level "$BENCH_LEVEL" --mode kernel || status=$?
 
   rm -rf "$CACHE_DIR"
+  [[ -n "${TUNER_CACHE_DIR:-}" ]] && rm -rf "$TUNER_CACHE_DIR"
 
   if (( status == 0 )); then
     echo "${C_GREEN}>>> done. outputs in $OUT_DIR${C_RESET}"