feat: check the run's preconditions, and record which tuner cache it used

Two failure modes that cost this harness real results, both of them silent.

A wrong path does not fail as a wrong path. pytest cannot collect a benchmark
directory that is not there, so benchmark/conftest.py never registers --level
and --mode, and the run dies with "unrecognized arguments: --level core --mode
kernel" -- an error about the flags, several layers away from the missing
directory. run_pytest.sh now checks the FlagGems directory, the benchmark file,
SHAPE_FILE and REPLAY_FROM before starting pytest, prints
"PREFLIGHT_FAILED check=<the named check>" and exits 3, distinct from pytest's
own 1 and 4 so the batch layer can tell a broken environment from a broken
operator. Only the file half of a node is checked; a missing ::test_<fn> already
produces a clear "not found: ..." from pytest and needs no help.

FlagGems keeps libtuner winners in a sqlite DB under $FLAGGEMS_CACHE_DIR
(default ~/.flaggems), shared by every run on the machine. Whichever side of an
A/B runs first therefore hands its winners to the second, which then compiles
only those: measured on router_gemm, the second of two shared-cache runs
compiled 10 variants instead of 234 and finished in 17s instead of 52s. That is
how an mm A/B in this repo produced "945 variants -> 33, 136 converts -> 10" and
read as a compiler result when it was cache inheritance; isolated, both runs
compile 234.

ZL_ISOLATE_TUNER_CACHE=1 gives a run its own cache directory, dropped afterwards
with the Triton one since the chosen configs are already in autotune_records/.
It defaults to 0, because cold tuning is expensive and the standard REPLAY_FROM
A/B pins configs anyway; only record-against-record is exposed. The default is
unchanged, but the caliber is no longer invisible: either way it goes into the
run header and run_identity.json as tuner_cache, which is what was missing when
the mm numbers could not be explained after the fact.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

run_pytest.sh

@@ -75,8 +75,30 @@ case "$BENCH_LEVEL" in
   *) echo "!!! BENCH_LEVEL 只能是 core 或 comprehensive（当前：$BENCH_LEVEL）" >&2; exit 2 ;;
 esac
 
+# FlagGems 的 libtuner winner 存在 $FLAGGEMS_CACHE_DIR/config_cache 的 sqlite 里，
+# 默认 ~/.flaggems，跨 run 长存。1=本次 run 用独立缓存目录，两侧各自冷调优
+# （README「各自最优」口径）；0=沿用共享缓存（默认，快）。
+#
+# 共享缓存的风险是单向的：先跑的一侧把 winner 写进去，后跑的一侧直接命中，
+# 于是它只编译那几个胜出变体。实测 mm 上后跑侧的 TTGIR 变体从 945 塌到 33、
+# cvt 从 136 变 10，看起来像被测改动的效果，其实只是继承了对侧的 config。
+# 走 REPLAY_FROM 的标准 A/B 不受此影响（config 本来就锁死），record 对 record
+# 才会中招。无论取哪个值都会写进 run_identity.json，口径不再隐形。
+ZL_ISOLATE_TUNER_CACHE="${ZL_ISOLATE_TUNER_CACHE:-0}"
+
 # ============================== 执行逻辑 ==============================
 
+# 前置检查。静默失败最贵：路径不存在时 pytest 收集不到 benchmark 目录，注册
+# --level/--mode 的 conftest 就没加载，报错变成 "unrecognized arguments"，指向
+# 完全错误的方向。每项检查自报名字，失败即停。
+preflight() {
+  local what="$1"; shift
+  if ! "$@"; then
+    echo "!!! PREFLIGHT_FAILED check=$what" >&2
+    exit 3
+  fi
+}
+
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 export PYTHONPATH="$SCRIPT_DIR${PYTHONPATH:+:$PYTHONPATH}"
 eval "$(python "$SCRIPT_DIR/providers/flaggems.py" --emit-shell \
@@ -88,6 +110,17 @@ eval "$(python "$SCRIPT_DIR/providers/flaggems.py" --emit-shell \
   --shape-mode "$SHAPE_MODE")"
 : "${TEST_FILE:=$FLAGGEMS_DIR/benchmark/test_${OP_FILE}.py::test_${OP}}"
 
+preflight "FlagGems directory exists: $FLAGGEMS_DIR" test -d "$FLAGGEMS_DIR"
+# The node's file half; the ::test_<fn> half is left to pytest, which reports a
+# missing function clearly ("not found: ...") while a missing FILE does not.
+preflight "benchmark file exists: ${TEST_FILE%%::*}" test -f "${TEST_FILE%%::*}"
+if [[ -n "$SHAPE_FILE" ]]; then
+  preflight "shape file exists: $SHAPE_FILE" test -f "$SHAPE_FILE"
+fi
+if [[ -n "$REPLAY_FROM" ]]; then
+  preflight "replay run directory exists: $REPLAY_FROM" test -d "$REPLAY_FROM"
+fi
+
 # 产物目录，可用 OUT_DIR 指定。时间戳只到秒，并发启动会撞名，故用不带 -p 的
 # mkdir 抢占（已存在即失败），撞上就退让到 -2、-3……
 if [[ -n "${OUT_DIR:-}" ]]; then
@@ -162,6 +195,18 @@ else
   SHAPE_ARGS=(--shape_file "$SHAPE_FILE")
 fi
 
+# libtuner 缓存：隔离时用本 run 的私有目录，跑完随 Triton 缓存一起删（选中的
+# config 已经在 autotune_records/ 里，DB 本身没有留存价值）。
+TUNER_CACHE_ARGS=()
+if [[ "$ZL_ISOLATE_TUNER_CACHE" == "1" ]]; then
+  TUNER_CACHE_DIR="$OUT_DIR/.flaggems_cache"
+  rm -rf "$TUNER_CACHE_DIR"; mkdir -p "$TUNER_CACHE_DIR"
+  TUNER_CACHE_ARGS=(FLAGGEMS_CACHE_DIR="$TUNER_CACHE_DIR")
+  TUNER_DESC=isolated
+else
+  TUNER_DESC="shared(${FLAGGEMS_CACHE_DIR:-~/.flaggems})"
+fi
+
 python "$SCRIPT_DIR/_run_identity.py" "$OUT_DIR" \
   --backend "$ZL_BACKEND" \
   --op "$OP" \
@@ -171,11 +216,12 @@ python "$SCRIPT_DIR/_run_identity.py" "$OUT_DIR" \
   --device "${ZL_DEVICE:-}" \
   --test-file "$TEST_FILE" \
   --shape-mode "$SHAPE_MODE" \
-  --bench-level "$BENCH_LEVEL"
+  --bench-level "$BENCH_LEVEL" \
+  --tuner-cache "$TUNER_DESC"
 
 status=0
 {
-  echo "${C_BOLD}>>> op=$OP  backend=$ZL_BACKEND  mode=$MODE_DESC  USE_FLAGTUNE=$USE_FLAGTUNE  shape=$SHAPE_MODE  level=$BENCH_LEVEL${C_RESET}"
+  echo "${C_BOLD}>>> op=$OP  backend=$ZL_BACKEND  mode=$MODE_DESC  USE_FLAGTUNE=$USE_FLAGTUNE  shape=$SHAPE_MODE  level=$BENCH_LEVEL  tuner=$TUNER_DESC${C_RESET}"
   echo "${C_DIM}>>> node=$TEST_FILE${C_RESET}"
   echo "${C_DIM}>>> out=$OUT_DIR${C_RESET}"
 
@@ -186,13 +232,14 @@ status=0
   status=0
   TRITON_CACHE_DIR="$CACHE_DIR" \
   FLAGGEMS_PERF_TTGIR_DUMP_DIR="$OUT_DIR/ttgir" \
-  env "$AUTOTUNE_ENV" \
+  env "$AUTOTUNE_ENV" ${TUNER_CACHE_ARGS[@]+"${TUNER_CACHE_ARGS[@]}"} \
   USE_FLAGTUNE=$USE_FLAGTUNE python -u -m pytest -s "$TEST_FILE" \
     "${PLUGINS[@]}" "${PYTEST_COLOR[@]}" \
     ${SHAPE_ARGS[@]+"${SHAPE_ARGS[@]}"} ${DTYPE_ARGS[@]+"${DTYPE_ARGS[@]}"} \
     --level "$BENCH_LEVEL" --mode kernel || status=$?
 
   rm -rf "$CACHE_DIR"
+  [[ -n "${TUNER_CACHE_DIR:-}" ]] && rm -rf "$TUNER_CACHE_DIR"
 
   if (( status == 0 )); then
     echo "${C_GREEN}>>> done. outputs in $OUT_DIR${C_RESET}"