QwenJev v0.4.3: 25-public-games eval in-repo (run_eval incremental report, fetch_games, CPU feasibility verified)

HANDOVER.md

@@ -5,19 +5,34 @@
 
 ## 0. 一句话现状
 
-**RCA 已定案 + v0.4.9 视觉读出修复已实施，v4 重训中**。v3 阴性结果（0 关 0 胜）的根因
+**RCA 已定案 + v0.4.9 视觉读出修复已实施；v4 重训在 real_b 完成后（止损 s160，
+loss 0.25-0.55）遇 T4 回收，等待新 T4 恢复 real_c**。v3 阴性结果（0 关 0 胜）的根因
 不是数据而是架构：`pooled_last_position` 从序列末位读出，帧内容信号被淹没
 （实测： radically different frames 的 last-pos cos=0.991，真游戏上 0.9999；而视觉塔
 在图像 token 位本身活得好好的 per-token cos=0.60）。**A→B→C 三阶段的决策头从未见过
 帧内容** —— 候选分布坍缩（ACTION3 占 70%）、视觉塔形同虚设。修复：池化改为
 `[last_text; mean(vision_tokens)]`（2H），头部首层扩维，检查点记录 `pool_vision`
-标志（旧 v3 检查点自动兼容，build_s1 自动探测）。v4 重训（现有 167k 数据）+
-ε0.45 重采（CPU 并行）已同时启动。
+标志（旧 v3 检查点自动兼容，build_s1 自动探测）。修复直接兑现在训练信号上：
+同数据同预算，v4 real_b loss 在 s56 即降到 0.485（v3 同期 3-9 震荡）。
 
 v3 历史结论保留：real_b(SFT) 收敛止损 + real_c(RLCD) ECE 0.4151→0.2727，
 对照评测 ar25+bp35 各 150 步 0 关 0 胜；产物在 HF `t4_assets_v3/checkpoints/`。
 v0.4.8 铁律：**权重一律 .pt 入仓 + HEAD 验证**（HF 静默剔除 .safetensors/.bin，坑 9）。
 
+**T4 FUKJA61G 已回收（~5.8h 超时，502）—— 死于 v4 real_c 中途（~step 60-100，无 WIP）**。
+损失：① real_c 进度（Stage C 无断点保险，重跑 ~50min）；② ε0.45 重采 traces_real3
+（275+ 集未推 HF，重采 ~2h，CPU 与训练并行不额外占时间）。其余全部安全：v4
+Stage B 权重（s160 止损版，loss 0.25-0.55）已 HEAD 验证在 HF
+`t4_assets_v4/checkpoints/hf_lora/adapter_model.pt`。
+
+**新 T4 恢复（一键）**：标准 bootstrap 装好代码+依赖后：
+```bash
+cd /content/qwenjev && QWENJEV_OUT=qwenjev-s1-hf-v4 QWENJEV_ASSETS=t4_assets_v4 \
+    nohup python3 -u -B scripts/v4_recover_bootstrap.py > /content/v4_recover.log 2>&1 &
+```
+自动：抽 dataset_final.tgz → 拉 v4 权重到 stage_a_lora/hf_lora → 盖 prep/data/real_b 章
+→ 直进 real_c → ALL_DONE → 对照评测。之后合并 traces_real3 重训 v4b（见 §6）。
+
 ## 0.4 RCA 实录（v0.4.9，复现工具已入仓）
 
 | 探针 | 读数 | 结论 |

scripts/v4_recover_bootstrap.py

@@ -0,0 +1,100 @@
+#!/usr/bin/env python3
+"""One-command v4 recovery bootstrap for a FRESH Colab T4.
+
+Context (2026-09-23): the FUKJA61G T4 died mid-real_c (uptime ~5.8h).
+Everything that matters is on HF:
+  * code v0.4.9 (vision-aware pooling fix)           -> repo root
+  * v4 Stage B weights (loss 0.25-0.55 @ s160-164)   -> t4_assets_v4/checkpoints/hf_lora/adapter_model.pt
+  * the merged 167,635-sample dataset                -> t4_assets_v3/data/dataset_final.tgz
+  * HANDOVER with full RCA                           -> HANDOVER.md
+
+This script (run INSIDE the fresh T4, after the standard code+deps bootstrap):
+  1. extracts dataset_final.tgz            -> data/dataset_final  (data stage equivalent)
+  2. pulls v4 hf_lora .pt                  -> checkpoints/qwenjev-s1-hf-v4/stage_a_lora + hf_lora
+     (Stage B is NOT re-run: s160 weights are the operator-approved stop)
+  3. stamps prep/data/real_b as done       -> orchestrator goes straight to real_c
+  4. launches the orchestrator (GPU)       -> real_c -> ALL_DONE -> upload
+  5. launches the eps0.45 collector (CPU)  -> data/traces_real3 (in parallel; ~2h)
+
+Usage (on the T4, inside /content/qwenjev after deps install):
+  QWENJEV_OUT=qwenjev-s1-hf-v4 QWENJEV_ASSETS=t4_assets_v4 \
+      nohup python3 -u -B scripts/v4_recover_bootstrap.py > /content/v4_recover.log 2>&1 &
+"""
+
+import json
+import os
+import shutil
+import subprocess
+import sys
+import tarfile
+import time
+from pathlib import Path
+
+REPO = Path("/content/qwenjev")
+WORK = Path(os.environ.get("QWENJEV_WORK", "/content/qwenjev_work_v3"))
+OUT = REPO / "checkpoints" / os.environ.get("QWENJEV_OUT", "qwenjev-s1-hf-v4")
+ASSETS = os.environ.get("QWENJEV_ASSETS", "t4_assets_v4")
+DONE = WORK / "done_v3"
+HF_REPO = "tchbcb/qwenjev"
+
+os.chdir(REPO)
+token = None
+for cand in (os.environ.get("HF_TOKEN"), WORK / ".hf_token"):
+    if cand and str(cand).strip() and os.path.exists(str(cand)):
+        token = open(str(cand)).read().strip()
+        break
+if not token:
+    raise SystemExit("no HF_TOKEN / .hf_token found")
+os.environ["HF_TOKEN"] = token
+
+from huggingface_hub import hf_hub_download  # noqa: E402
+
+print("[recover] 1/4 dataset_final", flush=True)
+ds = REPO / "data" / "dataset_final"
+if not (ds / "train" / "samples.jsonl").exists():
+    tgz = hf_hub_download(HF_REPO, f"{ASSETS}/data/dataset_final.tgz"
+                          if False else "t4_assets_v3/data/dataset_final.tgz")
+    ds.mkdir(parents=True, exist_ok=True)
+    with tarfile.open(tgz) as t:
+        t.extractall(REPO)          # archive root is data/dataset_final
+print("   train rows:", sum(1 for _ in open(ds / "train" / "samples.jsonl")), flush=True)
+
+print("[recover] 2/4 v4 stage_b weights -> stage_a_lora + hf_lora", flush=True)
+pt = hf_hub_download(HF_REPO, f"{ASSETS}/checkpoints/hf_lora/adapter_model.pt")
+for sub in ("stage_a_lora", "hf_lora"):
+    d = OUT / sub
+    d.mkdir(parents=True, exist_ok=True)
+    shutil.copyfile(pt, d / "adapter_model.safetensors")
+    for f in ("adapter_config.json", "README.md"):
+        try:
+            src = hf_hub_download(HF_REPO, f"{ASSETS}/checkpoints/hf_lora/{f}")
+            shutil.copyfile(src, d / f)
+        except Exception:
+            pass
+meta = hf_hub_download(HF_REPO, f"{ASSETS}/checkpoints/stage_b_meta.json")
+shutil.copyfile(meta, OUT / "stage_b_meta.json")
+print("   restored ->", OUT, flush=True)
+
+print("[recover] 3/4 stamps", flush=True)
+DONE.mkdir(parents=True, exist_ok=True)
+now = time.strftime("%F %T")
+for name in ("prep", "data", "real_b"):
+    (DONE / f"{name}.stamp").write_text(
+        f"{now} (v4 recovery: prep=v2 lora re-seed, data=dataset_final from HF, "
+        "real_b=s160 operator-approved weights from HF)")
+print("   stamps:", sorted(p.name for p in DONE.glob("*.stamp")), flush=True)
+
+print("[recover] 4/4 launch orchestrator (GPU) + collector (CPU)", flush=True)
+env = {**os.environ, "QWENJEV_OUT": os.environ.get("QWENJEV_OUT", "qwenjev-s1-hf-v4"),
+       "QWENJEV_ASSETS": ASSETS, "PYTHONUNBUFFERED": "1"}
+subprocess.Popen(["python3", "-u", "-B", "ops/t4_real25.py"],
+                 cwd=str(REPO), stdout=open("/content/t4_v4.log", "w"),
+                 stderr=subprocess.STDOUT, env=env,
+                 start_new_session=True)
+subprocess.Popen(["python3", "-u", "-B", "-m", "data.collect_selfplay",
+                  "--config", "configs/collect_real_v4.yaml"],
+                 cwd=str(REPO), stdout=open("/content/collect_v4.log", "w"),
+                 stderr=subprocess.STDOUT, env=env,
+                 start_new_session=True)
+print("[recover] launched: orchestrator -> real_c; collector -> traces_real3", flush=True)
+print("[recover] monitor: tail /content/t4_v4.log /content/collect_v4.log", flush=True)