yxc20098 commited on
Commit
d608892
·
1 Parent(s): 9000fe3

Phase 4 fix: run_eval double-nest in full_playback dir

Browse files

run_eval was appending <run_id>__<model> inside full_playback_root,
producing <out>/<ts>__<model>/<ts>__<model>/<pack>__... when the
caller (scripts/collect_eval_data.py) already builds the per-model
dir. Now: if the leaf already looks like a per-model dir (starts
with run_id or contains the model safe-name), treat it as final;
otherwise append for back-compat with the legacy callers.

Verified: smoke test produces single-level dir with JSONL + per-
turn PNGs, complete=True on a 1-cell Qwen3.5-9B win.

Files changed (1) hide show
  1. openra_bench/run_eval.py +16 -1
openra_bench/run_eval.py CHANGED
@@ -359,8 +359,23 @@ def evaluate(
359
  _fog = getattr(compiled, "fog_mode", "vision") or "vision"
360
  if len(parts) >= 3:
361
  _fog = parts[-1]
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
362
  fpb = FullPlayback(
363
- Path(full_playback_root) / f"{run_id}__{_safe_model}",
364
  pack_id=_pack_id,
365
  level=_level,
366
  seed=seed,
 
359
  _fog = getattr(compiled, "fog_mode", "vision") or "vision"
360
  if len(parts) >= 3:
361
  _fog = parts[-1]
362
+ # `full_playback_root` is treated as the FINAL per-model dir
363
+ # — callers (e.g. scripts/collect_eval_data.py) already
364
+ # build `<out>/<timestamp>__<model>` and pass it through. We
365
+ # previously appended `<run_id>__<model>` here which
366
+ # produced a double-nested path; if the caller supplied a
367
+ # plain root we still want a per-model subdir, but only if
368
+ # the path doesn't already look like one. Heuristic: if the
369
+ # leaf already starts with the run_id or contains the model
370
+ # safe-name, treat it as final; otherwise append.
371
+ _fp_root = Path(full_playback_root)
372
+ _leaf = _fp_root.name
373
+ if (run_id and _leaf.startswith(run_id)) or _safe_model in _leaf:
374
+ _fp_dir = _fp_root
375
+ else:
376
+ _fp_dir = _fp_root / f"{run_id}__{_safe_model}"
377
  fpb = FullPlayback(
378
+ _fp_dir,
379
  pack_id=_pack_id,
380
  level=_level,
381
  seed=seed,