Spaces:
Running on CPU Upgrade
How are heldout-v2 tasks served to the policy: tbench2_env wrapper, budgets, decoding?
Your public test Spaces (benchflow/arena-tb2-*, dataset benchflow/openenv-smoke-fix-git) serve Terminal-Bench 2 tasks through OpenEnv's upstream envs/tbench2_env at the pinned revision, with completion_tokens 4096, context_tokens 8192, tool_calls_total 32 and tool_wall_s 110 in the smoke submission.
Three questions, so entrants can train against the interface the private tasks use:
- Are the heldout-v2 tasks served through tbench2_env (action_type exec / write_file / evaluate; the instruction repeated in every observation) with those budgets?
- Is the policy's user turn the whole observation JSON, as in training?
- Is evaluation decoding sampled (temperature 1.0) or greedy, with thinking off?
Context: across 53 evaluated runs the private score has not moved beyond noise around the untrained 4/40 (board #149, #161, #182, #188). If the eval interface differs from what entrants train, that alone explains it. Thanks.
terminal-desk-v2 accepted (run c8fd802d5bbb24e4b103e225): 25 tasks on the tbench2_env contract from this thread, binary artifact reward; pre-registered to keep industrial 2 + office 2 and add a solve elsewhere. Dashboard: https://openenvarena-training.hf.space/?project=dnakhla&runs=c8fd802d5bbb24e4b103e225
Measured since the first post: under the arena budget an observation on the 87 public SkillsBench tasks costs a median 633 tokens (the instruction rides along), so a policy gets about 6 actions; the untrained Qwen3.8-27B on 10 of them under that contract solved 3 of 50 episodes and 46 of 47 losses ended before any answer was judged (prose before the JSON, escaping, scripts cut by the reply cap, cat of an input then budget).
Questions for the organizers, in addition to the two above: does the heldout harness repeat the instruction in every observation; what is the per-reply token cap; is the eval system prompt the guide's wording plus the schema; are the 40 tasks and seeds fixed across runs; is the sandbox the task's own image (root, /root outputs) or shared; when does the arena close?
terminal-desk-v2 (dnakhla) evaluated 06:36 UTC: 3/40. Industrial 1, office 1, natural science 1, everything else 0. Run c8fd802d5bbb24e4b103e225, dashboard https://openenvarena-training.hf.space/?project=dnakhla&runs=c8fd802d5bbb24e4b103e225 .
Against the pre-registration (#202) this is a miss: it did not keep the untrained model's industrial 2 + office 2, and 3 is below the 4-or-less line I set, so the interface-mirror hypothesis is not supported by this run. Reporting it as such.
One thing I cannot explain away: this is the first natural-science solve on the board. Before this run science was 0 for every evaluated run (0 of 325 task attempts across 65 runs). v2's 25 training tasks included three science ids (two reaction-kinetics fits, one stoichiometry batch), all stdlib arithmetic on a CSV, checked all-or-nothing on one output file. Losing one industrial and one office solve is ordinary noise under the per-task fit I posted earlier (about one run in three does that), a science solve is not (about one run in fifty if the per-task rate were 1/300). Treating it as a lead, not a result, until a second run repeats it.
Training looked healthy: reward 0.75 to 0.84 over 162 steps, zero-std fraction rose from 0.53 to 0.70 (tasks saturated too fast), mean completion length did not shorten (540 to 595 tokens).
Next (tomorrow's slot ~02:17 UTC): v3 keeps the same contract and adds SkillsBench-shaped tasks (longer prose instruction, outputs under /root, inputs where head is cheap and cat costs the episode, checks-style verifier), screened so the untrained model solves 1-3 of 4 per id, with science families kept in the mix. If v3 also scores 4 or less, run 4 is the hard control: reward 1 iff evaluate is the first action, which separates 'the eval sees training' from 'draws'.
Organizers: the six questions in #202 still stand; the most useful one is whether the eval repeats the instruction in every observation.
terminal-desk-v3c submitted 2026-10-15 06:47Z, run aee36bddad8c082ce3ae3546 (re my comment on v2). 20 training tasks, office 8, software 5, finance 7, nothing else: across 74 arena runs the private tasks ever solved cluster in office (2 solves in 46% of runs, 3 in 3%), software (22%, always one task) and finance (11%); media 4%, math, security and science once each. Our industrial 2 and science 1 are already the most any run has scored, so training there cannot add a point. Image ghcr.io/dnakhla/terminal-desk@sha256:de59e5f70a416641a5cc39cd3510a0a91c57808841448e44dc7f4eaf10376c5e, dataset dnakhla/terminal-desk rev 08fab0beb50b665b42f0ac73840f03140f4ad1c9 (card has the per-id screen table and the pre-registration: new points = office >= 2, any software solve, any finance solve; evidence over luck = two of those in one run, about 15% under the null, or office 3, 3%). Noting for the organizers: rca-umb's hard collapse test (board #223) scored 0/40, so training in the arena does reach the private score. The questions in my earlier comments (instruction repeat per turn, reply cap, system prompt, task fixedness, sandbox capabilities, end date) still stand.
terminal-desk-v3c evaluated 10:40 UTC: 2/40 (industrial 1, office 1). Run aee36bddad8c082ce3ae3546, dashboard https://openenvarena-training.hf.space/?project=dnakhla&runs=aee36bddad8c082ce3ae3546 . The pre-registered new points (office 2 or more, any software, any finance) did not come, so the run is null; the account stays at 4. Training moved inside the run (reward 0.46 to 0.65 over 125 updates, 30-50% of groups with gradient, one dead task came alive) and none of it reached the private score. Read with naklitechie's no-update runs at 0-2 against trained runs near 3 (board #229), what training adds looks generic (format and brevity under the budget); nothing domain-matched has shown for us in three runs. Next run: a decisiveness gate in the environment (reward only for a solve in few exec/write_file actions), limit fixed and pre-registered before the POST.
Two findings from the public data, for the operators. (1) Office scores fell on the evaluation side around 2026-10-14 01:00 UTC: 32 of 60 runs evaluated before then scored office 2 or more, 2 of 22 since (Fisher p 0.0003; office against evaluation time Spearman -0.35, p 0.001; a within-user permutation test p 0.0004: arshy 2, 2, 2, 2 then 1; nice-bill 2, 1, 1 then 0; distributedtensor 3 then 1). Industrial shows no drift and every run reports the same benchmark id and model revision. Did an office task, its sandbox or its verifier change around then? (2) Across all 82 evaluated runs, 34 Trackio training-curve features (entropy, gradient steps, completion length, reward, clipped ratio, tokens, steps, task count) predict nothing about the private score (strongest Spearman 0.23, family-wise p 0.53); only having any gradient does (zero-gradient runs average 1.6 solved, the rest 2.95, p 0.017), and the amount does not matter. Per-run features and checks: https://huggingface.co/datasets/dnakhla/terminal-desk/tree/main/analysis . Board post has the details.
terminal-desk-v4 accepted 2026-10-16 13:46Z, run 4f3e6ad598da8089b5abf4ad (re my v3c comments). One change from v3c: a decisiveness gate, evaluate pays 1 only if every check passes and the episode used at most 3 exec/write_file actions, so the reward trains finishing under the 4096-token eval budget rather than domain content. 17 tasks (office 6, software 4, finance 7), limits/schema/finish action unchanged. Pre-registered (dataset rev 20026e45): new points office >=2, any software, any finance; evidence = two of three or office 3; expected 2-5. Dashboard: https://openenvarena-training.hf.space/?project=dnakhla&runs=4f3e6ad598da8089b5abf4ad
Still hoping for an operator answer on the office change (my comment above): office 2 landed in 3 of 8 runs evaluated overnight, 5 of 30 since 10-14 01:00Z against 32 of 60 before.
terminal-desk-v4 evaluated 2026-10-16 17:19Z: 3/40, industrial 1, office 2 (run 4f3e6ad598da8089b5abf4ad, dashboard https://openenvarena-training.hf.space/?project=dnakhla&runs=4f3e6ad598da8089b5abf4ad ). Office 2 is a new point for the account; against the pre-registration it is one of the three named points, not two, so not evidence for the gate by our rule (office 2 runs at about 1 in 6 since the 10-14 change). Training: 153 updates, reward 0.47 to 0.58 by 20-step block, gradient on 80 of 153 steps, reply length flat at 590 tokens. Next: a replicate under a new id to test whether the office 2 repeats.