# Experimental release benchmark proposal 29 September 2026. This is a proposal; no new comparative benchmark, training or Jev/Laya API calls have been run. Comparator identities provisionally inferred: TypeSafe Jev and Convai Innovations Laya; identities taken from primary project documentation. Freeze versions and adapters before runs. ## Three distinct reports 1. Installation/replay: verify each of the four retained candidates from the exact staged release, against approved examples. Report platform, command, hashes, restore/prediction parity and failures. This is release usability, not accuracy. 2. Exposed development: retain the September 16 negative result unchanged. Score all four candidates and available simple baselines on identical admitted cases. Do not describe reused examples, new seeds or descendant configurations as an independent acceptance set. 3. Prospective task comparison: only after live input reconstruction and observation boundaries are qualified, freeze a new case matrix and save each forecast before either branch executes. Label within-family development separately from future independent implementation transfer. Do not block an honest experimental release on completion of this new scientific study. ## Shared question Under identical pre-action evidence and declared conditions, does the proposed guard edit alter recovery/completion at the second scheduled opportunity (H2)? Record each branch's head state, phase, terminal and completion presence. Use an explicit technical-effect target; a changed outcome is not automatically harm. H1 and H4/H8 are separately reported horizons, not pooled extra independent cases. Initially score shared per-branch endpoint questions and, where supported, their probabilities. Derive a paired change label only through a frozen adapter. Separate per-branch marginals do not determine the probability that counterfactual branches differ: a joint/coupling assumption or directly predicted effect is required. Never multiply marginal probabilities silently to invent that probability. ## Comparator arms and equal evidence - All four retained Laxmi candidates; no selection using the comparison test set. - Persistence/current-state baseline; majority/prior baseline fitted on development data only; applicable source rules with explicit refusal outside their premises. - Exact execution reference for outcome labels. A comparator that executes the tested future is an oracle, not a same-information pre-action predictor. - Frozen Jev version through its official interface, when access/cost is resolved. - Laya base checkpoint and typed-decisions checkpoint as separate zero-shot arms. A later domain-fitted Laya arm is separate and shares training examples/budget with other fitted candidates; do not mix fitted and zero-shot scores. For each arm derive a canonical pre-action evidence packet from source, state, history, budget, edit and declared conditions. Exclude future fault firing, outcomes, case names revealing labels, truth files and runtime secrets. Native typed inputs and external text serialization must represent the same facts. Publish adapter hashes and information losses. Record token limits and truncation; refuse a fair comparison if required evidence cannot fit. No external transmission of uncleared source/data. Output schemas differ; score only their shared supported questions. Do not assert Jev/Laya joint forecasts or count-head support without testing it. ## Case construction and truth Freeze case definitions and sampling before predictions. Include supported no-op, effect-producing edits, timing/acknowledgement conditions and missing-observation cases. Conditions beyond a frozen candidate's contract belong in an admission/ refusal test, not its predictive-accuracy denominator. New conditions require a separate candidate contract. Counterfactual branches share starting snapshots and declared exogenous schedules; save actual executed conditions and failures. Ground truth comes from native branch observations, not another model's answers. Unobserved invocation/endpoints remain censored, never fabricated zero. Count and report censoring per arm/case. Keep related trajectories/templates/implementation ancestries together when splitting and computing uncertainty. No row-level random split across descendants. Label domain limits and absent transfer evidence. Choose sample size after a pilot estimates prevalence, paired differences and effective independent groups. A small engineering screen can find bugs; it cannot justify strong accuracy or safety claims. Freeze a finite run/cost cap before collection and do not expand sampling opportunistically after inspecting scores. ## Scoreboard - Coverage/admission/refusal: all attempted cases and reasons; accuracy among answered cases plus risk versus coverage. Failures must not disappear. - Effect classification: balanced accuracy, per-class recall/precision, misses of actual effects and false alarms. Avoid calling technical effects harmful without a reviewer-use contract. - Endpoints: per-field accuracy and exact joint accuracy only for models that actually provide the joint task. Report each horizon separately. - Probabilities: Brier/log loss for the shared supported event, reliability plots, and raw versus development-calibrated results separately. Fit calibration only on development data; small-set ECE is not a calibration guarantee. - Controlled pairs: both branches correct, effect direction, no-op invariance; add option order/key changes and semantically equivalent serializations to detect presentation shortcuts. - Resources: end-to-end p50/p95 including serialization, network for APIs, device, dtype, batch size, cold/warm timing, peak memory and actual paid cost. Local inference uses hardware/electricity and is not economically free. - Uncertainty: paired differences with intervals grouped by independent ancestry or episode, disclosed group/sample counts and prevalence; no unjustified winner from point estimates alone. Predeclare the primary metric and secondary metrics. ## Publication and iteration First release: publish replay verification and existing negative development results, with this prospective comparison clearly marked not run. After a run, publish safe cases if rights permit, evaluator/adapters, version pins, raw safe predictions, aggregate results and failure/censoring counts. Preserve release tags. Public benchmark cases become development material after exposure. Retain a fresh sealed test set for later acceptance; do not repeatedly optimize a public leaderboard and call the result independent generalization. Reviewer-use pass/fail limits remain to be agreed separately; no thresholds are inferred from current failures. ## Primary comparison references https://typesafe.ai/blog/introducing-system-one-models-and-jev https://huggingface.co/convaiinnovations/laya https://github.com/NandhaKishorM/laya These describe external interfaces and author-reported results. They do not measure Jev/Laya on Laxmi inputs or establish a cross-project performance ranking.