|
Download BENCHMARK.md from Executespec/laxmi-controller-experimental: direct link, hf CLI and curl.
- Browser
- Download file 7.17 kB
-
https://huggingface.co/Executespec/laxmi-controller-experimental/resolve/main/BENCHMARK.md
- Command line
-
hf download hf://Executespec/laxmi-controller-experimental/BENCHMARK.md
-
curl -L -o BENCHMARK.md https://huggingface.co/Executespec/laxmi-controller-experimental/resolve/main/BENCHMARK.md
7.17 kB
| # Experimental release benchmark proposal | |
| 29 September 2026. This is a proposal; no new comparative benchmark, training or Jev/Laya API calls have been run. | |
| Comparator identities provisionally inferred: TypeSafe Jev and Convai Innovations | |
| Laya; identities taken from primary project documentation. Freeze versions and adapters before runs. | |
| ## Three distinct reports | |
| 1. Installation/replay: verify each of the four retained candidates from the exact | |
| staged release, against approved examples. Report platform, command, hashes, | |
| restore/prediction parity and failures. This is release usability, not accuracy. | |
| 2. Exposed development: retain the September 16 negative result unchanged. Score | |
| all four candidates and available simple baselines on identical admitted cases. | |
| Do not describe reused examples, new seeds or descendant configurations as an | |
| independent acceptance set. | |
| 3. Prospective task comparison: only after live input reconstruction and observation | |
| boundaries are qualified, freeze a new case matrix and save each forecast before | |
| either branch executes. Label within-family development separately from future | |
| independent implementation transfer. Do not block an honest experimental release | |
| on completion of this new scientific study. | |
| ## Shared question | |
| Under identical pre-action evidence and declared conditions, does the proposed | |
| guard edit alter recovery/completion at the second scheduled opportunity (H2)? | |
| Record each branch's head state, phase, terminal and completion presence. Use an | |
| explicit technical-effect target; a changed outcome is not automatically harm. | |
| H1 and H4/H8 are separately reported horizons, not pooled extra independent cases. | |
| Initially score shared per-branch endpoint questions and, where supported, their | |
| probabilities. Derive a paired change label only through a frozen adapter. Separate | |
| per-branch marginals do not determine the probability that counterfactual branches | |
| differ: a joint/coupling assumption or directly predicted effect is required. | |
| Never multiply marginal probabilities silently to invent that probability. | |
| ## Comparator arms and equal evidence | |
| - All four retained Laxmi candidates; no selection using the comparison test set. | |
| - Persistence/current-state baseline; majority/prior baseline fitted on development | |
| data only; applicable source rules with explicit refusal outside their premises. | |
| - Exact execution reference for outcome labels. A comparator that executes the | |
| tested future is an oracle, not a same-information pre-action predictor. | |
| - Frozen Jev version through its official interface, when access/cost is resolved. | |
| - Laya base checkpoint and typed-decisions checkpoint as separate zero-shot arms. | |
| A later domain-fitted Laya arm is separate and shares training examples/budget | |
| with other fitted candidates; do not mix fitted and zero-shot scores. | |
| For each arm derive a canonical pre-action evidence packet from source, state, | |
| history, budget, edit and declared conditions. Exclude future fault firing, outcomes, | |
| case names revealing labels, truth files and runtime secrets. Native typed inputs | |
| and external text serialization must represent the same facts. Publish adapter | |
| hashes and information losses. Record token limits and truncation; refuse a fair | |
| comparison if required evidence cannot fit. No external transmission of uncleared | |
| source/data. Output schemas differ; score only their shared supported questions. | |
| Do not assert Jev/Laya joint forecasts or count-head support without testing it. | |
| ## Case construction and truth | |
| Freeze case definitions and sampling before predictions. Include supported no-op, | |
| effect-producing edits, timing/acknowledgement conditions and missing-observation | |
| cases. Conditions beyond a frozen candidate's contract belong in an admission/ | |
| refusal test, not its predictive-accuracy denominator. New conditions require a | |
| separate candidate contract. Counterfactual branches share starting snapshots and | |
| declared exogenous schedules; save actual executed conditions and failures. | |
| Ground truth comes from native branch observations, not another model's answers. | |
| Unobserved invocation/endpoints remain censored, never fabricated zero. Count and | |
| report censoring per arm/case. Keep related trajectories/templates/implementation | |
| ancestries together when splitting and computing uncertainty. No row-level random | |
| split across descendants. Label domain limits and absent transfer evidence. | |
| Choose sample size after a pilot estimates prevalence, paired differences and | |
| effective independent groups. A small engineering screen can find bugs; it cannot | |
| justify strong accuracy or safety claims. Freeze a finite run/cost cap before | |
| collection and do not expand sampling opportunistically after inspecting scores. | |
| ## Scoreboard | |
| - Coverage/admission/refusal: all attempted cases and reasons; accuracy among | |
| answered cases plus risk versus coverage. Failures must not disappear. | |
| - Effect classification: balanced accuracy, per-class recall/precision, misses of | |
| actual effects and false alarms. Avoid calling technical effects harmful without | |
| a reviewer-use contract. | |
| - Endpoints: per-field accuracy and exact joint accuracy only for models that | |
| actually provide the joint task. Report each horizon separately. | |
| - Probabilities: Brier/log loss for the shared supported event, reliability plots, | |
| and raw versus development-calibrated results separately. Fit calibration only | |
| on development data; small-set ECE is not a calibration guarantee. | |
| - Controlled pairs: both branches correct, effect direction, no-op invariance; | |
| add option order/key changes and semantically equivalent serializations to | |
| detect presentation shortcuts. | |
| - Resources: end-to-end p50/p95 including serialization, network for APIs, device, | |
| dtype, batch size, cold/warm timing, peak memory and actual paid cost. Local | |
| inference uses hardware/electricity and is not economically free. | |
| - Uncertainty: paired differences with intervals grouped by independent ancestry | |
| or episode, disclosed group/sample counts and prevalence; no unjustified winner | |
| from point estimates alone. Predeclare the primary metric and secondary metrics. | |
| ## Publication and iteration | |
| First release: publish replay verification and existing negative development | |
| results, with this prospective comparison clearly marked not run. After a run, | |
| publish safe cases if rights permit, evaluator/adapters, version pins, raw safe | |
| predictions, aggregate results and failure/censoring counts. Preserve release tags. | |
| Public benchmark cases become development material after exposure. Retain a fresh | |
| sealed test set for later acceptance; do not repeatedly optimize a public leaderboard | |
| and call the result independent generalization. Reviewer-use pass/fail limits remain | |
| to be agreed separately; no thresholds are inferred from current failures. | |
| ## Primary comparison references | |
| https://typesafe.ai/blog/introducing-system-one-models-and-jev | |
| https://huggingface.co/convaiinnovations/laya | |
| https://github.com/NandhaKishorM/laya | |
| These describe external interfaces and author-reported results. They do not measure | |
| Jev/Laya on Laxmi inputs or establish a cross-project performance ranking. | |