adminspec's picture
Release Laxmi 0.1.0 experimental controller package
4b5ca98 verified
|
Raw History Blame Contribute Delete
7.17 kB
# Experimental release benchmark proposal
29 September 2026. This is a proposal; no new comparative benchmark, training or Jev/Laya API calls have been run.
Comparator identities provisionally inferred: TypeSafe Jev and Convai Innovations
Laya; identities taken from primary project documentation. Freeze versions and adapters before runs.
## Three distinct reports
1. Installation/replay: verify each of the four retained candidates from the exact
staged release, against approved examples. Report platform, command, hashes,
restore/prediction parity and failures. This is release usability, not accuracy.
2. Exposed development: retain the September 16 negative result unchanged. Score
all four candidates and available simple baselines on identical admitted cases.
Do not describe reused examples, new seeds or descendant configurations as an
independent acceptance set.
3. Prospective task comparison: only after live input reconstruction and observation
boundaries are qualified, freeze a new case matrix and save each forecast before
either branch executes. Label within-family development separately from future
independent implementation transfer. Do not block an honest experimental release
on completion of this new scientific study.
## Shared question
Under identical pre-action evidence and declared conditions, does the proposed
guard edit alter recovery/completion at the second scheduled opportunity (H2)?
Record each branch's head state, phase, terminal and completion presence. Use an
explicit technical-effect target; a changed outcome is not automatically harm.
H1 and H4/H8 are separately reported horizons, not pooled extra independent cases.
Initially score shared per-branch endpoint questions and, where supported, their
probabilities. Derive a paired change label only through a frozen adapter. Separate
per-branch marginals do not determine the probability that counterfactual branches
differ: a joint/coupling assumption or directly predicted effect is required.
Never multiply marginal probabilities silently to invent that probability.
## Comparator arms and equal evidence
- All four retained Laxmi candidates; no selection using the comparison test set.
- Persistence/current-state baseline; majority/prior baseline fitted on development
data only; applicable source rules with explicit refusal outside their premises.
- Exact execution reference for outcome labels. A comparator that executes the
tested future is an oracle, not a same-information pre-action predictor.
- Frozen Jev version through its official interface, when access/cost is resolved.
- Laya base checkpoint and typed-decisions checkpoint as separate zero-shot arms.
A later domain-fitted Laya arm is separate and shares training examples/budget
with other fitted candidates; do not mix fitted and zero-shot scores.
For each arm derive a canonical pre-action evidence packet from source, state,
history, budget, edit and declared conditions. Exclude future fault firing, outcomes,
case names revealing labels, truth files and runtime secrets. Native typed inputs
and external text serialization must represent the same facts. Publish adapter
hashes and information losses. Record token limits and truncation; refuse a fair
comparison if required evidence cannot fit. No external transmission of uncleared
source/data. Output schemas differ; score only their shared supported questions.
Do not assert Jev/Laya joint forecasts or count-head support without testing it.
## Case construction and truth
Freeze case definitions and sampling before predictions. Include supported no-op,
effect-producing edits, timing/acknowledgement conditions and missing-observation
cases. Conditions beyond a frozen candidate's contract belong in an admission/
refusal test, not its predictive-accuracy denominator. New conditions require a
separate candidate contract. Counterfactual branches share starting snapshots and
declared exogenous schedules; save actual executed conditions and failures.
Ground truth comes from native branch observations, not another model's answers.
Unobserved invocation/endpoints remain censored, never fabricated zero. Count and
report censoring per arm/case. Keep related trajectories/templates/implementation
ancestries together when splitting and computing uncertainty. No row-level random
split across descendants. Label domain limits and absent transfer evidence.
Choose sample size after a pilot estimates prevalence, paired differences and
effective independent groups. A small engineering screen can find bugs; it cannot
justify strong accuracy or safety claims. Freeze a finite run/cost cap before
collection and do not expand sampling opportunistically after inspecting scores.
## Scoreboard
- Coverage/admission/refusal: all attempted cases and reasons; accuracy among
answered cases plus risk versus coverage. Failures must not disappear.
- Effect classification: balanced accuracy, per-class recall/precision, misses of
actual effects and false alarms. Avoid calling technical effects harmful without
a reviewer-use contract.
- Endpoints: per-field accuracy and exact joint accuracy only for models that
actually provide the joint task. Report each horizon separately.
- Probabilities: Brier/log loss for the shared supported event, reliability plots,
and raw versus development-calibrated results separately. Fit calibration only
on development data; small-set ECE is not a calibration guarantee.
- Controlled pairs: both branches correct, effect direction, no-op invariance;
add option order/key changes and semantically equivalent serializations to
detect presentation shortcuts.
- Resources: end-to-end p50/p95 including serialization, network for APIs, device,
dtype, batch size, cold/warm timing, peak memory and actual paid cost. Local
inference uses hardware/electricity and is not economically free.
- Uncertainty: paired differences with intervals grouped by independent ancestry
or episode, disclosed group/sample counts and prevalence; no unjustified winner
from point estimates alone. Predeclare the primary metric and secondary metrics.
## Publication and iteration
First release: publish replay verification and existing negative development
results, with this prospective comparison clearly marked not run. After a run,
publish safe cases if rights permit, evaluator/adapters, version pins, raw safe
predictions, aggregate results and failure/censoring counts. Preserve release tags.
Public benchmark cases become development material after exposure. Retain a fresh
sealed test set for later acceptance; do not repeatedly optimize a public leaderboard
and call the result independent generalization. Reviewer-use pass/fail limits remain
to be agreed separately; no thresholds are inferred from current failures.
## Primary comparison references
https://typesafe.ai/blog/introducing-system-one-models-and-jev
https://huggingface.co/convaiinnovations/laya
https://github.com/NandhaKishorM/laya
These describe external interfaces and author-reported results. They do not measure
Jev/Laya on Laxmi inputs or establish a cross-project performance ranking.