eric-ml-nlp/Delveta-LayaChoice-v1

This version is deprecated. Use Delveta-LayaChoice-v2 instead.

Delveta LayaChoice v1

1. Model Summary

Delveta-LayaChoice-v1 is a capability-selection decision model: given a user turn and a short list of candidate capabilities, it returns the single capability that should handle the request. It is a full-parameter fine-tune of Laya's multilingual decision model on Delveta's internal capability-selection task, and it is not a general-purpose chat model.

Task	choose one capability from K <= 3 candidates
Base model	convaiinnovations/laya (multilingual), revision 55cf4c4e…
Encoder	jhu-clsp/mmBERT-base (~322M parameters)
Languages	Chinese (zh), English (en)
Export	FP32 (dtype preserve), model.safetensors = 1,287,653,720 bytes
Temperature	[2.038968563079834, 1.0, 1.0] (choice slot calibrated)
Runtime	laya==0.3.21

2. Intended Use

LayaChoice is a routing component inside Delveta's chat Intent Funnel. It is invoked only after the funnel's Recall stage has produced a candidate set and the business layer has normalized it (see §5). Its only job is to pick the winner from that set; it does not converse, does not author tool arguments, does not decide whether an action is permitted, and never sees open-world requests.

Intended use: ranking a closed, already-normalized set of 2–3 Delveta capabilities for a user turn, in Chinese or English.

Not intended for:

• open-world intent classification or general chat;
• tool-argument binding (a separate layer does that);
• safety adjudication (the funnel's write-gate owns that);
• anything outside the 17 capabilities that were frozen at training time.

3. Base Model

Repository	convaiinnovations/laya
Revision	55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851
Subfolder	multilingual
Encoder	jhu-clsp/mmBERT-base
Head	2-layer TransformerEncoder (head_layers=2) + type_emb(3, d) + a scalar scorer
Parameters	~322M (encoder-dominated)
Consumed via	laya Python package, pinned to 0.3.21 (inference-only; the package ships no trainer)

Training followed the structure of the upstream Laya fine-tuning recipe (dual loss = a policy-gradient term over a Gaussian perturbation of the logits plus a cross-entropy term against the gold one-hot), reimplemented against the laya runtime rather than through PEFT/Trainer.

4. Fine-tuning Task

Each example is one user turn plus exactly three candidate capability cards. The model scores each card and is trained with a hard one-hot target on the unique gold capability. Cards are rendered from the B_noprov view: the card carries the capability's description but no evidence: line and no other provenance that would leak the label.

The formal card view, the 17-capability roster and the card text are frozen; the same view was used for the zero-shot baseline, the fine-tune and the benchmark, so the comparison is like-for-like.

5. Candidate Contract

The production contract is defined over the candidate count K. Normalization happens in the business layer, before LayaChoice is ever invoked — the model only receives an already-normalized list:

K	Path
K = 0	not routed into LayaChoice (no candidates)
K = 1	business layer executes directly; LayaChoice is not consulted
K = 2	LayaChoice
K = 3	LayaChoice
K >= 4	business layer truncates to the top-3 first, then LayaChoice

The training and benchmark data are fixed at K = 3. Post-truncation behaviour for K >= 4 was not trained or measured — the model's decision head only ever sees three options.

6. Input Configuration

OPTION_CAP    = 256     # token budget per candidate capability card
head_max_len  = 768     # COMBINED instruction-head + options budget
max_len       = 1024    # whole-sequence ceiling

head_max_len is a budget shared by the instruction head and the concatenated option cards, not a per-option length and not the encoder's context limit. The encoder's own context window is larger; 1024 is an experiment-level sequence ceiling. The input builder is a verbatim copy of laya.common.build_sequence with exactly one line changed (max_length=48 → 256); the stock even-share fallback and the head-budget floor are preserved.

Deployment consequence: a stock laya install renders at 48 tokens per option and head_max_len=256. Reproducing this model's behaviour requires the same override (the same builder) and the same 256/768/1024 budget before the Agent is constructed. Under stock rendering expect a materially lower score.

7. Dataset

Split	Rows	Notes
Source train	856	the corpus the calibration split is drawn from
Calibration	85	fixed IDs, seed 20260922, removed from training
Effective train	771	856 − 85, after world-size alignment
Validation v3	150	used only for checkpoint selection
Final Test v3	900	used only for the final benchmark

• Candidate count K = 3; languages zh / en.
• Card view: B_noprov.
• The dataset is frozen: SHA256s of the raw and bundled files are recorded in the companion dataset repository (Delveta-LayaChoice-v1-data).
• Train / validation / test are disjoint; the calibration IDs are excluded from training by construction.

8. Training Setup

Method	full-parameter fine-tuning (no adapters)
Epochs	4
Micro-batch	8
Gradient accumulation	4
Group size (policy sampling)	4
LR (encoder / head)	2.5e-5 / 1.0e-4
Weight decay	0.01
Sigma schedule	0.4 → 0.1
Grad clip	1.0
Total optimizer updates	96
Precision	autocast bfloat16 (no GradScaler)
Hardware	1× NVIDIA L4
Runtime	torch 2.11.0+cu130, transformers 5.7.0, python 3.10.20

The best checkpoint is selected only by Validation v3 top-1 (ties resolved to the earlier epoch). The Final Test set is never used for checkpoint selection or for any training decision.

9. Validation

Epoch	Train loss	Val top-1	Val ECE	Val EN	Val ZH
1	0.320957	0.933333	0.065180	0.945205	0.922078
2 (best)	0.063713	0.986667	0.013017	0.986301	0.987013
3	0.056521	0.980000	0.016209	0.958904	1.000000
4	0.001806	0.980000	0.019153	0.958904	1.000000

Best epoch: 2, Validation top-1 = 98.67 % (150 rows).

10. Final Test

Best epoch 2, calibrated temperature, 900 rows:

Metric	Value
Top-1	93.5556 %
Hard-negative rate (= 1 − top-1)	6.4444 %
EN (445 rows)	91.4607 %
ZH (455 rows)	95.6044 %
ECE	0.029528
Mean confidence	0.959938
Mean gold probability	0.923789
Predicted gold / d1 / d2	842 / 42 / 16

11. Baselines

Configuration	Final Test top-1
Zero-shot base model, 256/768 (main baseline)	37.00 %
Fine-tuned LayaChoice v1, 256/768	93.5556 %
Delta	+56.5556 pp
Zero-shot base model, 48/256 (reference only)	57.8889 %

The 48/256 arm reproduces the original Laya input distribution and is reported as a reference only; it is not the configuration this model was trained or measured under. A historical "53.74 %" figure circulated earlier is not a valid unified baseline — see the experiment document for the mixed-configuration artifact that produced it.

12. Calibration

Temperature scaling is applied to the logits, not to probabilities. A single scalar is fitted for the choice slot and written into the length-3 temperature vector:

temperature = [2.038968563079834, 1.0, 1.0]
raw fitted value = final written value (clamp [0.5, 5.0] not engaged)
calibration rows  = 85 (seed 20260922)

Temperature is a positive scalar per question type, so it changes confidence but never the top-1 decision.

13. Reproducibility

• Base revision is pinned (55cf4c4e…); the laya package is pinned (0.3.21).
• Dataset SHA256s are frozen and published in the dataset repository.
• Training seed base 42; the calibration split seed is 20260922 with explicit, recorded row IDs.
• The export is byte-identical to the epoch-2 checkpoint (model.safetensors SHA256 8f139df0…).
• The export was reloaded through laya.Agent with strict=True and a 5-row smoke test; the calibrated temperature was read back unchanged.

Full method, ablations and invalid runs: see docs/experiments/LayaChoice-v1-Fine-Tuning.md.

14. Limitations

• 93.56 % is benchmark accuracy on the frozen Final Test v3 split. It is not production-traffic accuracy and not open-world generalization accuracy. The split is a fixed, curated 900-row artifact over 17 known capabilities with a known hard-negative construction.
• The model is trained and measured at K = 3 only. Behaviour for K = 2 and for post-truncation K >= 4 is untested.
• The dataset is synthetic / curated, not sampled from live traffic; it shares no distributional guarantees with production requests.
• Capabilities introduced after the freeze are out of scope; the model cannot route to a capability it has never seen.
• The deployment artifact's rl_agent_config.json carries a training metadata block inherited verbatim from the base checkpoint (values updates=15987, hours=4.97). These are base-model provenance values and do not describe this fine-tune (which ran 96 updates over ~84 s). The block is retained for byte-level continuity with the export; it is not a claim about this run.

15. Artifact Information

File	Detail
model.safetensors	1,287,653,720 bytes, FP32, sha256 8f139df0fed909f8b66c09e9064db424536bc90c274b6fd08ae4485edee20ba6
rl_agent_config.json	export config, fine-tuned flag, temperature vector
encoder/	jhu-clsp/mmBERT-base weights as exported
tokenizer/	tokenizer files
manifest.json	machine-readable release record
eval/	final-test report + rows, baseline reports, per-epoch validation reports

Licensing note. This repository does not declare a license. The laya software package is distributed under Apache-2.0, but the license of the base model weights (convaiinnovations/laya) could not be determined at release time — the upstream repository is no longer publicly reachable and its snapshot carries no LICENSE or model card. The dataset is Delveta-curated/synthetic. No license is asserted here for the weights or the dataset; downstream use should confirm terms with the upstream rights holder. This artifact is publicly hosted; no license is asserted.

16. Citation / References

If you build on this model, cite the base model and the Delveta experiment record:

@misc{delveta_layachoice_v1,
  title  = {Delveta LayaChoice v1: a capability-selection decision model},
  author = {Delveta},
  year   = {2026},
  note   = {Fine-tuned from convaiinnovations/laya@55cf4c4e (multilingual)},
}

Base model: convaiinnovations/laya, revision 55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851, subfolder multilingual. Encoder: jhu-clsp/mmBERT-base. Runtime: laya==0.3.21.

Source code

The fine-tuning code for this model lives in the LayaChoice v1 track of the Delveta repository: fine_tuning/laya_finetune/V1/ — the training entry layachoice_finetune.py, the prepare/eval modules (layachoice_prepare.py, layachoice_eval.py), and a one-command run.sh.