Thoriumi/Fuuga-Laya-emotion

Laya, retrained with Fuuga on Emotion

What this is. Laya, a typed-decision model (a ModernBERT-large encoder, 395M parameters, with its own two-layer decision head), imported whole into Fuuga (an F# ML framework on TorchSharp) with fuuga import-laya, fine-tuned, head and encoder together, on DAIR Emotion (6 classes), and exported: the encoder as Transformers' ModernBertModel, the decision head as Fuuga holds it.

Why it exists. A test case for Fuuga's decision path end to end: import a published decision model exactly, measure it, retrain it on labelled data, measure it again on the same rows, export it, and rebuild it from the export alone. Emotion is the set where Laya was weak (58.7% zero-shot on these rows), so retraining has something to show.

Files

File
model.safetensors, config.json	the fine-tuned encoder as ModernBertModel, F32 (1.6 GB). config.json carries the RoPE bases in both the Transformers 4 (global_rope_theta / local_rope_theta) and 5 (rope_parameters, layer_types) spellings
tokenizer.json, tokenizer_config.json	Laya's tokenizer, unchanged
fuuga/decision.json, fuuga/marker_heads.dat, fuuga/marker_heads.json	Laya's decision head, retrained, in Fuuga's format (156 MB): type embedding, two transformer layers and the option-marker scorer. Laya's act head is not retrained and not used
fuuga_reference.json	Fuuga's own last hidden states for three texts, one longer than the 128-token local window
verify-modernbert-export.fsx	checks this folder without Python: reads config.json and model.safetensors as ModernBertModel does, runs its forward in plain TorchSharp and compares with fuuga_reference.json (dotnet fsi verify-modernbert-export.fsx -- .)
verify_modernbert_export.py	the same comparison through Transformers itself, where Python is available
eval/	the evaluation reports of every model below, the paired comparisons and the training logs
FUUGA_MODERNBERT_EXPORT_NOTES.md	what the export wrote and left out

Training

Data	DAIR Emotion (dair-ai/emotion): a fixed-seed sample of 4,000 training rows and 2,000 test rows from the published splits; the question "Which emotion does this message express?" over sadness, joy, love, anger, fear, surprise
Start	fuuga import-laya of Laya's root checkpoint: encoder, decision head and Laya's input layout, checked against Laya's own outputs (identical token ids and option markers, logits within 4e-6)
Method	fuuga decide train --decision-model: 3 epochs, batch 16, head at 1e-4, encoder at 2e-5, loss the log score (a strictly proper scoring rule), 512-token input budget (192 for question and options)
Hardware and time	RTX 4070 12 GB; 6.5 minutes

Results

Held-out accuracy on the 2,000 test rows (chance 16.7%). The stock answerdotai/ModernBERT-large with Fuuga's own heads, trained the same way, is the control:

Model	Accuracy	NLL	Brier	ECE	Acts at confidence ≥ 0.90
Laya zero-shot, with its own temperatures	58.7%	1.963	0.702	0.305	68.3% of rows, 66.2% correct
Laya, head retrained, encoder frozen	58.6%	1.108	0.552	0.019	4.7% of rows, 92.5% correct
Laya, fine-tuned (this model)	89.9%	0.525	0.173	0.074	92.3% of rows, 93.6% correct
ModernBERT-large stock, heads trained, encoder frozen	47.6%	1.367	0.667	0.015	0.1% of rows
ModernBERT-large stock, fine-tuned	90.0%	0.348	0.157	0.060	88.7% of rows, 94.5% correct

Paired on the same rows (McNemar's exact test over the rows two models decide differently):

• This model against Laya zero-shot: right on 679 of the 734 split rows, p < 0.0001.
• This model against the stock ModernBERT-large fine-tuned the same way: 71 against 74, p = 0.87, indistinguishable. On this task, once the encoder is trained, starting from Laya does not beat starting from the plain encoder; with the encoder frozen it does (58.6% against 47.6%, p < 0.0001).

Calibration: the third epoch loosened the probabilities (held-out NLL 0.258 after the second epoch, 0.525 after the third). Fit a temperature table with fuuga decide calibrate on rows apart from these test rows before putting the model behind a confidence gate.

How to use it

In Transformers the export is an encoder: AutoModel.from_pretrained(<this folder>) gives the hidden states of the fine-tuned encoder. Transformers has no counterpart for the decision head, and Laya's own code does not read Fuuga's head format, so neither classifies with this folder alone.

In Fuuga the encoder and the head make the decision model again:

fuuga transfer --mapping modernbert --donor <this folder> --tokenizer <this folder> --checkpoint laya-emotion
copy fuuga\* into laya-emotion, and tokenizer.json, tokenizer_config.json into laya-emotion\tokenizer
fuuga decide predict --decision-model laya-emotion ...     (or fuuga-serve --decision-model laya-emotion)

A decision model rebuilt this way from this folder alone gave the trained model's own prediction on all 2,000 test rows (examples/showcase/export-laya.ps1 does it and compares line for line).

Verification

• The import is exact: Laya imported with fuuga import-laya matches Laya's own outputs (token ids, option markers, logits within 4e-6), and Fuuga's ModernBERT forward is tested against an independent implementation of Transformers' forward, including the local window and padded batches.
• This folder, read independently of Fuuga's model code: verify-modernbert-export.fsx takes the layer kinds, RoPE bases, window, activation and norm biases from config.json, the weights by ModernBertModel's names, runs Transformers' ModernBERT forward written out in TorchSharp, and matches Fuuga's hidden states within 6.4e-5 on all three texts (tolerance 1e-3).
• Not yet checked: loading this folder in Transformers itself. Run python verify_modernbert_export.py (needs torch and transformers). It loads the folder with AutoModel, runs the token ids in fuuga_reference.json, and passes when every hidden state agrees with Fuuga's within 1e-3.

Limitations and terms

• A 6-class English emotion classifier trained on 4,000 examples, fine-tuned on this one question only: Laya's other decision tasks were not measured after training and are likely degraded.
• Publicly available classifiers trained on the full 16,000-row Emotion training split report higher accuracy; this model is a test case, not a recommendation.
• The weights derive from Laya (convaiinnovations/laya, Apache 2.0) and are released under the same licence, with the changes described above.
• The training data, DAIR Emotion, is for "educational and research purposes only" (its dataset card), so this model is released for research and evaluation. If you use it, cite the dataset: Saravia et al., "CARER: Contextualized Affect Representations for Emotion Recognition", EMNLP 2018.