Reproduced your 78.33% on MASSIVE exactly, but struggling on my own data. What am I missing?
Hi! First, thanks for releasing Laya with open weights.
I reproduced the published MASSIVE (en) result: 78.33% accuracy and 0.786 macro-F1. It matched to 4 decimals on three setups: ROCm on an RX 9060 XT, CPU on Windows, and CPU on Linux. So I believe the install is right.
Then I tried it on my own text and couldn't get it to work:
- Paper sections. 5,489 English paragraphs from 73 physics papers, with section titles removed. I asked for intro/methods/results/discussion/conclusion using the English-only variant and an English question. It got 25.6% accuracy; the majority class is 42.4%. On a balanced 5-way subset it got 18.9%, against 20% chance. A keyword rule got 35.5%, and TF-IDF + logistic regression, split by paper, got 70.5%. With the question in Portuguese it was about the same (28.1% / 21.9%).
- Portuguese work notes, multilingual variant. It was at or below a keyword rule.
- Balanced yes/no question. It got 49.6% accuracy with 97.5% mean confidence.
- Calibration depends on the prompt. On the papers task, mean confidence was 31.5% vs 28.1% accuracy with the Portuguese prompt, but 42.3% vs 25.6% with the English one.
This looks like the same pattern as #7 and #9. One thing I haven't ruled out: each of my labels carried a one-line description, the criteria-style prompt the docs suggest, and #7 found that adding criteria flattens the scores. I haven't rerun with bare labels yet. My other guess is that long passages are further from the benchmark than the language is, but I haven't measured that either.
Would you expect bare labels to do better here? And is there a limit on the number of labels, or a text length or type, where you'd expect it to degrade?
I packaged the test as a small kit, in case it helps others check their own data. Step 1 reproduces your MASSIVE number, and step 2 compares Laya against majority, a keyword rule and TF-IDF on labeled data: https://github.com/JhouCode/laya-fit-check
Full write-up: https://ksmit.com.br/en/blog/mais-ia-pra-que
Probably nothing on your side. Zero-shot Laya is strong when the answer is stated in the words of the state and the label set is small, and it drops fast when labels are many or closely worded: jevbench measured 92% on sst2 but 38% on banking77's 77 intents. What fixed it for me was not prompt tuning but training a small head on my own rows on top of the frozen encoder: a 12-label intent task went from 89.5% to 100% agreement with its labels on 3000 rows, support triage from 69/34/66% to 99/86/98%. Training takes a couple of minutes on a laptop GPU because the encoder output is cached once.
I packaged that loop as stuntd (https://github.com/bladedevoff/stuntd): stuntd import <site> rows.jsonl --labels ..., stuntd train, then stuntd report shows holdout agreement, calibration and the coverage curve per question before you serve anything. If you share roughly what your data looks like (question type, number of options), I can tell you what to expect.
Thanks, I tried it on the paper-sections task, split by paper (5-fold GroupKFold). I froze the encoder, cached its output once and retrained Laya's own decision head, using the same defaults as stuntd's trainer: AdamW, lr 1e-4, 3 epochs, batch 16.
The retrained head got 40.8% on the full set (majority class: 42.4%) and 21.6% on the balanced subset (chance: 20%).
A plain logistic regression on the mean-pooled encoder output did much better: 68.5% and 39.7%, with macro-F1 0.71 against 0.46 for TF-IDF. TF-IDF + logistic regression still won on accuracy: 70.5% and 47.6%.
So the encoder does carry section signal, but the pretrained head didn't adapt with those defaults on about 4k long paragraphs. I haven't tuned beyond that.