hoja-solutions/laya-NVFP4A16

laya, NVFP4A16

This repo holds the three laya checkpoints with the linear layers of their encoder stored in NVFP4: FP4 (E2M1) values, one FP8 scale per 16 weights and one FP32 scale per tensor. Activations, embeddings, norms and the decision head are unchanged. The files are 349 MB instead of 843 MB for the English and typed-decisions checkpoints, and 485 MB instead of 644 MB for the multilingual one, whose 256k-token embedding table makes up most of its size.

laya answers typed questions about a piece of text (a choice between options, a score, or a yes/no probability) in one forward pass, with calibrated probabilities. The original card explains the model and its API. This card covers the quantization.

Checkpoint	Folder	Encoder	Original	This repo
English	repo root	ModernBERT-large, 28 layers	843 MB	349 MB
typed-decisions	typed-decisions/	ModernBERT-large, 28 layers	843 MB	349 MB
multilingual	multilingual/	mmBERT-base, 22 layers	644 MB	485 MB

Usage

laya.load can't read these files directly. laya_nvfp4.py, in this repo, unpacks the weights and returns a regular laya.Agent, so predict, predict_batch and the rest of laya's API work as usual. It needs only torch and laya; tested with laya 0.4.6 and torch 2.14.

import sys
from huggingface_hub import snapshot_download

path = snapshot_download("hoja-solutions/laya-NVFP4A16")
sys.path.insert(0, path)
import laya_nvfp4

agent = laya_nvfp4.load(path)                                   # English, at the repo root
agent_ml = laya_nvfp4.load(path + "/multilingual", device="cuda")
agent_td = laya_nvfp4.load(path + "/typed-decisions")

questions = {"department": {"type": "choice", "instructions": "Which department should handle this?",
                            "criteria": {"billing": "invoices, payments, refunds",
                                         "technical": "bugs, outages, system errors",
                                         "other": "everything else"}}}
print(agent.predict("I was charged twice for March.", questions)["answers"]["department"]["choice"])

To download one checkpoint only, pass allow_patterns=["laya_nvfp4.py", "multilingual/*"] to snapshot_download. For automatic routing by language, hand the loaded agents to laya's Router, which then uses them instead of downloading the originals:

from laya import Router

router = Router()
router.attach("english", agent)
router.attach("multilingual", agent_ml)
router.predict("La aplicación se cierra cada vez que abro la configuración.", questions)  # multilingual, technical

Quality

Each checkpoint was compared with the original in bf16 on an RTX 5090 (laya's default on recent GPUs). The 8,899 test questions were rebuilt from the laya authors' own benchmark scripts with their seeds and wording:

• the ten application tasks in research/scripts/bench_apps.py, 400 questions each (AG News, DAIR Emotion, Banking77, support-ticket routing, spam, phishing, jailbreak and toxicity detection, MS MARCO relevance, request routing);
• the typed-decisions test split, 2,000 decisions;
• MASSIVE intent with 20 options in 14 languages, and XNLI in 15 languages, 100 questions per language.

Accuracy uses the temperatures shipped with each checkpoint, so it's what predict reports.

Accuracy, original → this repo	10 app tasks	typed-decisions	MASSIVE	XNLI
English	0.696 → 0.699	0.359 → 0.364	0.356 → 0.337	0.530 → 0.528
typed-decisions	0.704 → 0.700	0.767 → 0.774	0.348 → 0.344	0.541 → 0.533
multilingual	0.694 → 0.690	0.353 → 0.354	0.506 → 0.486	0.718 → 0.711

Accuracy stays within a point on most suites. Individual answers change more than that suggests, because the changes go both ways. This table counts how often the top answer is the same as the original's:

Same answer as the original	10 app tasks	typed-decisions	MASSIVE	XNLI	all questions	when the original is ≥ 90% sure
English	95.5%	94.2%	70.9%	82.0%	89.1%	96.3%
typed-decisions	95.4%	97.0%	71.9%	85.7%	90.4%	99.1%
multilingual	94.7%	88.1%	82.4%	90.7%	90.7%	98.2%
original, run twice	99.6 to 99.9%	99.4 to 99.5%	96.5 to 99.2%	99.2 to 99.7%	99.0 to 99.6%	100%

The last row is the floor: running the original again with the questions batched differently changes 0.4 to 1% of its answers through floating-point noise alone. Most of the answers that change are close calls between options, which is why the 20-option MASSIVE questions move the most. Answers the original gives with at least 90% probability (40% of all questions for English, 15% for typed-decisions, 56% for multilingual) change far less, and least on the typed-decisions and multilingual checkpoints. The English checkpoint is the most sensitive. If your application depends on getting exactly the original's answers, check this build on your own data.

Calibration barely moves. ECE on the ten app tasks goes from 0.160 to 0.158 (English), 0.123 to 0.117 (typed-decisions) and 0.174 to 0.179 (multilingual). On its own benchmark the typed-decisions checkpoint scores accuracy 0.767 → 0.774, ECE 0.215 → 0.225 and score MAE 0.243 → 0.243, against 0.766, 0.213 and 0.242 on the original card.

Speed

laya_nvfp4.load unpacks the weights once, so the model runs exactly like the original: same speed, same GPU memory (laya keeps weights in fp32 and computes in bf16, 1.6 GB for the English checkpoint). The gain is the download and the disk. On an RTX 5090, with the original card's latency setup (one support ticket, p50 over 20 calls):

	1 question	10 questions	50 questions	4,000 questions, batched
English, original	10.9 ms	18.0 ms	74.9 ms	775 per second
English, this repo	11.0 ms	18.2 ms	75.4 ms	778 per second
multilingual, original	8.8 ms	10.4 ms	34.3 ms	1,577 per second
multilingual, this repo	8.9 ms	10.4 ms	34.7 ms	1,521 per second

How it was made

• llm-compressor 0.14.0 with compressed-tensors 0.19.0, transformers 5.17.0, torch 2.14.0 and laya 0.4.6.
• GPTQ (static activation ordering, block size 128, dampening 0.01) with the NVFP4A16 scheme on the four linear layers of every encoder block (attn.Wqkv, attn.Wo, mlp.Wi, mlp.Wo): 112 layers in the ModernBERT checkpoints, 88 in the multilingual one. The decision head passes raw weights to PyTorch's attention function, so it stays in fp16 with the embeddings and norms. laya's model isn't a transformers model, so calibration ran whole-model forward passes through llm-compressor's basic pipeline.
• Calibration used 1,343 questions (337,098 tokens) for the English and typed-decisions checkpoints and 1,474 questions (277,456 tokens) for the multilingual one: typed-decisions train cases, the app tasks' question wording on train rows or rows the tests don't use, MASSIVE train utterances and XNLI validation pairs. No calibration text appears in the test set.

Other variants were measured on the English checkpoint:

Variant	Size	Same answer, all questions	≥ 90% sure
this repo (GPTQ, every encoder layer)	349 MB	89.1%	96.3%
plain rounding instead of GPTQ	349 MB	85.4%	94.0%
first four encoder layers kept in fp16	420 MB	90.9%	97.7%
attention kept in fp16, MLP quantized	518 MB	91.2%	97.5%
MLP kept in fp16, attention quantized	674 MB	92.7%	98.5%
FP8 weights and activations, for reference		94.0%	99.4%

Keeping parts of the encoder in fp16 buys a little agreement at a large cost in size, because the changes come from every layer. Even 8-bit FP8 changes about 3% of the app-task answers, almost all of them close calls.

Limits

• Tested on an RTX 5090 with torch 2.14 and laya 0.4.6. Loading on CPU was checked on a few examples.
• laya.load and Router don't read these files by themselves; use laya_nvfp4.load and Router.attach. The loader hands laya.Agent the unpacked weights by temporarily replacing safetensors.torch.load_file while the agent is built, so a future laya that reads weights differently may need an updated loader.
• Everything on the original card's "Honest Limits" still applies, including over-confident probabilities before temperature refitting and noul answers that can follow their option labels.

Credits

laya and its three checkpoints are by Convai Innovations (GitHub), released under Apache-2.0. They build on ModernBERT-large and mmBERT-base. The test and calibration questions come from the laya benchmark scripts and the public datasets they use: AG News, DAIR Emotion, Banking77, customer-support-tickets, Enron spam, phishing emails, ToxicChat, MS MARCO, GSM8K, MBPP, typed-decisions, MASSIVE and XNLI.