hoja-solutions/laya-NVFP4

laya, NVFP4 (W4A4)

This repo holds the three laya checkpoints with both the weights and the input activations of their encoder's linear layers in NVFP4: FP4 (E2M1) values with one FP8 scale per group of 16 and one FP32 scale per tensor. On Blackwell GPUs those layers multiply on FP4 tensor cores. The files are the same size as the NVFP4A16 build, 349 MB for the English and typed-decisions checkpoints and 485 MB for the multilingual one, and the model takes about a third of the GPU memory of the original.

The NVFP4A16 build, which keeps activations in bf16, runs at the original's speed, stays closer to the original's answers and gives the same answer for the same question every time. Use this W4A4 build only when GPU memory is what limits you. The original card explains the model itself.

Usage

import sys
from huggingface_hub import snapshot_download

path = snapshot_download("hoja-solutions/laya-NVFP4")
sys.path.insert(0, path)
import laya_nvfp4

agent = laya_nvfp4.load(path, device="cuda")                    # English, at the repo root
agent_ml = laya_nvfp4.load(path + "/multilingual", device="cuda")
agent_td = laya_nvfp4.load(path + "/typed-decisions", device="cuda")

laya_nvfp4.load returns a regular laya.Agent, so predict and the rest of laya's API work, and Router.attach adds it to laya's router. Each quantized layer quantizes its input on every call. On Blackwell GPUs (sm_100, sm_120) the multiplication runs on FP4 tensor cores through torch._scaled_mm, and torch.compile fuses the input quantization. On any other device the loader emulates the same arithmetic in fp32, which is correct but slow. Tested with laya 0.4.6 and torch 2.14.

Quality

The tests are the same as for the NVFP4A16 build: 8,899 questions rebuilt from the laya authors' benchmark scripts (ten application tasks, the typed-decisions test split, MASSIVE intent in 14 languages and XNLI in 15), each checkpoint compared with the original in bf16 on an RTX 5090, accuracy with the shipped temperatures.

Accuracy, original → this repo	10 app tasks	typed-decisions	MASSIVE	XNLI
English	0.696 → 0.695	0.359 → 0.357	0.356 → 0.320	0.530 → 0.526
typed-decisions	0.704 → 0.701	0.767 → 0.766	0.348 → 0.319	0.541 → 0.525
multilingual	0.694 → 0.686	0.353 → 0.346	0.506 → 0.459	0.718 → 0.701

Same answer as the original	10 app tasks	typed-decisions	MASSIVE	XNLI	all questions	when the original is ≥ 90% sure
English	91.9%	88.8%	61.7%	76.3%	83.8%	92.5%
typed-decisions	92.5%	93.7%	62.8%	81.3%	86.2%	97.6%
multilingual	90.6%	79.5%	72.4%	87.0%	84.6%	95.4%
NVFP4A16, for comparison	94.7 to 95.5%	88.1 to 97.0%	70.9 to 82.4%	82.0 to 90.7%	89.1 to 90.7%	96.3 to 99.1%
original, run twice	99.6 to 99.9%	99.4 to 99.5%	96.5 to 99.2%	99.2 to 99.7%	99.0 to 99.6%	100%

Accuracy holds better than agreement because the changed answers go both ways, but this build changes 14 to 16% of the answers, including some the original gives with 90% probability or more.

The same question can get different answers

Every quantized layer rounds its input to FP4 again. A tiny difference upstream, such as a different floating-point summation order, moves some values across an FP4 rounding boundary, and a full FP4 step is a large error, so the difference grows from layer to layer instead of fading. In practice:

• Batching changes answers. The English checkpoint answered all 8,899 questions twice, with the questions grouped into different batches. 6.1% of its answers differed between the two runs. The original and the NVFP4A16 build change about 1% this way.
• Hardware changes answers. The emulation path, which repeats the same arithmetic in fp32, agrees with the original about as often as the GPU path (84.0% against 83.8% of all questions), but on different questions: the two runs of this build agree with each other on only 84%.

If you need repeatable decisions, use the NVFP4A16 build.

Speed

RTX 5090, the original card's latency setup (one support ticket, p50 over 20 calls):

	1 question	10 questions	50 questions	4,000 questions, batched	GPU memory for weights
English, original	10.9 ms	18.0 ms	74.9 ms	775 per second	1,607 MB
English, this repo	23.8 ms	26.6 ms	72.1 ms	852 per second	489 MB
multilingual, original	8.8 ms	10.4 ms	34.3 ms	1,577 per second	1,247 MB
multilingual, this repo	20.4 ms	21.6 ms	32.7 ms	1,507 per second	868 MB

An FP4 matrix multiply is about five times faster than bf16 at batch sizes of a few thousand tokens. Matrix multiplies are a small part of laya's run time, though, and every layer pays for quantizing its input. So small requests take about twice as long as with the original, and large batches gain about 10% at best. The memory saving is real: the English checkpoint needs a third of the original's GPU memory, because laya otherwise keeps its weights in fp32.

How it was made

GPTQ with llm-compressor 0.14.0 (static activation ordering, block size 128, dampening 0.01) and the NVFP4 scheme on the four linear layers of every encoder block. The scales for each group of 16 input values are computed at run time, under a per-tensor scale fixed from the calibration data. Embeddings, norms and the decision head stay in fp16. The calibration set and the NVFP4A16 variants are described on the NVFP4A16 card.

Plain rounding instead of GPTQ keeps the same answer on 81.3% (English), 85.0% (typed-decisions) and 82.5% (multilingual) of the questions, so GPTQ helps by 1 to 3 points.

Limits

• The FP4 tensor-core path needs a Blackwell GPU. Elsewhere the emulation gives the same quality at a fraction of the speed.
• Tested on an RTX 5090 with torch 2.14 and laya 0.4.6.
• laya.load and Router don't read these files by themselves; use laya_nvfp4.load and Router.attach.
• Everything on the original card's "Honest Limits" still applies.

Credits

laya and its three checkpoints are by Convai Innovations (GitHub), released under Apache-2.0. They build on ModernBERT-large and mmBERT-base.