Perfica-Dev/laya-multilingual-base-onnx

Laya multilingual (base), ONNX int8

The multilingual checkpoint of Laya by Convai Innovations (github.com/NandhaKishorM/laya), converted to ONNX and quantised to int8 so it runs on a CPU with ONNX Runtime.

This is the base checkpoint, converted. It is not fine-tuned: the weights are Laya's own, unchanged apart from the int8 quantisation described below. It is not an official Convai Innovations release; the conversion is by Perfica. The model, its architecture and its weights are Convai Innovations' work. Laya is licensed under Apache-2.0, and so is this conversion. See LICENSE and NOTICE.

Laya is a non-generative decision model: given a state (text) and a typed question (a choice, a score or a yes/no noul), it returns a probability for each option in one forward pass. Its multilingual checkpoint is mmBERT-base (22 layers, 256k vocabulary) with a two-layer decision head.

Source

The weights are convaiinnovations/laya, subfolder multilingual, at revision 55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851 (its model.safetensors is byte-identical to convaiinnovations/laya-multilingual's: SHA-256 9d628fd9…aa8f204). The architecture comes from that revision's rl_common.py.

Files

File	What it is
laya-multilingual.int8.onnx	The model (see below).
tokenizer.json, tokenizer_config.json	Laya's multilingual tokenizer, unchanged.
laya_config.json	maxLen 1024, headMaxLen 256, the temperatures and the source revision.

Inputs

Name	Type, shape	What
input_ids	int64 [batch, seq]	Token ids
attention_mask	int64 [batch, seq]	1 for tokens, 0 for padding
marker_pos	int64 [batch, options]	Position of each option's [MASK]
marker_mask	bool [batch, options]	Which option slots are real
qtype	int64 [batch]	0 choice, 1 score, 2 noul

Outputs

Name	Shape	What
logits	[batch, options]	One logit per option (padding slots at -1e4)
act_logits	[batch, acts]	Laya's act head

To get probabilities, divide logits by the temperature in laya_config.json and take a softmax over the options.

Inputs are laid out as in Laya's rl_common.build_sequence: [CLS] <type> question: <instructions> [SEP] [MASK] option… [SEP] <state> [SEP], and each option is scored at its [MASK].

The conversion

• ONNX opset 17, with dynamic batch, sequence and option axes, traced at 300 tokens so mmBERT's 128-token sliding-window branch is recorded.
• The decision head is rewritten for export. Laya's head is an nn.TransformerEncoder, which traces with the batch size baked into its reshapes. It is exported as the same arithmetic (pre-norm, ReLU, the same weights) with shape-agnostic reshapes. The fp32 export is checked against PyTorch at four shapes, up to batch 3 and 1,024 tokens with padding, to within 0.01 of a logit.
• int8. ONNX Runtime's dynamic int8 quantisation of the MatMul and Gather weights (the 256k × 768 embedding table is most of the model), except the 22 MatMuls named /encoder/layers.N/mlp/Wo/MatMul (the MLP down-projections), which stay fp32: their input after the gated activation is what a per-tensor dynamic int8 scale handles worst, and keeping them fp32 costs about 41 MB. 1.29 GB in fp32, about 364 MB here.

Reproduce

With torch, transformers>=5, onnx and onnxruntime (on Windows, in a venv at a short path):

python scripts/quickJudge/export_laya_onnx.py <workdir> --fp32-matmuls "mlp/Wo/"

The script (in Perfica's source) downloads the pinned revision above, exports, verifies, quantises and writes this folder to <workdir>/hf-repo.

Licence

Apache-2.0. Laya is © Convai Innovations; the ONNX conversion and this card are by Perfica (Uezar Labs). The mmBERT-base encoder Laya is built on is by JHU CLSP (jhu-clsp/mmBERT-base, MIT).