lapis-sal-150M-MLX

The Apple Silicon deployment of the bilingual Lapis SAL student, with one FP16 checkpoint and optional INT8 quantization at load.

Give the router a message and a menu of tools. It returns the best matching tool ID, or abstains with None.

Quick start · Model family · Tools · Training · Evaluation · Limitations

Model family

Release	Format	Download	Best for
lapis-sal-150M	PyTorch FP32	~658 MB	Fine-tuning, research and conversion
lapis-sal-421M	PyTorch FP32	~1.69 GB	Teacher inference and soft-label generation
lapis-sal-150M-MLX	MLX FP16 / load-time INT8	~332 MB	Apple Silicon deployment

Download sizes include the tokenizer and runtime. Each repository contains one checkpoint or backend; you do not need the full family to run one model. The family links point to the released checkpoint and MLX repositories. The ONNX export is retained locally for evaluation and is not part of this Hub release.

Quick start

Use Python 3.12. Download this repository and run from the downloaded folder:

python -m pip install 'huggingface_hub==1.33.0'
hf download lapislabsai/lapis-sal-150M-MLX --local-dir lapis-sal-150M-MLX
cd lapis-sal-150M-MLX
python -m pip install -r requirements.txt

Run the following Python example from the downloaded folder. Replace the menu with the tools in your application:

from laya_intent_router import LayaIntentRouter

router = LayaIntentRouter('.', backend='mlx', shortlist_k=0)
paths = {
    'WEATHER': ['Check weather conditions, forecasts and weather alerts'],
    'CALENDAR': ['Create, read or update calendar events and meetings'],
}
result = router.route('Will it rain in Madrid tomorrow?', paths)
print(result['match'], result['score'])

match contains your tool ID, or None for abstention. score is the selected route's calibrated score; probabilities contains the option scores, including the internal __none__ option. Evaluate calibration for your own menu and traffic.

Requires macOS on Apple Silicon with a Metal GPU. FP16 is the default. To use the same weights with 8-bit quantization at load time:

from laya_intent_router import LayaIntentRouter
router = LayaIntentRouter('int8', backend='mlx', shortlist_k=0)

int8/mlx/config.json points to the single FP16 checkpoint, so there is no second weight file in this release. The configuration quantizes eligible Linear layers at load (8 bits, group size 64). Stored weights and download size remain FP16; embeddings and noneligible layers remain unquantized. This custom architecture uses the included runtime, not mlx_lm.load().

Why Lapis SAL?

A bilingual assistant needs to decide which tool to use before it can act. Lapis SAL makes that decision a dedicated, local step: pass the user's message and the tools currently available, then receive a tool ID or an abstention.

The project pairs a larger teacher with a smaller distilled student so applications can choose between a research checkpoint and compact deployment exports. The aim is practical English/Spanish routing with explicit tool descriptions and a way to say that none of the tools fits.

What can you build?

A personal assistant that routes requests to weather, calendars, reminders, email or search.
An agent or conversational workflow that chooses among the tools available for a particular request.
A local routing layer for desktop assistants, with CPU inference through ONNX or Apple Silicon inference through MLX.
Teacher-based soft-label generation, student distillation or further task-specific fine-tuning with the PyTorch checkpoints.

The router selects a tool; your application supplies tool arguments, permissions and execution.

Tools and abstention

The adaptation and evaluation cover 13 tools plus NONE:

Tool	Intended route
`CALCULATOR`	Arithmetic and numerical calculations
`CALENDAR`	Calendar events, appointments and meetings
`CLOCK`	Time, timers, alarms and stopwatches
`CODE`	Programming questions and code assistance
`CONTACTS`	Contact lookup and contact management
`HEALTH`	Health, fitness and wellness information
`LOCATION`	Current location and location information
`MAIL`	Email search, reading and composition
`MAPS`	Maps, directions, places and navigation
`MESSAGES`	Instant messaging and text messages
`REMINDERS`	Reminders and personal tasks
`WEATHER`	Weather conditions, forecasts and alerts
`WEB_SEARCH`	Web information search
`NONE`	No appropriate tool or out-of-scope request

These are the evaluated tool families, rather than a fixed output vocabulary. You provide IDs, descriptions and optional example phrasings at runtime. The router also considers its internal out-of-scope option; new tools and descriptions need independent evaluation.

Architecture and size

Property	Value
Encoder	Ettin-encoder-150M (ModernBERT-style)
Encoder layers / hidden size / attention heads	22 / 768 / 12
Intermediate size / vocabulary	1,152 / 50,368
Encoder parameters	149,014,272
Complete learned parameters	163,983,619
Temperature-buffer elements	3
Decision head	Two norm-first transformer layers, typed embeddings and per-option scorer
Input budget	512 tokens total; 384-token head budget
Release format	MLX FP16 / load-time INT8

The saved action head is retained for checkpoint compatibility; production tool routing reads the option logits. The 150M name identifies the 149M-parameter encoder family; the decision head brings the complete model to approximately 164M parameters.

Training

The Lapis adaptation workflow was configured in an environment with 2 × NVIDIA T4 GPUs, using FP16 mixed precision. The retained training implementation runs on a single device (world_size: 1); the release does not claim that both GPUs participated in distributed training.

Adapt the teacher: fine-tune the Laya / ModernBERT-large decision model on bilingual tool-routing episodes.
Create soft targets: run the teacher over training episodes and retain probabilities for the candidate tools and NONE.
Distill the student: train an Ettin-150M encoder with the Laya decision head, mixing teacher probabilities and gold labels with a teacher-target weight of 0.5.
Calibrate and export: fit temperature buckets on held-out training/development data, then export and check the student backends on development episodes.

Setting	Teacher	Student
Epochs	1	2
Micro batch / gradient accumulation	2 / 16	8 / 6
Encoder / head learning rate	2.5e-5 / 1e-4	5e-5 / 3e-4
Batch token budget	4,096	4,096
Prompt / total token limit / head budget	`compact_fill` / 512 / 384	Same
Teacher-target weight	Gold labels	0.5
Training seed	20260926	20260926

Dataset

The privately supplied Lapis bilingual tool-routing v4.1-refined source has 10,000 synthetic examples: 8,000 train, 1,000 validation and 1,000 final test. Deterministic preparation produces 100,000 training episodes and 5,000 development episodes, with seed 7, shuffled variable tool menus and nominal 5% gold-tool removal to exercise abstention. Only train/validation rows are expanded; the reserved test stays separate.

The source dataset and teacher soft labels are not redistributed. The author's private reproducibility repository retains preparation/training/export scripts, a dependency lock and source data. Original complete training logs and historical initialization revisions were not retained; exact reproduction of the original optimizer trajectory is not claimed. The released weights can be used for a new fine-tuning run, but original optimizer/scheduler state is not included.

Evaluation

The reserved final test contains 1,000 examples: 500 English and 500 Spanish. All 13 tools plus the internal NONE option are available, with shortlist disabled and threshold 0.625. Test examples and tool descriptions were preserved. Development episodes were used for numerical checks and latency measurements.

Backend	Correct / total	Accuracy	English	Spanish
MLX FP16	951 / 1000	95.10%	97.80%	92.40%
MLX load-time INT8	951 / 1000	95.10%	97.80%	92.40%

Macro F1 for MLX FP16: 95.54%. Class recall is the fraction of examples of that class routed correctly.

Per-class precision, recall and F1

Class	Test examples	Precision	Recall	F1
CALCULATOR	60	100.00%	100.00%	100.00%
CALENDAR	60	98.18%	90.00%	93.91%
CLOCK	60	100.00%	95.00%	97.44%
CODE	60	98.21%	91.67%	94.83%
CONTACTS	60	89.55%	100.00%	94.49%
HEALTH	60	90.16%	91.67%	90.91%
LOCATION	60	100.00%	100.00%	100.00%
MAIL	60	98.28%	95.00%	96.61%
MAPS	60	100.00%	100.00%	100.00%
MESSAGES	60	94.34%	83.33%	88.50%
NONE	220	89.12%	96.82%	92.81%
REMINDERS	60	93.33%	93.33%	93.33%
WEATHER	60	100.00%	90.00%	94.74%
WEB_SEARCH	60	100.00%	100.00%	100.00%

MLX load-time INT8: per-class precision, recall and F1

Class	Test examples	Precision	Recall	F1
CALCULATOR	60	100.00%	100.00%	100.00%
CALENDAR	60	98.18%	90.00%	93.91%
CLOCK	60	100.00%	95.00%	97.44%
CODE	60	98.21%	91.67%	94.83%
CONTACTS	60	89.55%	100.00%	94.49%
HEALTH	60	90.16%	91.67%	90.91%
LOCATION	60	100.00%	100.00%	100.00%
MAIL	60	98.28%	95.00%	96.61%
MAPS	60	100.00%	100.00%	100.00%
MESSAGES	60	94.34%	83.33%	88.50%
NONE	220	89.12%	96.82%	92.81%
REMINDERS	60	93.33%	93.33%	93.33%
WEATHER	60	100.00%	90.00%	94.74%
WEB_SEARCH	60	100.00%	100.00%	100.00%

On 100 development episodes, this backend agreed with the pinned PyTorch reference on 100% of route decisions, with maximum absolute probability difference 0.000545. This sampled check is not a proof for all possible inputs.

MLX load-time INT8 also reached 100% development route agreement; maximum probability difference was 0.001187.

The historical 63.80% reference used different tool descriptions and is not a controlled same-prompt baseline. No causal improvement claim is based on that comparison. Detailed audit records and file checksums are retained in the private reproducibility repository. No raw evaluation messages are shipped.

Measured routing latency

Apple M5, macOS arm64; 150 seeded development episodes, 2–13 tools plus NONE, 10 warm-up calls, full synchronous route(), shortlist disabled. Load time is excluded.

Backend	p50	p95
MLX FP16, GPU	39.91 ms	45.86 ms
MLX load-time INT8, GPU	40.09 ms	45.80 ms

These are measurements on this hardware and sample, not performance guarantees. The machine-readable timing reports are retained in the private reproducibility repository.

Limitations

Evaluation uses a small synthetic bilingual test. Its scores are not estimates for arbitrary real-world traffic; validate with your own held-out examples.
Only English and Spanish were evaluated. Health or code route selection does not validate the correctness of health advice or generated code.
Overlapping descriptions, implicit intent, long conversations and ambiguous messages can produce confident routing errors. Menus/history share a 512-token budget.
Thresholds and scores depend on the menu. Keep threshold tuning and model selection separate from your final test.
The model scores options; it does not generate text, extract tool arguments, execute actions or verify user authorization.
Apple MLX measurements cover macOS Apple Silicon. iPhone/iOS, other languages and unreported accelerators have not been validated.
This custom DecisionModel requires the included runtime. Standard Transformers pipelines and hosted inference providers are not advertised as supported.

License and acknowledgements

The release is distributed under Apache-2.0, retaining applicable upstream notices. Ettin's MIT notice is included under third_party/. The model license does not grant rights to the private source dataset.

Built on Laya DecisionModel, the Laya intent-router implementation, ModernBERT-large, and Ettin-encoder-150M. See LICENSE and NOTICE.md for attribution.

Keywords

semantic routing, tool routing, intent classification, dynamic tool selection, abstention, out-of-scope detection, bilingual English Spanish, knowledge distillation, local inference, ModernBERT, Ettin, Laya