solintellegence/Sol-Lassi

Sol Lassi 600K

Lassi is an M-SimOW optimizer experiment: 600,000 parameters, trained from scratch with MLX on Apple Silicon. The run reached 1 billion token exposures using a balanced FinePhrase stream.

There are no Open SLM, ArithMark, or Intelligence Index results for this checkpoint. The available loss logs come from individual training minibatches; they don't measure held-out performance.

Model and training settings

Setting	Value
Parameters	600,000
Blocks	6 independent transformer blocks
Hidden width	96
Attention	3 heads, head dimension 32
FFN	Gated SiLU, width 104
Tokenizer	2,048-entry byte-level BPE
Token embeddings	Tied to the output head
Training context	128 tokens
Training exposures	1,000,000,000
Weights	MLX NPZ

The data uses FinePhrase's faq, math, table, and tutorial configurations. FinePhrase-balanced-500m-2k-v2 records 500M stream tokens, while the run accumulated 1B exposures.

We trained with 128-token sequences, batch size 32, and seed 7. M-SimOW used momentum beta 0.8 and decoupled weight decay 0.1. The learning rate was 0.006 for the first 500M exposures and 0.003 for the remaining 500M.

Load and generate

Download the repository, install requirements.txt, and add the downloaded directory to Python's import path:

pip install -r requirements.txt

import sys
from huggingface_hub import snapshot_download

model_dir = snapshot_download("solintellegence/sol-lassi")
sys.path.insert(0, model_dir)
from modeling_sol_lassi import load_model, generate

model, tokenizer = load_model(model_dir)
print(generate(model, tokenizer, "A tiny language model can", max_new_tokens=48))

Lassi is a base model for optimizer tests, with no instruction tuning. Generated text may repeat, stop making sense, or state incorrect facts.

Files and license

model.npz contains the weights. The loader and generator in modeling_sol_lassi.py depend on keystone_mlx/. The tokenizer and architecture configuration are included; training_state.json records the run.

CC BY 4.0 covers the model. FinePhrase and its source datasets retain their own terms.