solintellegence
/

Sol-Lassi

@@ -14,79 +14,59 @@ tags:
 ![Sol Lassi](./sol-lassi-banner.png)
-# Sol Lassi 600K Base
-Sol Lassi is an experimental **600,000-parameter** decoder-only language model from **Sol Intelligence**. It was trained from scratch for **1 billion token exposures** with M-SimOW.
-This is a small **base language model**, not an instruction-tuned assistant. No Intelligence Index or leaderboard score is claimed for this checkpoint.
-## Model summary
-| Property | Value |
 |---|---|
-| Parameters | **600,000** |
-| Architecture | Dense decoder-only causal language model |
-| Transformer blocks | 6 independent blocks |
 | Hidden width | 96 |
-| Attention | 3 heads, 32 dimensions per head |
-| MLP | Gated SiLU, width 104 |
-| Vocabulary | 2,048-token byte-level BPE |
-| Input/output embeddings | Tied |
 | Training context | 128 tokens |
-| Weight format | MLX NPZ |
-| Training tokens | **1,000,000,000 exposures** |
-## Training
-Sol Lassi was trained on a balanced FinePhrase stream composed of its `faq`, `math`, `table`, and `tutorial` configurations. Training used 128-token sequences, batch size 32, and seed 7.
-The optimizer was M-SimOW (momentum beta 0.8) with decoupled weight decay of 0.1. The fixed learning-rate schedule was:
-| Training span | Learning rate |
-|---:|---:|
-| First 500M tokens | 0.006 |
-| Next 500M tokens | 0.003 |
-The saved run used the FinePhrase-balanced-500m-2k-v2 manifest. It contains 500M training-stream tokens; the model received 1B token exposures. The tokenizer and training metadata are included in this repository.
-## Quick start
-Sol Lassi uses MLX and is intended to run on Apple Silicon.
 ```bash
 pip install -r requirements.txt
 ```
 ```python
 from huggingface_hub import snapshot_download
-from modeling_sol_lassi import load_model, generate
 model_dir = snapshot_download("solintellegence/sol-lassi")
 model, tokenizer = load_model(model_dir)
 print(generate(model, tokenizer, "A tiny language model can", max_new_tokens=48))
 ```
-## Evaluation status
-This release does not include Open-SLM, ArithMark, or Intelligence Index evaluations. Its training loss logs are per-minibatch measurements and are not presented as held-out performance.
-## Intended use and limitations
-Sol Lassi is for research on very small language models and optimizer experiments. At 600K parameters, it has substantial capability limits; its output may be incoherent, repetitive, factually incorrect, or unsafe. It is not suitable for high-stakes use.
-## Repository files
-| File | Purpose |
-|---|---|
-| `model.npz` | Trained MLX weights |
-| `modeling_sol_lassi.py` | Standalone loader and generation helper |
-| `keystone_mlx/` | Architecture implementation required by the loader |
-| `tokenizer.json` | Frozen 2,048-token tokenizer |
-| `tokenizer_config.json` | Tokenizer metadata |
-| `config.json` | Architecture metadata |
-| `training_state.json` | Training and provenance summary |
-| `sol-lassi-banner.png` | Sol Lassi model artwork |
-## License
-Released under **CC BY 4.0**. Dataset terms remain with the upstream dataset providers.

 ![Sol Lassi](./sol-lassi-banner.png)
+# Sol Lassi 600K
+Sol Lassi is a 600,000-parameter language model trained from scratch on 1 billion token exposures. It uses M-SimOW and runs with MLX on Apple Silicon.
+## Model
+| Setting | Value |
 |---|---|
+| Parameters | 600,000 |
+| Blocks | 6 independent transformer blocks |
 | Hidden width | 96 |
+| Attention | 3 heads, head dimension 32 |
+| FFN | Gated SiLU, width 104 |
+| Tokenizer | 2,048-entry byte-level BPE |
+| Token embeddings | Tied to the output head |
 | Training context | 128 tokens |
+| Training exposures | 1,000,000,000 |
+| Weights | MLX NPZ |
+## Use
+Download the repository, install its dependencies, and add its directory to the Python path before importing the loader:
 ```bash
 pip install -r requirements.txt
 ```
 ```python
+import sys
 from huggingface_hub import snapshot_download
 model_dir = snapshot_download("solintellegence/sol-lassi")
+sys.path.insert(0, model_dir)
+from modeling_sol_lassi import load_model, generate
 model, tokenizer = load_model(model_dir)
 print(generate(model, tokenizer, "A tiny language model can", max_new_tokens=48))
 ```
+## Training
+The model used a balanced FinePhrase stream drawn from the `faq`, `math`, `table`, and `tutorial` configurations. The FinePhrase-balanced-500m-2k-v2 manifest contains 500M stream tokens, which produced 1B training exposures.
+Training used 128-token sequences, a batch size of 32, and seed 7. M-SimOW used momentum beta 0.8 and decoupled weight decay 0.1. The learning rate was 0.006 for the first 500M exposures, then 0.003 for the remaining 500M.
+## Evaluation and limits
+No Open SLM, ArithMark, or Intelligence Index evaluation is included. The training-loss logs measure individual minibatches and are not held-out scores.
+Lassi is a base-model and optimizer experiment. At this size, completions can be incoherent, repetitive, or incorrect. It has not been instruction-tuned.
+## Files and license
+`model.npz` contains the weights. `modeling_sol_lassi.py` provides the loader and generator, which depend on `keystone_mlx/`. The repository also includes the tokenizer, architecture configuration, and `training_state.json`.
+The model is licensed under [CC BY 4.0](LICENSE). FinePhrase and its sources retain their own terms.