solintellegence
/

Sol-Milkshake

@@ -17,41 +17,46 @@ tags:
 # Sol Milkshake
-Sol Milkshake is an exact **2,990,000-parameter** recurrent, hyperspherical decoder-only language model trained from scratch and continued through **2,526,565,888 total token exposures**.
-It combines recurrent block reuse, grouped-query attention, parameter-free XSA value subtraction, value residuals, tensorized 2–5-gram memory, completed-chunk rolling memory, adaptive recurrent routing, and nGPT-style hyperspherical optimization.
-**This is a base model, not an instruction-tuned assistant.**
-## Model summary
-| Property | Value |
-| --- | --- |
-| Parameters | **2,990,000** |
-| Total token exposures | **2,526,565,888** |
-| Architecture | Decoder-only recurrent causal language model |
-| Physical / effective blocks | 5 / 11 |
-| Residual width | 192 |
-| Vocabulary | 2,048-token byte-level BPE |
-| Maximum context | 2,048 tokens |
-| Weight format | MLX NPZ |
-| License | CC BY 4.0 |
-## Quick start
-Install the runtime dependencies:
 ```bash
 pip install "mlx>=0.29" "tokenizers>=0.22" huggingface_hub
 ```
-Load the model and generate text on Apple Silicon:
 ```python
 from huggingface_hub import snapshot_download
 import sys
-model_dir = snapshot_download("solintellegence/Sol-Milkshake-3M-Base")
 sys.path.insert(0, model_dir)
 from modeling_sol_milkshake import load_model, generate
@@ -68,114 +73,32 @@ text = generate(
 print(text)
 ```
-The repository includes the complete standalone MLX implementation and does not require the original training repository.
-## Results
-| Benchmark | Examples | Score |
-| --- | ---: | ---: |
 | HellaSwag | 10,042 | 25.02% |
 | ARC-Easy | 2,376 | 29.76% |
 | ARC-Challenge | 1,172 | 22.70% |
 | PIQA | 1,838 | 52.50% |
 | ArithMark-3 | 1,000 | 30.60% |
-| Intelligence Index | — | **3.158** |
-The four language-model tasks were evaluated zero-shot on their complete evaluation splits with `lm-eval` 0.4.12 using normalized accuracy.
-ArithMark-3 used its independent-tokenization normalized-accuracy protocol.
-Raw evaluation outputs are included under `evals/`.
-These measurements are not independently verified, and no leaderboard position is claimed. The release checkpoint was selected from recovery checkpoints using this evaluation suite, so the results should not be treated as a pristine held-out estimate.
-## Architecture
-| Component | Configuration |
-| --- | --- |
-| Stored parameters | **2,990,000** |
-| Physical / effective blocks | 5 / 11 |
-| Recurrent layout | 1 prelude + 3 middle blocks × 3 passes + 1 coda |
-| Residual width | 192 |
-| Attention | 6 query heads, 2 key/value heads, head dimension 32 |
-| Attention modification | XSA value subtraction after causal attention |
-| Attention normalization | Unit-normalized Q/K with RoPE |
-| Recurrent routing | Full first pass, then 75% and 50% token capacity |
-| MLP | Gated SiLU MLP, width 512 |
-| Vocabulary | 2,048-token byte-level BPE |
-| Learned memory | Rank-26 tensorized 2–5-gram memory |
-| Rolling memory | 32-token chunks, 32 slots, width 64 |
-| Embeddings | Tied input/output table |
-| Maximum context | 2,048 tokens |
-## Training data
-Initial pretraining used a staged English curriculum built from:
-- FineWeb-Edu
-- FinePDFs-Edu
-- English UltraFineWeb multi-domain and question-answer subsets
-- Cosmopedia
-- FineMath
-Recovery training used a **70% / 15% / 15%** mixture of the original frozen curriculum, Cosmopedia-v2 English, and FinePhrase.
-The tokenizer and prepared training streams were frozen before training. Existing quality filtering, deduplication, and decontamination rules were retained.
-## Intended use
-Sol Milkshake is intended for research on highly parameter-efficient language modeling, including:
-- extremely small language models
-- recurrent-depth architectures
-- nGPT-style hyperspherical optimization
-- compact learned and rolling memory
-- grouped-query attention at small scale
-- Apple Silicon and MLX inference research
-Because this is a base model, users should expect raw next-token completion behavior rather than instruction-following or assistant-style responses.
-## Limitations
-At approximately 3M parameters, Sol Milkshake has severe capability limitations.
-Outputs may be:
-- inconsistent
-- factually incorrect
-- repetitive
-- incoherent
-- unsafe
-The model is not instruction tuned and should not be treated as a conversational assistant.
-Do not use it for high-stakes decisions.
-Users are responsible for evaluating outputs and determining whether the model is suitable for their application.
-## Repository files
-| File | Purpose |
-| --- | --- |
-| `model.npz` | MLX model weights |
-| `modeling_sol_milkshake.py` | Standalone model, loader, and generator |
-| `sol_config.py` | Frozen architecture configuration |
-| `config.json` | Model metadata |
-| `tokenizer.json` | Frozen tokenizer |
-| `tokenizer_config.json` | Tokenizer metadata |
-| `training_state.json` | Training and provenance summary |
-| `evals/` | Raw benchmark results |
-## Reproducibility and evaluation notes
-The repository includes raw evaluation outputs under `evals/`.
-Reported benchmark values should be interpreted as reference measurements for this release. They have not been independently verified, and checkpoint selection used the same suite.
-No leaderboard position is claimed.
-## License
-Sol Milkshake Base is released under the **CC BY 4.0** license.
-Dataset licenses and terms remain with their respective owners.

 # Sol Milkshake
+Sol Milkshake is a 2,990,000-parameter recurrent language model for MLX on Apple Silicon. Its checkpoint received 2,526,565,888 token exposures across pretraining and recovery training.
+It uses hyperspherical optimization, shared transformer blocks, and local and rolling memory. This is a base model for text completion.
+## Model
+| Setting | Value |
+|---|---|
+| Parameters | 2,990,000 |
+| Token exposures | 2,526,565,888 |
+| Context | 2,048 tokens |
+| Tokenizer | 2,048-entry byte-level BPE |
+| Hidden width | 192 |
+| Stored blocks / effective applications | 5 / 11 |
+| Recurrent layout | 1 prelude, 3 middle blocks used three times, 1 coda |
+| Attention | 6 query heads, 2 KV heads, head dimension 32 |
+| Q/K normalization | Unit normalization with RoPE |
+| Attention features | XSA value subtraction and value residuals |
+| Routing | Full first pass, then 75% and 50% token capacity |
+| FFN | Gated SiLU, width 512 |
+| Local memory | Rank-26 tensorized 2-5-gram memory |
+| Rolling memory | 32-token chunks, 32 slots, width 64 |
+| Token embeddings | Tied to the output head |
+| Weights | MLX NPZ |
+## Use
+Install MLX, the tokenizer library, and the Hub client:
 ```bash
 pip install "mlx>=0.29" "tokenizers>=0.22" huggingface_hub
 ```
+Download the repository and use its included loader:
 ```python
 from huggingface_hub import snapshot_download
 import sys
+model_dir = snapshot_download("solintellegence/Sol-Milkshake")
 sys.path.insert(0, model_dir)
 from modeling_sol_milkshake import load_model, generate
 print(text)
 ```
+## Evaluation
+| Benchmark | Examples | Normalized accuracy |
+|---|---:|---:|
 | HellaSwag | 10,042 | 25.02% |
 | ARC-Easy | 2,376 | 29.76% |
 | ARC-Challenge | 1,172 | 22.70% |
 | PIQA | 1,838 | 52.50% |
 | ArithMark-3 | 1,000 | 30.60% |
+The reported Intelligence Index is 3.158. The four language-model tasks use full zero-shot splits with `lm-eval` 0.4.12. ArithMark-3 uses independent tokenization and normalized accuracy. Raw outputs are in `evals/`.
+The checkpoint was selected from recovery runs using this same evaluation suite. These scores therefore include checkpoint-selection effects and are not an untouched held-out estimate. They have not been independently verified.
+## Training
+Initial pretraining used FineWeb-Edu, FinePDFs-Edu, English UltraFineWeb multi-domain and question-answer subsets, Cosmopedia, and FineMath. Recovery training mixed 70% of the original frozen curriculum with 15% Cosmopedia-v2 English and 15% FinePhrase.
+The tokenizer and prepared streams were fixed. Recovery retained the existing filtering, deduplication, and decontamination rules. Training details are recorded in `training_state.json`.
+## Files
+`model.npz` holds the MLX weights. `modeling_sol_milkshake.py` includes the loader and generator, and `sol_config.py` contains the architecture settings. The repository also includes `config.json`, the tokenizer files, training metadata, and evaluation outputs.
+## Limits and license
+Milkshake is a small research model for recurrence, memory, hyperspherical optimization, and MLX inference. It can produce repetitive or incoherent text and incorrect answers. It has not been instruction-tuned.
+The model is licensed under [CC BY 4.0](LICENSE). Dataset licenses remain separate.