solintellegence
/

Sol-Lite-Base

@@ -16,36 +16,37 @@ tags:
 # Sol Lite Base
-Sol Lite Base is a compact **14,995,843-parameter** decoder-only language model trained from scratch on **8,153,333,760 tokens**.
-It combines grouped-query attention, selective recurrent depth, XSA value subtraction, loop conditioning, and a small learned n-gram memory.
-**This is a base model, not an instruction-tuned assistant.**
-## Model summary
-| Property | Value |
-| --- | --- |
-| Parameters | **14,995,843** |
-| Training tokens | **8,153,333,760** |
-| Architecture | Decoder-only causal language model |
-| Physical / effective blocks | 10 / 14 |
-| Residual width | 256 |
-| Vocabulary | 4,096-token digit-aware byte-level BPE |
-| Maximum context | 2,048 tokens |
-| Weight format | FP32 safetensors |
-| License | CC BY 4.0 |
-## Quick start
-Install the runtime dependencies:
 ```bash
 pip install "torch>=2.5" "transformers>=5" safetensors huggingface_hub
 ```
-Load the model and generate text:
 ```python
 from huggingface_hub import snapshot_download
 import sys
@@ -67,115 +68,28 @@ text = generate(
 print(text)
 ```
-Sol Lite Base uses the included custom PyTorch class `SolForCausalLM`.
-The repository includes the complete standalone model implementation and does not require the original training code.
-## Results
-| Benchmark | Examples | Score |
-| --- | ---: | ---: |
 | HellaSwag | 10,042 | 27.72% |
 | ARC-Easy | 2,376 | 34.72% |
 | ARC-Challenge | 1,172 | 22.87% |
 | PIQA | 1,838 | 57.73% |
 | ArithMark-3 | 1,000 | 34.20% |
-| Intelligence Index | — | **8.799** |
-The four language-model tasks were evaluated zero-shot on their complete evaluation splits with `lm-eval` 0.4.12 using normalized accuracy.
-ArithMark-3 used its independent-tokenization normalized-accuracy protocol.
-Raw evaluation outputs are included under `evals/`.
-These measurements are not independently verified, and no leaderboard position is claimed.
-## Architecture
-| Component | Configuration |
-| --- | --- |
-| Stored parameters | **14,995,843** |
-| Physical / effective blocks | 10 / 14 |
-| Recurrent layout | 1 prelude + 4 middle blocks × 2 passes + 5 coda |
-| Residual width | 256 |
-| Attention | 8 query heads, 2 key/value heads, head dimension 32 |
-| Attention modification | XSA value subtraction after causal attention |
-| Attention normalization | Per-head Q/K RMSNorm with RoPE |
-| Recurrent conditioning | Learned pass embeddings and channel-wise refresh gates |
-| MLP | Gated SiLU MLP, width 1,465 |
-| Vocabulary | 4,096-token digit-aware byte-level BPE |
-| Learned memory | 2,048-entry bigram/trigram EngramLite |
-| Embeddings | Tied input/output table |
-| Maximum context | 2,048 tokens |
-## Training data
-Training used a gradual English curriculum built from:
-- FineWeb-Edu
-- DCLM
-- English UltraFineWeb levels 1–3
-- FineWeb-HQ
-- FineMath
-- FinePhrase
-The tokenizer and training stream were frozen before training.
-Public benchmark examples were not used for checkpoint selection.
-## Intended use
-Sol Lite Base is intended for research on highly parameter-efficient language modeling, including:
-- extremely small language models
-- recurrent-depth architectures
-- grouped-query attention at small scale
-- compact learned memory
-- distillation experiments
-- efficient inference research
-Because this is a base model, users should expect raw next-token completion behavior rather than instruction-following or assistant-style responses.
-## Limitations
-At approximately 15M parameters, Sol Lite Base has significant capability limitations.
-Outputs may be:
-- inconsistent
-- factually incorrect
-- repetitive
-- incoherent
-- unsafe
-The model is not instruction tuned and should not be treated as a conversational assistant.
-Do not use it for high-stakes decisions.
-Users are responsible for evaluating outputs and determining whether the model is suitable for their application.
-## Repository files
-| File | Purpose |
-| --- | --- |
-| `model.safetensors` | FP32 model weights |
-| `modeling_sol_lite.py` | Standalone model, loader, and generator |
-| `config.json` | Architecture configuration |
-| `tokenizer.json` | Frozen tokenizer |
-| `tokenizer_config.json` | Tokenizer metadata |
-| `training_state.json` | Training and provenance summary |
-| `evals/` | Raw benchmark results |
-## Reproducibility and evaluation notes
-The repository includes raw evaluation outputs under `evals/`.
-Reported benchmark values should be interpreted as reference measurements for this release. They have not been independently verified.
-No leaderboard position is claimed.
-## License
-Sol Lite Base is released under the **CC BY 4.0** license.
-Dataset licenses and terms remain with their respective owners.

 # Sol Lite Base
+Sol Lite Base is a 14,995,843-parameter language model trained from scratch on 8,153,333,760 tokens. It generates text completions and has not been instruction-tuned.
+The model reuses four transformer blocks for a second pass. Grouped-query attention, loop conditioning, XSA value subtraction, and EngramLite local memory make up the rest of the architecture.
+## Model
+| Setting | Value |
+|---|---|
+| Parameters | 14,995,843 |
+| Training tokens | 8,153,333,760 |
+| Context | 2,048 tokens |
+| Tokenizer | 4,096-entry digit-aware byte-level BPE |
+| Hidden width | 256 |
+| Stored blocks / effective applications | 10 / 14 |
+| Recurrent layout | 1 prelude, 4 middle blocks used twice, 5 coda blocks |
+| Attention | 8 query heads, 2 KV heads, head dimension 32 |
+| Q/K normalization | Per-head RMSNorm with RoPE |
+| Recurrence | Learned pass embeddings and channel-wise refresh gates |
+| FFN | Gated SiLU, width 1,465 |
+| Local memory | 2,048-entry bigram/trigram EngramLite |
+| Token embeddings | Tied to the output head |
+| Weights | FP32 safetensors |
+## Use
+The repository includes a standalone PyTorch implementation, loader, and generation helper.
 ```bash
 pip install "torch>=2.5" "transformers>=5" safetensors huggingface_hub
 ```
 ```python
 from huggingface_hub import snapshot_download
 import sys
 print(text)
 ```
+## Evaluation
+| Benchmark | Examples | Normalized accuracy |
+|---|---:|---:|
 | HellaSwag | 10,042 | 27.72% |
 | ARC-Easy | 2,376 | 34.72% |
 | ARC-Challenge | 1,172 | 22.87% |
 | PIQA | 1,838 | 57.73% |
 | ArithMark-3 | 1,000 | 34.20% |
+The reported Intelligence Index is 8.799. The language-model tasks use the complete zero-shot evaluation splits with `lm-eval` 0.4.12. ArithMark-3 uses independent tokenization and normalized accuracy. These are local measurements; raw outputs are in `evals/`.
+## Training
+The English curriculum draws from FineWeb-Edu, DCLM, UltraFineWeb levels 1-3, FineWeb-HQ, FineMath, and FinePhrase. The tokenizer and prepared training stream were fixed before training. Public benchmark examples were not used to select the checkpoint.
+## Files
+`model.safetensors` contains the weights. `modeling_sol_lite.py` defines `SolForCausalLM`, the loader, and the generator. Architecture settings are in `config.json`; tokenizer files are `tokenizer.json` and `tokenizer_config.json`. `training_state.json` records training provenance.
+## Limits and license
+This is a research model for small-scale language modeling, recurrence, memory, and distillation experiments. Completions can be repetitive, incoherent, or factually wrong. It has no instruction-following training or assistant safety tuning.
+The model is licensed under [CC BY 4.0](LICENSE). Upstream datasets retain their own terms.