Update README.md
Browse files
README.md
CHANGED
|
@@ -1,109 +0,0 @@
|
|
| 1 |
-
---
|
| 2 |
-
license: cc-by-4.0
|
| 3 |
-
language: [en]
|
| 4 |
-
pipeline_tag: text-generation
|
| 5 |
-
library_name: pytorch
|
| 6 |
-
tags: [pytorch, text-generation, causal-lm, recurrent-transformer, xsa, grouped-query-attention, small-language-models, from-scratch]
|
| 7 |
-
datasets: [HuggingFaceFW/fineweb-edu, HuggingFaceFW/finephrase, HuggingFaceTB/finemath]
|
| 8 |
-
model-index:
|
| 9 |
-
- name: Sol Lite Base
|
| 10 |
-
results:
|
| 11 |
-
- task: {type: text-generation, name: Zero-shot multiple-choice language modeling}
|
| 12 |
-
dataset: {name: HellaSwag, type: hellaswag}
|
| 13 |
-
metrics: [{type: accuracy_norm, value: 0.27723561043616807, verified: false}]
|
| 14 |
-
- task: {type: text-generation, name: Zero-shot multiple-choice language modeling}
|
| 15 |
-
dataset: {name: ARC-Easy, type: ai2_arc, config: ARC-Easy}
|
| 16 |
-
metrics: [{type: accuracy_norm, value: 0.3472222222222222, verified: false}]
|
| 17 |
-
- task: {type: text-generation, name: ARC-Challenge}
|
| 18 |
-
dataset: {name: ARC-Challenge, type: ai2_arc, config: ARC-Challenge}
|
| 19 |
-
metrics: [{type: accuracy_norm, value: 0.22866894197952217, verified: false}]
|
| 20 |
-
- task: {type: text-generation, name: PIQA}
|
| 21 |
-
dataset: {name: PIQA, type: piqa}
|
| 22 |
-
metrics: [{type: accuracy_norm, value: 0.5772578890097932, verified: false}]
|
| 23 |
-
- task: {type: text-generation, name: ArithMark-3}
|
| 24 |
-
dataset: {name: ArithMark-3, type: AxiomicLabs/Arithmark-3.0}
|
| 25 |
-
metrics: [{type: accuracy_norm, value: 0.342, verified: false}]
|
| 26 |
-
---
|
| 27 |
-
|
| 28 |
-
# Sol Lite Base
|
| 29 |
-
|
| 30 |
-
Sol Lite Base is a compact **14,995,843-parameter** decoder-only language model trained from scratch on **8,153,333,760 tokens**. It combines grouped-query attention, selective recurrent depth, XSA value subtraction, loop conditioning, and a small learned n-gram memory.
|
| 31 |
-
|
| 32 |
-
This is a base model, not an instruction-tuned assistant.
|
| 33 |
-
|
| 34 |
-
## Quick start
|
| 35 |
-
|
| 36 |
-
```bash
|
| 37 |
-
pip install "torch>=2.5" "transformers>=5" safetensors huggingface_hub
|
| 38 |
-
```
|
| 39 |
-
|
| 40 |
-
```python
|
| 41 |
-
from huggingface_hub import snapshot_download
|
| 42 |
-
import sys
|
| 43 |
-
|
| 44 |
-
model_dir = snapshot_download("solintellegence/Sol-Lite-Base")
|
| 45 |
-
sys.path.insert(0, model_dir)
|
| 46 |
-
|
| 47 |
-
from modeling_sol_lite import load_model, generate
|
| 48 |
-
|
| 49 |
-
model, tokenizer = load_model(model_dir, device="cpu")
|
| 50 |
-
text = generate(model, tokenizer, "The future of efficient language models is", max_new_tokens=64)
|
| 51 |
-
print(text)
|
| 52 |
-
```
|
| 53 |
-
|
| 54 |
-
Sol Lite Base uses the included custom PyTorch class `SolForCausalLM`. The repository includes the complete standalone implementation and does not require the original training code.
|
| 55 |
-
|
| 56 |
-
## Results
|
| 57 |
-
|
| 58 |
-
| Benchmark | Examples | Score |
|
| 59 |
-
|---|---:|---:|
|
| 60 |
-
| HellaSwag | 10,042 | 27.72% |
|
| 61 |
-
| ARC-Easy | 2,376 | 34.72% |
|
| 62 |
-
| ARC-Challenge | 1,172 | 22.87% |
|
| 63 |
-
| PIQA | 1,838 | 57.73% |
|
| 64 |
-
| ArithMark-3 | 1,000 | 34.20% |
|
| 65 |
-
| Intelligence Index | — | **8.799** |
|
| 66 |
-
|
| 67 |
-
The four language-model tasks were evaluated zero-shot on their complete evaluation splits with `lm-eval` 0.4.12 and normalized accuracy. ArithMark-3 used its independent-tokenization normalized-accuracy protocol. Raw evaluation outputs are included under `evals/`. These measurements are not independently verified, and no leaderboard position is claimed.
|
| 68 |
-
|
| 69 |
-
## Architecture
|
| 70 |
-
|
| 71 |
-
| Component | Configuration |
|
| 72 |
-
|---|---|
|
| 73 |
-
| Stored parameters | **14,995,843** |
|
| 74 |
-
| Physical / effective blocks | 10 / 14 |
|
| 75 |
-
| Recurrent layout | 1 prelude + 4 middle blocks × 2 passes + 5 coda |
|
| 76 |
-
| Residual width | 256 |
|
| 77 |
-
| Attention | 8 query heads, 2 key/value heads, head dimension 32 |
|
| 78 |
-
| Attention modification | XSA value subtraction after causal attention |
|
| 79 |
-
| Attention normalization | Per-head Q/K RMSNorm with RoPE |
|
| 80 |
-
| Recurrent conditioning | Learned pass embeddings and channel-wise refresh gates |
|
| 81 |
-
| MLP | Gated SiLU MLP, width 1,465 |
|
| 82 |
-
| Vocabulary | 4,096-token digit-aware byte-level BPE |
|
| 83 |
-
| Learned memory | 2,048-entry bigram/trigram EngramLite |
|
| 84 |
-
| Embeddings | Tied input/output table |
|
| 85 |
-
| Maximum context | 2,048 tokens |
|
| 86 |
-
|
| 87 |
-
## Training data
|
| 88 |
-
|
| 89 |
-
Training used a gradual English curriculum built from FineWeb-Edu, DCLM, English UltraFineWeb levels 1–3, FineWeb-HQ, FineMath, and FinePhrase. The tokenizer and stream were frozen before training. Public benchmark examples were not used for checkpoint selection.
|
| 90 |
-
|
| 91 |
-
## Intended use and limitations
|
| 92 |
-
|
| 93 |
-
Sol Lite Base is a research model for studying highly parameter-efficient language modeling. At 15M parameters, generations can be inconsistent, factually wrong, repetitive, or unsafe. Do not use it for high-stakes decisions. Users are responsible for evaluating outputs for their application.
|
| 94 |
-
|
| 95 |
-
## Files
|
| 96 |
-
|
| 97 |
-
| File | Purpose |
|
| 98 |
-
|---|---|
|
| 99 |
-
| `model.safetensors` | FP32 model weights |
|
| 100 |
-
| `modeling_sol_lite.py` | Standalone model, loader, and generator |
|
| 101 |
-
| `config.json` | Architecture configuration |
|
| 102 |
-
| `tokenizer.json` | Frozen tokenizer |
|
| 103 |
-
| `tokenizer_config.json` | Tokenizer metadata |
|
| 104 |
-
| `training_state.json` | Training and provenance summary |
|
| 105 |
-
| `evals/` | Raw benchmark results |
|
| 106 |
-
|
| 107 |
-
## License
|
| 108 |
-
|
| 109 |
-
CC BY 4.0. Dataset licenses and terms remain with their respective owners.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|