j0no12 commited on
Commit
54f08e4
·
verified ·
1 Parent(s): fb23993

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +0 -109
README.md CHANGED
@@ -1,109 +0,0 @@
1
- ---
2
- license: cc-by-4.0
3
- language: [en]
4
- pipeline_tag: text-generation
5
- library_name: pytorch
6
- tags: [pytorch, text-generation, causal-lm, recurrent-transformer, xsa, grouped-query-attention, small-language-models, from-scratch]
7
- datasets: [HuggingFaceFW/fineweb-edu, HuggingFaceFW/finephrase, HuggingFaceTB/finemath]
8
- model-index:
9
- - name: Sol Lite Base
10
- results:
11
- - task: {type: text-generation, name: Zero-shot multiple-choice language modeling}
12
- dataset: {name: HellaSwag, type: hellaswag}
13
- metrics: [{type: accuracy_norm, value: 0.27723561043616807, verified: false}]
14
- - task: {type: text-generation, name: Zero-shot multiple-choice language modeling}
15
- dataset: {name: ARC-Easy, type: ai2_arc, config: ARC-Easy}
16
- metrics: [{type: accuracy_norm, value: 0.3472222222222222, verified: false}]
17
- - task: {type: text-generation, name: ARC-Challenge}
18
- dataset: {name: ARC-Challenge, type: ai2_arc, config: ARC-Challenge}
19
- metrics: [{type: accuracy_norm, value: 0.22866894197952217, verified: false}]
20
- - task: {type: text-generation, name: PIQA}
21
- dataset: {name: PIQA, type: piqa}
22
- metrics: [{type: accuracy_norm, value: 0.5772578890097932, verified: false}]
23
- - task: {type: text-generation, name: ArithMark-3}
24
- dataset: {name: ArithMark-3, type: AxiomicLabs/Arithmark-3.0}
25
- metrics: [{type: accuracy_norm, value: 0.342, verified: false}]
26
- ---
27
-
28
- # Sol Lite Base
29
-
30
- Sol Lite Base is a compact **14,995,843-parameter** decoder-only language model trained from scratch on **8,153,333,760 tokens**. It combines grouped-query attention, selective recurrent depth, XSA value subtraction, loop conditioning, and a small learned n-gram memory.
31
-
32
- This is a base model, not an instruction-tuned assistant.
33
-
34
- ## Quick start
35
-
36
- ```bash
37
- pip install "torch>=2.5" "transformers>=5" safetensors huggingface_hub
38
- ```
39
-
40
- ```python
41
- from huggingface_hub import snapshot_download
42
- import sys
43
-
44
- model_dir = snapshot_download("solintellegence/Sol-Lite-Base")
45
- sys.path.insert(0, model_dir)
46
-
47
- from modeling_sol_lite import load_model, generate
48
-
49
- model, tokenizer = load_model(model_dir, device="cpu")
50
- text = generate(model, tokenizer, "The future of efficient language models is", max_new_tokens=64)
51
- print(text)
52
- ```
53
-
54
- Sol Lite Base uses the included custom PyTorch class `SolForCausalLM`. The repository includes the complete standalone implementation and does not require the original training code.
55
-
56
- ## Results
57
-
58
- | Benchmark | Examples | Score |
59
- |---|---:|---:|
60
- | HellaSwag | 10,042 | 27.72% |
61
- | ARC-Easy | 2,376 | 34.72% |
62
- | ARC-Challenge | 1,172 | 22.87% |
63
- | PIQA | 1,838 | 57.73% |
64
- | ArithMark-3 | 1,000 | 34.20% |
65
- | Intelligence Index | — | **8.799** |
66
-
67
- The four language-model tasks were evaluated zero-shot on their complete evaluation splits with `lm-eval` 0.4.12 and normalized accuracy. ArithMark-3 used its independent-tokenization normalized-accuracy protocol. Raw evaluation outputs are included under `evals/`. These measurements are not independently verified, and no leaderboard position is claimed.
68
-
69
- ## Architecture
70
-
71
- | Component | Configuration |
72
- |---|---|
73
- | Stored parameters | **14,995,843** |
74
- | Physical / effective blocks | 10 / 14 |
75
- | Recurrent layout | 1 prelude + 4 middle blocks × 2 passes + 5 coda |
76
- | Residual width | 256 |
77
- | Attention | 8 query heads, 2 key/value heads, head dimension 32 |
78
- | Attention modification | XSA value subtraction after causal attention |
79
- | Attention normalization | Per-head Q/K RMSNorm with RoPE |
80
- | Recurrent conditioning | Learned pass embeddings and channel-wise refresh gates |
81
- | MLP | Gated SiLU MLP, width 1,465 |
82
- | Vocabulary | 4,096-token digit-aware byte-level BPE |
83
- | Learned memory | 2,048-entry bigram/trigram EngramLite |
84
- | Embeddings | Tied input/output table |
85
- | Maximum context | 2,048 tokens |
86
-
87
- ## Training data
88
-
89
- Training used a gradual English curriculum built from FineWeb-Edu, DCLM, English UltraFineWeb levels 1–3, FineWeb-HQ, FineMath, and FinePhrase. The tokenizer and stream were frozen before training. Public benchmark examples were not used for checkpoint selection.
90
-
91
- ## Intended use and limitations
92
-
93
- Sol Lite Base is a research model for studying highly parameter-efficient language modeling. At 15M parameters, generations can be inconsistent, factually wrong, repetitive, or unsafe. Do not use it for high-stakes decisions. Users are responsible for evaluating outputs for their application.
94
-
95
- ## Files
96
-
97
- | File | Purpose |
98
- |---|---|
99
- | `model.safetensors` | FP32 model weights |
100
- | `modeling_sol_lite.py` | Standalone model, loader, and generator |
101
- | `config.json` | Architecture configuration |
102
- | `tokenizer.json` | Frozen tokenizer |
103
- | `tokenizer_config.json` | Tokenizer metadata |
104
- | `training_state.json` | Training and provenance summary |
105
- | `evals/` | Raw benchmark results |
106
-
107
- ## License
108
-
109
- CC BY 4.0. Dataset licenses and terms remain with their respective owners.