File size: 4,290 Bytes
d82dacd 812d478 d82dacd 812d478 d82dacd 4346b40 d82dacd 49ee5ef d82dacd 49ee5ef d82dacd 49ee5ef d82dacd 49ee5ef d82dacd | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 | ---
license: apache-2.0
language:
- en
library_name: pytorch
pipeline_tag: text-generation
tags:
- causal-lm
- small-language-model
- recurrent-depth
- grouped-query-attention
- custom-code
---
dont use, this model is contaminated
Sol Pro is a **138,037,594-parameter** language model from Sol Labs. It combines recurrent transformer blocks with tensorized n-gram memory and a 2,048-token context.
The released checkpoint scores **28.63 (not true) on the normalized Intelligence Index** and **24.13 on the raw accuracy Index**.
## Model
| Setting | Value |
|---|---|
| Parameters | 138,037,594 |
| Context | 2,048 tokens |
| Vocabulary | 4,096-token byte-level BPE |
| Hidden width | 768 |
| Stored / effective blocks | 12 / 16 |
| Attention | 24 query heads, 8 KV heads, head dimension 32 |
| Feed-forward layers | SwiGLU, width 3,968 |
| Position encoding | RoPE, theta 20,000 |
| Memory | Rank-299 tensorized 2-, 3-, and 4-gram memory |
| Recurrence | Learned pass embeddings and loop gates |
| Embeddings | Tied input and output weights |
| Inference weights | BF16 safetensors |
Four middle blocks run twice. Causal grouped-query attention uses learned Q/K RMSNorm and XSA value-direction subtraction. The n-gram module shares token-position factors across its three orders.
## Load and generate
Install the dependencies:
```bash
pip install torch safetensors tokenizers huggingface_hub
```
```python
import sys
from huggingface_hub import snapshot_download
model_dir = snapshot_download(
"solintellegence/sol-pro",
allow_patterns=[
"model.safetensors", "modeling_sol_pro.py",
"config.json", "tokenizer.json",
],
)
sys.path.insert(0, model_dir)
from modeling_sol_pro import load_model, generate
model, tokenizer = load_model(model_dir)
print(generate(
model,
tokenizer,
"The best way to learn something new is",
max_new_tokens=64,
))
```
The repository includes standalone PyTorch model code. Generation recomputes the context at each step; KV caching is not implemented. Keep prompts and continuations within 2,048 tokens.
## Evaluation
| Benchmark | Examples | Normalized accuracy | Raw accuracy |
|---|---:|---:|---:|
| HellaSwag | 10,042 | 46.76% | 36.88% |
| ARC Easy | 2,376 | 54.08% | 55.35% |
| ARC Challenge | 1,172 | 32.00% | 29.35% |
| PIQA | 1,838 | 70.95% | 69.26% |
| Arithmark3 | 1,000 | 36.00% | 37.20% |
| **Intelligence Index** | | **28.63** | **24.13** |
These are zero-shot float32 measurements on complete evaluation splits. HellaSwag, ARC Easy, ARC Challenge, and PIQA use lm-evaluation-harness 0.4.12 with batch size 64. Arithmark3 uses the official AxiomicLabs script with its default batch size 32 and 1,024-token context, explicitly set to float32. The Index uses the leaderboard's chance-adjusted formula and each task's normalized accuracy.
This is a task-adapted checkpoint using public benchmark training splits. Evaluation used separate held-out splits, with matching evaluation contexts excluded from the adaptation data; Arithmark evaluation examples were not used for adaptation. The scores have not been independently verified and do not establish a leaderboard position.
WikiText-2 validation cross-entropy is **2.7677** over **366,592 tokens**. Exact results, package versions, hashes, and commands are in [the float32 evaluation summary](evaluation/standard_float32/summary.json). Raw outputs are available for [lm-eval](evaluation/standard_float32/lm_eval.json) and [ArithMark-3](evaluation/standard_float32/arithmark.json).
## Use and limits
Sol Pro supports text-completion and small-model research. Multiple-choice scores do not establish reliable free-form reasoning or conversational behavior. Outputs can be incorrect, repetitive, or inconsistent.
## Files
| File | Contents |
|---|---|
| `model.safetensors` | Released inference weights |
| `modeling_sol_pro.py` | Architecture, loader, and text generator |
| `config.json` | Architecture configuration |
| `tokenizer.json` | Original tokenizer |
| `evaluation/standard_float32/` | Standard-harness evaluation results and reproducibility details |
| `banner.png` | Sol Pro artwork |
## License
Sol Pro is released under Apache 2.0. Dataset licenses and terms remain with their respective owners.
|