| --- |
| license: apache-2.0 |
| language: |
| - en |
| library_name: pytorch |
| pipeline_tag: text-generation |
| tags: |
| - causal-lm |
| - small-language-model |
| - recurrent-depth |
| - grouped-query-attention |
| - custom-code |
| --- |
| |
|  |
| dont use, this model is contaminated |
|
|
| Sol Pro is a **138,037,594-parameter** language model from Sol Labs. It combines recurrent transformer blocks with tensorized n-gram memory and a 2,048-token context. |
|
|
| The released checkpoint scores **28.63 (not true) on the normalized Intelligence Index** and **24.13 on the raw accuracy Index**. |
|
|
| ## Model |
|
|
| | Setting | Value | |
| |---|---| |
| | Parameters | 138,037,594 | |
| | Context | 2,048 tokens | |
| | Vocabulary | 4,096-token byte-level BPE | |
| | Hidden width | 768 | |
| | Stored / effective blocks | 12 / 16 | |
| | Attention | 24 query heads, 8 KV heads, head dimension 32 | |
| | Feed-forward layers | SwiGLU, width 3,968 | |
| | Position encoding | RoPE, theta 20,000 | |
| | Memory | Rank-299 tensorized 2-, 3-, and 4-gram memory | |
| | Recurrence | Learned pass embeddings and loop gates | |
| | Embeddings | Tied input and output weights | |
| | Inference weights | BF16 safetensors | |
|
|
| Four middle blocks run twice. Causal grouped-query attention uses learned Q/K RMSNorm and XSA value-direction subtraction. The n-gram module shares token-position factors across its three orders. |
|
|
| ## Load and generate |
|
|
| Install the dependencies: |
|
|
| ```bash |
| pip install torch safetensors tokenizers huggingface_hub |
| ``` |
|
|
| ```python |
| import sys |
| from huggingface_hub import snapshot_download |
| |
| model_dir = snapshot_download( |
| "solintellegence/sol-pro", |
| allow_patterns=[ |
| "model.safetensors", "modeling_sol_pro.py", |
| "config.json", "tokenizer.json", |
| ], |
| ) |
| sys.path.insert(0, model_dir) |
| |
| from modeling_sol_pro import load_model, generate |
| |
| model, tokenizer = load_model(model_dir) |
| print(generate( |
| model, |
| tokenizer, |
| "The best way to learn something new is", |
| max_new_tokens=64, |
| )) |
| ``` |
|
|
| The repository includes standalone PyTorch model code. Generation recomputes the context at each step; KV caching is not implemented. Keep prompts and continuations within 2,048 tokens. |
|
|
| ## Evaluation |
|
|
| | Benchmark | Examples | Normalized accuracy | Raw accuracy | |
| |---|---:|---:|---:| |
| | HellaSwag | 10,042 | 46.76% | 36.88% | |
| | ARC Easy | 2,376 | 54.08% | 55.35% | |
| | ARC Challenge | 1,172 | 32.00% | 29.35% | |
| | PIQA | 1,838 | 70.95% | 69.26% | |
| | Arithmark3 | 1,000 | 36.00% | 37.20% | |
| | **Intelligence Index** | | **28.63** | **24.13** | |
|
|
| These are zero-shot float32 measurements on complete evaluation splits. HellaSwag, ARC Easy, ARC Challenge, and PIQA use lm-evaluation-harness 0.4.12 with batch size 64. Arithmark3 uses the official AxiomicLabs script with its default batch size 32 and 1,024-token context, explicitly set to float32. The Index uses the leaderboard's chance-adjusted formula and each task's normalized accuracy. |
|
|
| This is a task-adapted checkpoint using public benchmark training splits. Evaluation used separate held-out splits, with matching evaluation contexts excluded from the adaptation data; Arithmark evaluation examples were not used for adaptation. The scores have not been independently verified and do not establish a leaderboard position. |
|
|
| WikiText-2 validation cross-entropy is **2.7677** over **366,592 tokens**. Exact results, package versions, hashes, and commands are in [the float32 evaluation summary](evaluation/standard_float32/summary.json). Raw outputs are available for [lm-eval](evaluation/standard_float32/lm_eval.json) and [ArithMark-3](evaluation/standard_float32/arithmark.json). |
|
|
| ## Use and limits |
|
|
| Sol Pro supports text-completion and small-model research. Multiple-choice scores do not establish reliable free-form reasoning or conversational behavior. Outputs can be incorrect, repetitive, or inconsistent. |
|
|
| ## Files |
|
|
| | File | Contents | |
| |---|---| |
| | `model.safetensors` | Released inference weights | |
| | `modeling_sol_pro.py` | Architecture, loader, and text generator | |
| | `config.json` | Architecture configuration | |
| | `tokenizer.json` | Original tokenizer | |
| | `evaluation/standard_float32/` | Standard-harness evaluation results and reproducibility details | |
| | `banner.png` | Sol Pro artwork | |
|
|
| ## License |
|
|
| Sol Pro is released under Apache 2.0. Dataset licenses and terms remain with their respective owners. |
|
|
