Blame for README.md

---
language:
- en
license: apache-2.0
pipeline_tag: text-generation
library_name: mlx
tags:
- small-language-model
- from-scratch
- text-generation
- conversational
- apple-silicon
- research
datasets:
- HuggingFaceTB/smollm-corpus
- HuggingFaceTB/dclm-edu
- HuggingFaceFW/finewiki
- hongliu9903/stack_edu_python
- hongliu9903/stack_edu_sql
- iNeil77/SynSQL-2.5M
- HuggingFaceTB/smol-smoltalk
- HuggingFaceTB/smoltalk
- rajpurkar/squad_v2
- allenai/sciq
---

# Solstice-1

**A 175M-parameter English chat model, trained from scratch on five Macs.**

William Zebrowski · OpenSML · Apache-2.0 · Research preview

[Weights](https://huggingface.co/OpenSML/Solstice-1) · [Source](https://github.com/williamzebrowskI/Solstice-1) · [Run guide](RUN_GUIDE.md) · [Technical report](docs/TECHNICAL_REPORT.md) · [Evaluation](docs/EVALUATION.md) · [Data](docs/DATA.md)

Solstice-1 is the first model in the Solstice family. It combines **42.001B pretraining tokens** with three stages of adapter fine-tuning for short conversations, reading comprehension and instruction following. The release includes merged FP32 weights, the original tokenizer, an MLX runner, training records and benchmark outputs.

## Model specification

| Setting | Value |
| --- | --- |
| Parameters | 175M rounded; 174,202,432 exact |
| Transformer | 44 layers; hidden width 576; SwiGLU width 1,536 |
| Attention | 9 query heads, 3 KV heads; head dimension 64 |
| Components | RoPE (base 10,000), RMSNorm, Q/K normalization, tied embeddings |
| Tokenizer | 32,000-token byte-level BPE, reused from OpenSML |
| Context | 2,048 tokens, including conversation history and generated output |
| Pretraining | 42,001,367,040 processed token positions |
| Release | Merged FP32 safetensors; native MLX on Apple Silicon |

Solstice-1 has a separate weight lineage from OpenSML-150M and OpenSML-JSONLite-175M. Reusing the tokenizer does not make those models its weight ancestors.

## Measured results

### Knowledge and commonsense

Zero-shot likelihood scoring on the complete ARC Easy, ARC Challenge, PIQA and HellaSwag evaluation splits. Accuracy ranks the total answer log-likelihood; `acc_norm` ranks it after dividing by answer length in characters. Higher is better.

<table>
<tbody>
<tr><td role="columnheader" align="center"><strong>Benchmark</strong></td><td role="columnheader" align="center"><strong>Solstice-1</strong><br><sub><em>acc / acc_norm</em></sub></td><td role="columnheader" align="center"><strong>OpenSML-150M</strong><br><sub><em>acc / acc_norm</em></sub></td></tr>
<tr><td>ARC Easy</td><td><strong>62.58</strong> / <strong>62.33</strong></td><td>56.65 / 55.43</td></tr>
<tr><td>ARC Challenge</td><td><strong>29.35</strong> / <strong>30.89</strong></td><td>26.02 / 29.52</td></tr>
<tr><td>PIQA</td><td><strong>66.49</strong> / <strong>66.21</strong></td><td>64.53 / 64.09</td></tr>
<tr><td>HellaSwag</td><td><strong>32.85</strong> / <strong>39.40</strong></td><td>30.55 / 33.94</td></tr>
</tbody>
</table>

Values are percentages, shown as **acc / acc_norm**. Bold marks the higher score for each metric in this comparison.

### Instruction following

IFEval checks whether generated responses follow explicit instructions. It measures accuracy, with separate strict and loose scoring at the prompt and instruction levels. **It has no `acc_norm` metric.**

| IFEval accuracy | Solstice-1 | OpenSML-150M |
| --- | --- | --- |
| Strict prompts | **17.93%** | 15.16% |
| Strict instructions | **30.10%** | 25.30% |
| Loose prompts | **19.22%** | 15.71% |
| Loose instructions | **32.49%** | 25.78% |

Solstice uses greedy generation, no system message or repetition penalty, and a 1,024-token output cap. Of 541 prompts, 130 reach that cap; they remain in the scores. The 150M release used a 1,280-token cap and a different chat format.

Comparison source: [OpenSML-150M model card](https://huggingface.co/OpenSML/OpenSML-150M).

Both models use FP32 choice scoring, but implementations and chat formats differ. The score differences have not been tested for statistical significance. [150M results and protocol](results/baselines/published_models.json).

### Conversation checks

A separate authored chat panel measures short-answer matching and response behavior. These automatic checks are narrower than human judgments of conversation quality.

| Measure | Solstice-1 |
| --- | ---: |
| Answer accuracy, objective cases | 59.44% |
| Natural stopping, all responses | 99.48% |
| Repetition flags, all responses | 1.56% |

The panel contains 180 checked answers and 12 open-ended responses. Repetition flags were reviewed: two were clear loops and one was a milder repeated idea. The full panel and scorer are included in [the saved results](results/public/chat/results.json).

On the separate reserved authored test, Solstice reaches **51.9%**, versus **20.0%** after SFT 1 and **25.9%** after SFT 2. On an older, reused panel, the corresponding scores are **53.84%, 52.13% and 54.55%**: the last stage does not improve every measure. The [evaluation report](docs/EVALUATION.md) explains selection, uncertainty and test exposure.

## Pretraining

Pretraining starts from random weights. Five Apple Silicon workers use BF16 compute with FP32 master weights, gradients and optimizer state. The 2,048-token sequence length and accumulated batch produce **131,072 token positions per update**. AdamW uses betas (0.9, 0.95), weight decay 0.1 and gradient clipping at 1.

### Data mixture

| Source | First 36.801B tokens | Remaining 5.200B tokens |
| --- | ---: | ---: |
| FineWeb-Edu | 70% | 56% |
| DCLM-Edu | 20% | 16% |
| English FineWiki | 10% | 8% |
| Python: public Stack-Edu text mirror | — | 15% |
| SQL: public Stack-Edu text mirror / configured continuation | — | 5% |

These are configured token shares, not measured unique-token counts. Data is streamed with bounded shuffling, quality filters, a recent-document deduplication window and fixed train/validation partitioning. The tokenizer stays frozen. Exact dataset revisions, filtering rules and upstream terms are in [Data provenance](docs/DATA.md).

### Loss and learning rate

![Recorded training loss, prose validation loss and learning rate](docs/figures/pretraining-overview.png)

The learning rate warms up to 3e-4, then changes through the recorded continuation schedules. At 40.001B tokens, a final 2B-token cooldown lowers it from 7.94e-5 to 1e-5. The selected base finishes at **42.001B tokens**, with prose validation loss **2.8356** on the training-time BF16 evaluation path.

![Late pretraining and the final cooldown](docs/figures/cooldown-detail.png)

The shaded region is the cooldown; the dashed line marks the addition of Python and SQL. Training loss uses the changing mixture, while prose validation uses a fixed holdout. Their levels should not be treated as the same measurement. [Curve data](results/pretraining_validation.csv) · [Schedule details](docs/TECHNICAL_REPORT.md#5-distributed-optimization-and-schedules)

## Fine-tuning

All three stages run on one Mac. They train rank-32 LoRA adapters on attention projections, plus the two chat-boundary token rows: **3.786M trainable parameters**. Each stage starts fresh optimizer moments and inherits the previous stage's adapters. The release merges these updates into the original FP32 base weights.

| Stage | Selected updates | Training data and objective |
| --- | ---: | --- |
| SFT 1 | 2,000 | Two passes over 48,000 conversations; assistant-only supervision with 20% prose-replay loss |
| SFT 2 | 256 | Editing, constraints, dialogue and grounding; conversational replay and a parent-distribution penalty |
| SFT 3 | 128 | Short reasoning, grounding, dialogue and abstention; replay and a lighter parent-distribution penalty |

Public SFT sources include Smol-SmolTalk, SmolTalk everyday conversations, SQuAD v2 and SciQ, alongside authored tasks. Whole conversations are retained; oversized examples and dedicated code, SQL, JSON and tool tasks are excluded from the selected text-chat recipe. The largest stage-1 conversation is 1,385 serialized tokens, within the 2,048-token context.

The final stage was planned for 256 updates; **only its first 128 are in this release**. It was selected using development checks before evaluation on the reserved test. This lineage uses supervised fine-tuning with replay and KL preservation, not DPO or reinforcement learning. Checkpoint 167 is a comparison model, not an ancestor. [Stage recipes, learning rates and selection](docs/TECHNICAL_REPORT.md#6-selected-fine-tuning-lineage) · [Exact lineage](provenance/LINEAGE.json)

## Recorded examples

Greedy FP32 outputs reproduced in the release environment. The incorrect example is included to show the gap between fluent answers and factual reliability.

| Prompt | Model answer | Assessment |
| --- | --- | --- |
| What is the capital of Japan? Give a short answer. | Tokyo | Correct |
| Anika brought a teal notebook. Greta brought a black notebook. What color was Anika’s notebook? | teal | Correct; supplied-text question¹ |
| Which planet is closest to the Sun? Give a short answer. | venus | Incorrect; the answer is Mercury |

¹ The full prompt also asks the model to use only the supplied information and give a short answer. Full prompts, a conversation-correction example and stop reasons are in [Recorded examples](docs/EXAMPLES.md).

## Run locally

Use Python 3.11 on Apple Silicon. The public model package can be downloaded without signing in.

```bash
python3.11 -m venv .venv
source .venv/bin/activate
python -m pip install huggingface_hub
hf download OpenSML/Solstice-1 --local-dir Solstice-1
cd Solstice-1
python -m pip install -r requirements.txt
python inference.py --prompt "What is the main function of the heart?" --max-new-tokens 96
python inference.py --interactive
```

The runner defaults to greedy FP32 generation and uses the native chat tokens. Interactive mode retains previous turns; `/reset` clears history and `/quit` exits. Inputs that exceed the shared context budget are rejected rather than silently truncated. [Sampling and conversation options](RUN_GUIDE.md).

The supplied runner is the supported inference path. Transformers AutoModel, MLX-LM auto-loading, GGUF and CUDA have not been validated for this package. A clean-environment check reproduced 27 saved responses exactly; this verifies the package, not general answer quality. [Verification record](PACKAGING_VERIFICATION.json).

## Limitations

- **Factual errors:** plausible explanations and short answers can be wrong.
- **Instruction following:** strict IFEval prompt accuracy is 17.93%; many longer outputs reach the token cap.
- **Answerability:** the model can decline questions whose answers are provided, yet invent answers when information is missing. Results depend strongly on wording.
- **Repetition:** loops and repeated ideas remain possible, particularly in open-ended responses.
- **Scope:** reliable coding, SQL, JSON extraction, tool use, multilingual use and long conversations are not established capabilities.
- **Evaluation coverage:** public and historical tests informed earlier research. The reserved test shares authored task families with training. No comprehensive semantic contamination audit, independent human study or safety assessment is claimed.

Solstice-1 is a research release for short English text and chat. It is not a dependable source of facts or a validated system for consequential decisions.

## Files, reproducibility and license

[Technical report](docs/TECHNICAL_REPORT.md) · [Data provenance](docs/DATA.md) · [Evaluation and failure analysis](docs/EVALUATION.md) · [Training archive](TRAINING_AND_REPRODUCIBILITY.md) · [Raw benchmark summary](results/public/summary.json)

The repository includes weights on Hugging Face, tokenizer files, inference code, sanitized training snapshots, frozen recipes, selected-checkpoint hashes, curves and saved evaluation outputs. Raw corpora, optimizer binaries and all ancestor weights are not bundled. The training archive documents the run; it is not a verified one-command retraining package.

Weights and author-controlled code use [Apache-2.0](https://huggingface.co/OpenSML/Solstice-1/blob/main/LICENSE). Upstream datasets and third-party code retain their own terms. [Source notice](SOURCE_NOTICE.md) · [Third-party notices](THIRD_PARTY_NOTICES.md) · [Citation](CITATION.cff)