Solstice-1

A 175M-parameter English chat model, trained from scratch on five Macs.

William Zebrowski · OpenSML · Apache-2.0 · Research preview

Weights · Source · Run guide · Technical report · Evaluation · Data

Solstice-1 is the first model in the Solstice family. It combines 42.001B pretraining tokens with three stages of adapter fine-tuning for short conversations, reading comprehension and instruction following. The release includes merged FP32 weights, the original tokenizer, an MLX runner, training records and benchmark outputs.

Model specification

Setting	Value
Parameters	175M rounded; 174,202,432 exact
Transformer	44 layers; hidden width 576; SwiGLU width 1,536
Attention	9 query heads, 3 KV heads; head dimension 64
Components	RoPE (base 10,000), RMSNorm, Q/K normalization, tied embeddings
Tokenizer	32,000-token byte-level BPE, reused from OpenSML
Context	2,048 tokens, including conversation history and generated output
Pretraining	42,001,367,040 processed token positions
Release	Merged FP32 safetensors; native MLX on Apple Silicon

Solstice-1 has a separate weight lineage from OpenSML-150M and OpenSML-JSONLite-175M. Reusing the tokenizer does not make those models its weight ancestors.

Measured results

Knowledge and commonsense

Zero-shot likelihood scoring on the complete ARC Easy, ARC Challenge, PIQA and HellaSwag evaluation splits. Accuracy ranks the total answer log-likelihood; acc_norm ranks it after dividing by answer length in characters. Higher is better.

Benchmark	Solstice-1 acc / acc_norm	OpenSML-150M acc / acc_norm
ARC Easy	62.58 / 62.33	56.65 / 55.43
ARC Challenge	29.35 / 30.89	26.02 / 29.52
PIQA	66.49 / 66.21	64.53 / 64.09
HellaSwag	32.85 / 39.40	30.55 / 33.94

Values are percentages, shown as acc / acc_norm. Bold marks the higher score for each metric in this comparison.

Instruction following

IFEval checks whether generated responses follow explicit instructions. It measures accuracy, with separate strict and loose scoring at the prompt and instruction levels. It has no acc_norm metric.

IFEval accuracy	Solstice-1	OpenSML-150M
Strict prompts	17.93%	15.16%
Strict instructions	30.10%	25.30%
Loose prompts	19.22%	15.71%
Loose instructions	32.49%	25.78%

Solstice uses greedy generation, no system message or repetition penalty, and a 1,024-token output cap. Of 541 prompts, 130 reach that cap; they remain in the scores. The 150M release used a 1,280-token cap and a different chat format.

Comparison source: OpenSML-150M model card.

Both models use FP32 choice scoring, but implementations and chat formats differ. The score differences have not been tested for statistical significance. 150M results and protocol.

Conversation checks

A separate authored chat panel measures short-answer matching and response behavior. These automatic checks are narrower than human judgments of conversation quality.

Measure	Solstice-1
Answer accuracy, objective cases	59.44%
Natural stopping, all responses	99.48%
Repetition flags, all responses	1.56%

The panel contains 180 checked answers and 12 open-ended responses. Repetition flags were reviewed: two were clear loops and one was a milder repeated idea. The full panel and scorer are included in the saved results.

On the separate reserved authored test, Solstice reaches 51.9%, versus 20.0% after SFT 1 and 25.9% after SFT 2. On an older, reused panel, the corresponding scores are 53.84%, 52.13% and 54.55%: the last stage does not improve every measure. The evaluation report explains selection, uncertainty and test exposure.

Pretraining

Pretraining starts from random weights. Five Apple Silicon workers use BF16 compute with FP32 master weights, gradients and optimizer state. The 2,048-token sequence length and accumulated batch produce 131,072 token positions per update. AdamW uses betas (0.9, 0.95), weight decay 0.1 and gradient clipping at 1.

Data mixture

Source	First 36.801B tokens	Remaining 5.200B tokens
FineWeb-Edu	70%	56%
DCLM-Edu	20%	16%
English FineWiki	10%	8%
Python: public Stack-Edu text mirror	—	15%
SQL: public Stack-Edu text mirror / configured continuation	—	5%

These are configured token shares, not measured unique-token counts. Data is streamed with bounded shuffling, quality filters, a recent-document deduplication window and fixed train/validation partitioning. The tokenizer stays frozen. Exact dataset revisions, filtering rules and upstream terms are in Data provenance.

Loss and learning rate

The learning rate warms up to 3e-4, then changes through the recorded continuation schedules. At 40.001B tokens, a final 2B-token cooldown lowers it from 7.94e-5 to 1e-5. The selected base finishes at 42.001B tokens, with prose validation loss 2.8356 on the training-time BF16 evaluation path.

The shaded region is the cooldown; the dashed line marks the addition of Python and SQL. Training loss uses the changing mixture, while prose validation uses a fixed holdout. Their levels should not be treated as the same measurement. Curve data · Schedule details

Fine-tuning

All three stages run on one Mac. They train rank-32 LoRA adapters on attention projections, plus the two chat-boundary token rows: 3.786M trainable parameters. Each stage starts fresh optimizer moments and inherits the previous stage's adapters. The release merges these updates into the original FP32 base weights.

Stage	Selected updates	Training data and objective
SFT 1	2,000	Two passes over 48,000 conversations; assistant-only supervision with 20% prose-replay loss
SFT 2	256	Editing, constraints, dialogue and grounding; conversational replay and a parent-distribution penalty
SFT 3	128	Short reasoning, grounding, dialogue and abstention; replay and a lighter parent-distribution penalty

Public SFT sources include Smol-SmolTalk, SmolTalk everyday conversations, SQuAD v2 and SciQ, alongside authored tasks. Whole conversations are retained; oversized examples and dedicated code, SQL, JSON and tool tasks are excluded from the selected text-chat recipe. The largest stage-1 conversation is 1,385 serialized tokens, within the 2,048-token context.

The final stage was planned for 256 updates; only its first 128 are in this release. It was selected using development checks before evaluation on the reserved test. This lineage uses supervised fine-tuning with replay and KL preservation, not DPO or reinforcement learning. Checkpoint 167 is a comparison model, not an ancestor. Stage recipes, learning rates and selection · Exact lineage

Recorded examples

Greedy FP32 outputs reproduced in the release environment. The incorrect example is included to show the gap between fluent answers and factual reliability.

Prompt	Model answer	Assessment
What is the capital of Japan? Give a short answer.	Tokyo	Correct
Anika brought a teal notebook. Greta brought a black notebook. What color was Anika’s notebook?	teal	Correct; supplied-text question¹
Which planet is closest to the Sun? Give a short answer.	venus	Incorrect; the answer is Mercury

¹ The full prompt also asks the model to use only the supplied information and give a short answer. Full prompts, a conversation-correction example and stop reasons are in Recorded examples.

Run locally

Use Python 3.11 on Apple Silicon. The public model package can be downloaded without signing in.

python3.11 -m venv .venv
source .venv/bin/activate
python -m pip install huggingface_hub
hf download OpenSML/Solstice-1 --local-dir Solstice-1
cd Solstice-1
python -m pip install -r requirements.txt
python inference.py --prompt "What is the main function of the heart?" --max-new-tokens 96
python inference.py --interactive

The runner defaults to greedy FP32 generation and uses the native chat tokens. Interactive mode retains previous turns; /reset clears history and /quit exits. Inputs that exceed the shared context budget are rejected rather than silently truncated. Sampling and conversation options.

The supplied runner is the supported inference path. Transformers AutoModel, MLX-LM auto-loading, GGUF and CUDA have not been validated for this package. A clean-environment check reproduced 27 saved responses exactly; this verifies the package, not general answer quality. Verification record.

Limitations

• Factual errors: plausible explanations and short answers can be wrong.
• Instruction following: strict IFEval prompt accuracy is 17.93%; many longer outputs reach the token cap.
• Answerability: the model can decline questions whose answers are provided, yet invent answers when information is missing. Results depend strongly on wording.
• Repetition: loops and repeated ideas remain possible, particularly in open-ended responses.
• Scope: reliable coding, SQL, JSON extraction, tool use, multilingual use and long conversations are not established capabilities.
• Evaluation coverage: public and historical tests informed earlier research. The reserved test shares authored task families with training. No comprehensive semantic contamination audit, independent human study or safety assessment is claimed.

Solstice-1 is a research release for short English text and chat. It is not a dependable source of facts or a validated system for consequential decisions.

Files, reproducibility and license

Technical report · Data provenance · Evaluation and failure analysis · Training archive · Raw benchmark summary

The repository includes weights on Hugging Face, tokenizer files, inference code, sanitized training snapshots, frozen recipes, selected-checkpoint hashes, curves and saved evaluation outputs. Raw corpora, optimizer binaries and all ancestor weights are not bundled. The training archive documents the run; it is not a verified one-command retraining package.

Weights and author-controlled code use Apache-2.0. Upstream datasets and third-party code retain their own terms. Source notice · Third-party notices · Citation