Document JSONLite descendant training and main-checkpoint results

ea6ddf7 verified 2 days ago

9 kB

---
language:
- en
pipeline_tag: text-generation
library_name: mlx
tags:
- base-model
- pretrained
- small-language-model
- apple-silicon
- from-scratch
- research
datasets:
- HuggingFaceTB/smollm-corpus
- HuggingFaceTB/dclm-edu
- HuggingFaceFW/finewiki
- Avelina/python-edu-cleaned
---
# OpenSML-Base-175M

**Author:** William Zebrowski · **175.6M parameters** · **17.30 billion pretraining token positions**

[Run locally](RUN_GUIDE.md) · [Technical report](TECHNICAL_REPORT.md) · [Evaluation results](EVALUATION_RESULTS.json) · [Checkpoint provenance](checkpoint_provenance.json)

An English pretrained causal language model built from random initialization on
educational web text, Wikipedia, synthetic educational writing and Python. This
is the pretrained ancestor of [OpenSML-JSONLite-175M-Instruct](https://huggingface.co/OpenSML/OpenSML-JSONLite-175M).
It supports text completion and further fine-tuning. Its weights contain Stage A
pretraining and continuation only, without QA/extraction SFT or a JSON adapter.

## Download and complete text

The repository includes **`model.safetensors`**, frozen tokenizer, configuration,
native architecture source and a standalone completion runner. The verified
runtime is Apple Silicon macOS, Python 3.11 and stock MLX 0.32.3.

```bash
python3 -m venv .venv
source .venv/bin/activate
python -m pip install huggingface_hub
# Authenticate if access requires it: hf auth login
hf download OpenSML/OpenSML-Base-175M --local-dir ./OpenSML-Base-175M
cd OpenSML-Base-175M
python -m pip install -r requirements.txt
python inference.py --prompt "Plants need sunlight because" --max-tokens 64
```

The runner applies no chat template. It takes a raw prefix and prints its
continuation. Use `--prompt-file prompt.txt --json` for a UTF-8 prefix and a
machine-readable generation record. Greedy decoding is the default; optional
sampling uses `--temperature 0.7 --seed 42`. Prompt plus generation budget must
fit 2,048 tokens; overlong requests are rejected without silent truncation.

This custom architecture uses the supplied native runner, rather than
`transformers.AutoModel` or `mlx_lm.load`. CUDA, CPU-only, quantized and converted
exports have not been validated. After downloading artifacts and installing
dependencies, generation runs offline. The root runner defaults to the adjacent
weight file and verifies its SHA-256 and source bundle. The source bundle includes
all native tokenizer integrity companions. See [RUN_GUIDE.md](RUN_GUIDE.md).

## Model specification

| Setting | Value |
| --- | --- |
| Parameters | 175,611,648 |
| Transformer layers | 24 |
| Hidden / FFN width | 768 / 2,048 |
| Attention | GQA: 12 query heads / 4 KV heads; head dimension 64 |
| Normalization | RMSNorm and Q/K normalization |
| Position encoding | RoPE, base 10,000 |
| MLP | SwiGLU |
| Embeddings | Tied input/output |
| Linear biases | Disabled |
| Vocabulary | Frozen 32,000-token byte-level BPE |
| Context | 2,048 tokens |
| Published weights | BF16, unquantized safetensors |

The earlier OpenSML tokenizer is reused unchanged. OpenSML-150M weights did not
initialize this model. The native tokenizer's structural IDs are pad 0,
document-end 1, turn-start 2 and turn-end 3; the base runner uses raw text.

## Pretraining

The selected checkpoint processed **17,300,111,360 lifetime token positions**.
The intended sampling mixture was:

| Corpus component | Sampling weight |
| --- | --- |
| FineWeb-Edu | 50% |
| DCLM-Edu | 18% |
| English FineWiki | 13% |
| Cosmopedia v2 | 9% |
| Educational Python text mirror | 10% |

These are configured token-sampling weights, not unique-document shares. Source
revisions, score thresholds, filters and split policy are in `corpus.json` and
the report. Training uses next-token prediction on a five-Mac Apple Silicon
cluster with Thunderbolt/RDMA. AdamW uses betas (0.9, 0.95), epsilon 1e-8, weight
decay 0.1 and gradient clipping 1.0, with FP32 master weights and moments.

Nominal peak LR was 3e-4; an adaptive plateau controller caps the scheduled rate.
The effective cap at the selected checkpoint is **3.221225472e-5**, after nine
controller reductions. Validation occurs at 50M-token boundaries, with patience
12 evaluations, cooldown 10 and minimum delta 0.002. The original budget and
continuation target were higher than the selected artifact's 17.30B positions;
this card does not claim a completed 30B run. See the exact frozen optimizer and
scheduler record in `checkpoint_metadata.json`.

### Training run

The 175.6M-parameter base was trained from random initialization using next-token prediction across the educational web, Wikipedia, Cosmopedia and Python mixture. Five Apple Silicon Macs carried out distributed pretraining over Thunderbolt/RDMA. Validation was measured every 50M tokens. The curves show a steep early loss reduction followed by slower improvement; the published base was selected using held-out validation.

![Stage A pretraining training loss, EMA and held-out validation loss](pretraining-training-loss.png)

*Training loss and held-out validation during pretraining. Left: raw training loss and exponential moving average. Right: held-out validation loss.*

## Recorded evaluation

| Metric | Selected base |
| --- | --- |
| Fixed document-validation loss | **2.5563359552** |
| Perplexity | **12.8885** |
| FineWeb-Edu loss | 2.9417 |
| DCLM-Edu loss | 2.9943 |
| FineWiki loss | 1.9878 |
| Cosmopedia loss | 1.7874 |
| Python loss | 1.2725 |

These are historical document-prediction measurements for this exact artifact.
The fixed panel has 32 batches per source, batch size 2, five sources and
2,048-token sequences; aggregate loss follows the training mixture. The exact
validation fingerprint and source losses are in `EVALUATION_RESULTS.json`.

No verified downstream benchmark result for this exact weight hash was found in
the retained benchmark-results inventory. JSONLite extraction scores and other
OpenSML checkpoint ARC/IFEval scores are not attributed to this base. Historical
qualitative samples are in `PRETRAINING_SAMPLES.json`; they show repetition and
unsupported continuations as well as plausible first completions. They are not
factual accuracy or instruction-following benchmarks.

## Intended use and limitations

Use as a research text-completion model or a starting point for adaptation.
Pretraining does not establish reliable instruction following, conversation,
structured extraction, code execution correctness or calibrated factuality.
Outputs can repeat, invent facts and continue unrelated document patterns.
The small context and parameter budget also limit long-document work. Safety,
privacy, multilingual transfer and prompt-injection robustness have not been
comprehensively evaluated.

## Identity and reproducibility

Published `model.safetensors` has **351,242,342 bytes** and SHA-256:

`5151567e266244c9e32fd6e01775192633abe227320269ecdefe28ce43e9ccc3`

The repository name identifies the model; exact training counters and metadata
are kept in provenance. The tokenizer fingerprint is
`1194292c6d906ac19e6f01c6ad2b42825143fad426a22aa8856d903776701ecb`.
`SHA256SUMS` checks downloadable artifacts. The inference source bundle contains
model, kernels and tokenizer assets, without SFT data, optimizer states or cluster
credentials. Package smoke tests establish runnable native completion, not a
reproduction of the historical validation panel or a new blind benchmark.

## Licensing and attribution

Author-controlled source contributions use the included Apache-2.0 license.
The model-weight license remains unassigned; the source license does not grant
additional weight rights. YAML therefore omits a model-weight license identifier.
Training sources retain their own terms. See `THIRD_PARTY_NOTICES.md`,
`corpus.json`, the report and `CITATION.cff` for notices. Dataset declarations do
not establish blanket underlying-content redistribution clearance.


## JSONLite descendant

The [OpenSML-JSONLite-175M-Instruct](https://huggingface.co/OpenSML/OpenSML-JSONLite-175M) descendant uses 100,096,883 additional structured-data continued-pretraining tokens followed by one joint full-weight SFT. The main checkpoint inherits 32,000 SFT presentations. On the observed original 1,444-request extraction panel it scores 89.54% complete correctness, 94.91% field accuracy, 93.75% extracted-value F1, 96.13% missing-field accuracy and 99.93% schema validity. Broader structural diagnostics remain weaker: 183/384 complete requests (47.66%). These are descendant results, not scores of this base. See the [JSONLite technical report](https://huggingface.co/OpenSML/OpenSML-JSONLite-175M/blob/main/TECHNICAL_REPORT.md).

JSONLite's supplied runner targets short English text and flat string-based schemas. Broader JSON tasks, arbitrary domains, chat and multilingual reliability are not established. Check schema validity and source-grounded meaning separately. The original base weights, tokenizer and inference runtime are unchanged.

	---
	language:
	- en
	pipeline_tag: text-generation
	library_name: mlx
	tags:
	- base-model
	- pretrained
	- small-language-model
	- apple-silicon
	- from-scratch
	- research
	datasets:
	- HuggingFaceTB/smollm-corpus
	- HuggingFaceTB/dclm-edu
	- HuggingFaceFW/finewiki
	- Avelina/python-edu-cleaned
	---

	# OpenSML-Base-175M

	Author: William Zebrowski · 175.6M parameters · 17.30 billion pretraining token positions

	[Run locally](RUN_GUIDE.md) · [Technical report](TECHNICAL_REPORT.md) · [Evaluation results](EVALUATION_RESULTS.json) · [Checkpoint provenance](checkpoint_provenance.json)

	An English pretrained causal language model built from random initialization on
	educational web text, Wikipedia, synthetic educational writing and Python. This
	is the pretrained ancestor of [OpenSML-JSONLite-175M-Instruct](https://huggingface.co/OpenSML/OpenSML-JSONLite-175M).
	It supports text completion and further fine-tuning. Its weights contain Stage A
	pretraining and continuation only, without QA/extraction SFT or a JSON adapter.

	## Download and complete text

	The repository includes `model.safetensors`, frozen tokenizer, configuration,
	native architecture source and a standalone completion runner. The verified
	runtime is Apple Silicon macOS, Python 3.11 and stock MLX 0.32.3.

	```bash
	python3 -m venv .venv
	source .venv/bin/activate
	python -m pip install huggingface_hub
	# Authenticate if access requires it: hf auth login
	hf download OpenSML/OpenSML-Base-175M --local-dir ./OpenSML-Base-175M
	cd OpenSML-Base-175M
	python -m pip install -r requirements.txt
	python inference.py --prompt "Plants need sunlight because" --max-tokens 64
	```

	The runner applies no chat template. It takes a raw prefix and prints its
	continuation. Use `--prompt-file prompt.txt --json` for a UTF-8 prefix and a
	machine-readable generation record. Greedy decoding is the default; optional
	sampling uses `--temperature 0.7 --seed 42`. Prompt plus generation budget must
	fit 2,048 tokens; overlong requests are rejected without silent truncation.

	This custom architecture uses the supplied native runner, rather than
	`transformers.AutoModel` or `mlx_lm.load`. CUDA, CPU-only, quantized and converted
	exports have not been validated. After downloading artifacts and installing
	dependencies, generation runs offline. The root runner defaults to the adjacent
	weight file and verifies its SHA-256 and source bundle. The source bundle includes
	all native tokenizer integrity companions. See [RUN_GUIDE.md](RUN_GUIDE.md).

	## Model specification

	\| Setting \| Value \|
	\| --- \| --- \|
	\| Parameters \| 175,611,648 \|
	\| Transformer layers \| 24 \|
	\| Hidden / FFN width \| 768 / 2,048 \|
	\| Attention \| GQA: 12 query heads / 4 KV heads; head dimension 64 \|
	\| Normalization \| RMSNorm and Q/K normalization \|
	\| Position encoding \| RoPE, base 10,000 \|
	\| MLP \| SwiGLU \|
	\| Embeddings \| Tied input/output \|
	\| Linear biases \| Disabled \|
	\| Vocabulary \| Frozen 32,000-token byte-level BPE \|
	\| Context \| 2,048 tokens \|
	\| Published weights \| BF16, unquantized safetensors \|

	The earlier OpenSML tokenizer is reused unchanged. OpenSML-150M weights did not
	initialize this model. The native tokenizer's structural IDs are pad 0,
	document-end 1, turn-start 2 and turn-end 3; the base runner uses raw text.

	## Pretraining

	The selected checkpoint processed 17,300,111,360 lifetime token positions.
	The intended sampling mixture was:

	\| Corpus component \| Sampling weight \|
	\| --- \| --- \|
	\| FineWeb-Edu \| 50% \|
	\| DCLM-Edu \| 18% \|
	\| English FineWiki \| 13% \|
	\| Cosmopedia v2 \| 9% \|
	\| Educational Python text mirror \| 10% \|

	These are configured token-sampling weights, not unique-document shares. Source
	revisions, score thresholds, filters and split policy are in `corpus.json` and
	the report. Training uses next-token prediction on a five-Mac Apple Silicon
	cluster with Thunderbolt/RDMA. AdamW uses betas (0.9, 0.95), epsilon 1e-8, weight
	decay 0.1 and gradient clipping 1.0, with FP32 master weights and moments.

	Nominal peak LR was 3e-4; an adaptive plateau controller caps the scheduled rate.
	The effective cap at the selected checkpoint is 3.221225472e-5, after nine
	controller reductions. Validation occurs at 50M-token boundaries, with patience
	12 evaluations, cooldown 10 and minimum delta 0.002. The original budget and
	continuation target were higher than the selected artifact's 17.30B positions;
	this card does not claim a completed 30B run. See the exact frozen optimizer and
	scheduler record in `checkpoint_metadata.json`.

	### Training run

	The 175.6M-parameter base was trained from random initialization using next-token prediction across the educational web, Wikipedia, Cosmopedia and Python mixture. Five Apple Silicon Macs carried out distributed pretraining over Thunderbolt/RDMA. Validation was measured every 50M tokens. The curves show a steep early loss reduction followed by slower improvement; the published base was selected using held-out validation.

	![Stage A pretraining training loss, EMA and held-out validation loss](pretraining-training-loss.png)

	Training loss and held-out validation during pretraining. Left: raw training loss and exponential moving average. Right: held-out validation loss.

	## Recorded evaluation

	\| Metric \| Selected base \|
	\| --- \| --- \|
	\| Fixed document-validation loss \| 2.5563359552 \|
	\| Perplexity \| 12.8885 \|
	\| FineWeb-Edu loss \| 2.9417 \|
	\| DCLM-Edu loss \| 2.9943 \|
	\| FineWiki loss \| 1.9878 \|
	\| Cosmopedia loss \| 1.7874 \|
	\| Python loss \| 1.2725 \|

	These are historical document-prediction measurements for this exact artifact.
	The fixed panel has 32 batches per source, batch size 2, five sources and
	2,048-token sequences; aggregate loss follows the training mixture. The exact
	validation fingerprint and source losses are in `EVALUATION_RESULTS.json`.

	No verified downstream benchmark result for this exact weight hash was found in
	the retained benchmark-results inventory. JSONLite extraction scores and other
	OpenSML checkpoint ARC/IFEval scores are not attributed to this base. Historical
	qualitative samples are in `PRETRAINING_SAMPLES.json`; they show repetition and
	unsupported continuations as well as plausible first completions. They are not
	factual accuracy or instruction-following benchmarks.

	## Intended use and limitations

	Use as a research text-completion model or a starting point for adaptation.
	Pretraining does not establish reliable instruction following, conversation,
	structured extraction, code execution correctness or calibrated factuality.
	Outputs can repeat, invent facts and continue unrelated document patterns.
	The small context and parameter budget also limit long-document work. Safety,
	privacy, multilingual transfer and prompt-injection robustness have not been
	comprehensively evaluated.

	## Identity and reproducibility

	Published `model.safetensors` has 351,242,342 bytes and SHA-256:

	`5151567e266244c9e32fd6e01775192633abe227320269ecdefe28ce43e9ccc3`

	The repository name identifies the model; exact training counters and metadata
	are kept in provenance. The tokenizer fingerprint is
	`1194292c6d906ac19e6f01c6ad2b42825143fad426a22aa8856d903776701ecb`.
	`SHA256SUMS` checks downloadable artifacts. The inference source bundle contains
	model, kernels and tokenizer assets, without SFT data, optimizer states or cluster
	credentials. Package smoke tests establish runnable native completion, not a
	reproduction of the historical validation panel or a new blind benchmark.

	## Licensing and attribution

	Author-controlled source contributions use the included Apache-2.0 license.
	The model-weight license remains unassigned; the source license does not grant
	additional weight rights. YAML therefore omits a model-weight license identifier.
	Training sources retain their own terms. See `THIRD_PARTY_NOTICES.md`,
	`corpus.json`, the report and `CITATION.cff` for notices. Dataset declarations do
	not establish blanket underlying-content redistribution clearance.


	## JSONLite descendant

	The [OpenSML-JSONLite-175M-Instruct](https://huggingface.co/OpenSML/OpenSML-JSONLite-175M) descendant uses 100,096,883 additional structured-data continued-pretraining tokens followed by one joint full-weight SFT. The main checkpoint inherits 32,000 SFT presentations. On the observed original 1,444-request extraction panel it scores 89.54% complete correctness, 94.91% field accuracy, 93.75% extracted-value F1, 96.13% missing-field accuracy and 99.93% schema validity. Broader structural diagnostics remain weaker: 183/384 complete requests (47.66%). These are descendant results, not scores of this base. See the [JSONLite technical report](https://huggingface.co/OpenSML/OpenSML-JSONLite-175M/blob/main/TECHNICAL_REPORT.md).

	JSONLite's supplied runner targets short English text and flat string-based schemas. Broader JSON tasks, arbitrary domains, chat and multilingual reliability are not established. Check schema validity and source-grounded meaning separately. The original base weights, tokenizer and inference runtime are unchanged.