# OpenSML-Base-175M: pretraining technical report

**Author:** William Zebrowski · **Parameters:** 175,611,648 · **Processed token positions:** 17,300,111,360

This report describes the pretrained ancestor of OpenSML-JSONLite-175M. Its
weight hash is `5151567e266244c9e32fd6e01775192633abe227320269ecdefe28ce43e9ccc3`. It was trained from random initialization on the
Stage A corpus and continuation. No post-training stage is included in these
weights. Frozen checkpoint metadata is in `checkpoint_metadata.json`;
`checkpoint_provenance.json` pins the selected artifact.

## Architecture

The architecture has 24 transformer layers, hidden width 768, 12 query heads,
4 KV heads (head dimension 64), RMSNorm, Q/K normalization, RoPE base 10,000,
a SwiGLU FFN with width 2,048, tied embeddings, a 32,000-token vocabulary and a
2,048-token context. Linear layers have no biases. Compute weights are BF16;
pretraining used FP32 master weights/moments/gradient accumulation. The exact
native model and kernels are included in `native_bundle.zip` and the model
configuration is `config.json`.

## 4. Tokenizer and serialization

The model uses the project's frozen 32K byte-level BPE tokenizer at `reference/opensml-150m/tokenizer/bytebpe32k_v1`. The selected base reuses that tokenizer unchanged. Its own manifest has format `sml-v2-tokenizer-v1`; the format name is not a model-weight lineage claim.

| Tokenizer artifact | SHA-256 |
| --- | --- |
| Tokenizer manifest / contract fingerprint | 1194292c6d906ac19e6f01c6ad2b42825143fad426a22aa8856d903776701ecb |
| tokenizer.json | e095f0413a11fa91fbb58f37d00e4c6bcda59beb89b28f6f680db1436a7e8891 |
| tokenizer_config.json | 619f500553367eb76f5be00a98625130ec857b9c367561334984096c029ba2a9 |
| tokenizer corpus.json | e4c47b7b6aba08f948de7c2c7cb94c58dbd09dde2ded6ac185c60856ee686d27 |

Structural token IDs:

| ID | Token | Meaning |
| --- | --- | --- |
| 0 | `<\|pad\|>` | Padding |
| 1 | `<\|doc_end\|>` | Document/end-of-sequence marker |
| 2 | `<\|turn_start\|>` | Role/turn header |
| 3 | `<\|turn_end\|>` | End of a conversational-format turn |

The base completion runner uses a raw text prefix and ends on document-end or a reserved structural token. It does not construct chat turns. The reserved role/turn IDs exist in the tokenizer but do not imply instruction tuning in the base model.

Ordinary text must round-trip through the tokenizer and cannot emit reserved structural IDs. Special-token-looking content can be rejected by the tokenizer; the API does not promise unrestricted literal handling of all structural strings. Prompt/output limits are checked without truncating source text.

## 5. Pretraining corpus and curation

The selected base is the best retained checkpoint at update 132005 after **17,300,111,360 processed token positions**. The original budget was 20B; a continuation was configured toward 30B, but this inherited checkpoint is not a completed 20B or 30B artifact.

Source weights below are intended processed-token sampling weights, not measured unique-document shares. No claim of 17.3B globally unique corpus tokens is made.

| Source | Sampling weight (%) | Repository / configuration | Pinned revision | Additional selection |
| --- | --- | --- | --- | --- |
| FineWeb-Edu | 50 | HuggingFaceTB/smollm-corpus / fineweb-edu-dedup | 3ba9d605774198c5868892d7a8deda78031a781f | Deduplicated educational subset |
| DCLM-Edu | 18 | HuggingFaceTB/dclm-edu | dbad8ad71224482740cd9c9d353591adbf62fe04 | edu_int_score≥3 |
| FineWiki English | 13 | HuggingFaceFW/finewiki / en | 8bd13e72e6a002407649b3e898535f42ceb1aeb9 | English |
| Cosmopedia v2 | 9 | HuggingFaceTB/smollm-corpus / cosmopedia-v2 | 3ba9d605774198c5868892d7a8deda78031a781f | Format contains textbook/tutorial/blog/educational |
| Python text mirror | 10 | Avelina/python-edu-cleaned / default | 3fedf55e02e72dd1fd68963f991f5e91adcb0f8d | int_score≥4 |

Public source cards: [SmolLM corpus](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus/blob/3ba9d605774198c5868892d7a8deda78031a781f/README.md), [DCLM-Edu](https://huggingface.co/datasets/HuggingFaceTB/dclm-edu/blob/dbad8ad71224482740cd9c9d353591adbf62fe04/README.md), [FineWiki](https://huggingface.co/datasets/HuggingFaceFW/finewiki/blob/8bd13e72e6a002407649b3e898535f42ceb1aeb9/README.md). The Python-Edu upstream subset identifies files; this run used the configured text mirror. The mirror's pinned README could not be retrieved during report preparation, so its provenance statement here comes from the local corpus specification, not a newly completed external audit.

The exact corpus specification is [`corpus.json`](corpus.json). Training preserves source text and code indentation. NFC/whitespace normalization is used for identity hashing and partitioning, not to rewrite model input.

The content-split policy hashes identity into 10,000 buckets: 97% training, 1% validation, 1% final and 1% tokenizer development. Identity checks do not establish that a supervised benchmark article never appeared elsewhere in pretraining web data.

Recorded stream settings are a 256-document accepted-document shuffle buffer, bounded 100,000-document deduplication window, 4 prefetched batches and up to 5 retry attempts. The corpus JSON's generic 10,000 shuffle value is not the active stream override. Filtering rejects documents below 200 characters, above 200,000 UTF-8 bytes, with null characters, excess replacement characters or an excessive repeated-line pattern. Source exhaustion fails rather than silently recycling. Deduplication is bounded; it is not global semantic or fuzzy deduplication.

## 6. Pretraining execution, optimization and validation

Pretraining runs on the recorded five-Mac Apple Silicon cluster. Mac1's current local hardware was independently checked during report preparation as an Apple M5 Ultra with 96 GB unified memory. Other worker descriptions below are historical cluster records, not newly queried hardware inventories.

| Mac | Recorded chip | RAM | Microbatch | MLX cache cap |
| --- | --- | --- | --- | --- |
| 1 | M5 Ultra | 96 GB | 7 | 16 GiB |
| 2 | M3 Ultra | 96 GB | 3 | 16 GiB |
| 3 | M4 Max | 36 GB | 2 | 4 GiB |
| 4 | M4 Max | 36 GB | 2 | 4 GiB |
| 5 | M4 Max | 36 GB | 2 | 4 GiB |

The physical ring is 1→2→3→5→4→1. Four accumulation microsteps and 2,048-token sequences produce `(7+3+2+2+2)×4×2048=131072` positions/update. Unequal-rank gradients are weighted by token counts. Early batch transitions mean the lifetime token count is not simply `132005×131072`.

- First 125 updates: microbatches [4, 4, 2, 2, 2], 114,688 positions/update, 14,336,000 tokens.
- From 125: microbatches [6, 4, 2, 2, 2], 131,072 positions/update.
- From 963 / 124,174,336 tokens: [7, 3, 2, 2, 2], same global batch; per-Mac cache caps recorded.
- Stage A continuation starts from 113313 / 14,850,113,536 tokens, retaining weights, FP32 optimizer state, source cursors, counters and controller history.
- Selected base: 132005 / 17,300,111,360 tokens.

Ethernet supports SSH orchestration; the training ring uses verified Thunderbolt/RDMA links. Sequential microbatch materialization, FP32 gradient accumulation and custom Metal paths were used.

| Pretraining optimizer/schedule | Recorded setting |
| --- | --- |
| Optimizer | AdamW |
| Nominal peak LR | 3e-4 |
| Original warmup | 200M tokens, 1% of the 20B budget |
| Base schedule | Warmup/stable interval/cosine; continuation stretches nominal decay to 24B–30B |
| Adaptive controller | Factor 0.8; patience 12 evaluations; cooldown 10; minimum delta 0.002 |
| Controller LR floor | 3e-5 |
| Effective cap at selected 132005 | 3.221225472e-5 |
| Recorded controller reductions / evaluations | 9 / 242 |
| Betas | (0.9,0.95) |
| Epsilon | 1e-8 |
| Weight decay | 0.1 |
| Gradient clipping | 1.0 |
| Seed | 7337 |
| Loss | Next-token cross entropy; FP32 reductions |

The nominal 3e-4 schedule is not the effective late-run learning rate: the retained controller cap bounds it. The record does not justify a claim that the run trained at 3e-4 for all 17.3B positions.

Validation and qualitative samples occur at 50M-token boundaries. The fixed validation panel has 32 batches/source, batch 2, five sources, context 2,048: nominally 655,360 positions/event. Aggregate loss is source-mixture weighted. Validation fingerprint: `4ddd461c131d4d27d07bb07f5aab3fcb5a00d4b1202066e3d9b6ee6b3129087b`. Selected-base validation loss is **2.5563359551876785**, perplexity **12.8885**. This measures document prediction, not extraction or chat reliability.

Regular saves occur every 100M tokens, plus best-validation and clean-stop saves. Exact checkpoint metadata is authoritative. The initial October 2 technical report's early observations, 20B target and old schedule describe history; they are superseded here for this selected artifact. Contemporary qualitative logs often showed roughly 41k tokens/s, but no reproducible end-to-end cluster performance/energy benchmark is established by this report. The separate 42.7k profiling candidate is not claimed as the configuration that trained this checkpoint.


### Training run

The 175.6M-parameter base was trained from random initialization using next-token prediction across the educational web, Wikipedia, Cosmopedia and Python mixture. Five Apple Silicon Macs carried out distributed pretraining over Thunderbolt/RDMA. Validation was measured every 50M tokens. The curves show a steep early loss reduction followed by slower improvement; the published base was selected using held-out validation.

![Stage A pretraining training loss, EMA and held-out validation loss](pretraining-training-loss.png)

*Training loss and held-out validation during pretraining. Left: raw training loss and exponential moving average. Right: held-out validation loss.*

## Evaluation and interpretation

`EVALUATION_RESULTS.json` gives the exact selected-base loss, source losses and
validation identity. There is no verified full downstream benchmark result for
this exact weight hash in the retained benchmark-results inventory. No older
OpenSML-150M or post-trained descendant benchmark score is attributed here.
Historical qualitative generations are in `PRETRAINING_SAMPLES.json`. They show
both plausible first completions and repeated or unsupported continuations;
they are not accuracy measurements.

## Standalone package and verification

The downloadable `model.safetensors` is byte-identical to the preserved selected
pretrained artifact, with 351,242,342 bytes. The native runtime bundle contains
only model/kernel/tokenizer code and tokenizer assets; it contains no SFT data,
JSON adapter, optimizer or source cursor state. The root runner is tested with
a raw text prefix in stock MLX. Packaging verification is distinct from a
reproduction of the historical validation run or a new benchmark.

## Use and limitations

This model supports raw text completion and further adaptation. Instruction
following, conversation, text-to-JSON extraction, multilingual reliability,
prompt-injection robustness and general safety are not established by these
pretraining results. BF16 native MLX on Apple Silicon is the verified packaging
platform. No framework-conversion, quantized or CPU/CUDA parity claim is made.

The source code uses Apache-2.0. The model-weight license remains unassigned in
this prepared repository. Dataset licenses apply separately; see
`THIRD_PARTY_NOTICES.md`. The report does not establish blanket commercial or
underlying-source redistribution clearance.


## JSONLite descendant

The [OpenSML-JSONLite-175M-Instruct](https://huggingface.co/OpenSML/OpenSML-JSONLite-175M) descendant uses 100,096,883 additional structured-data continued-pretraining tokens followed by one joint full-weight SFT. The main checkpoint inherits 32,000 SFT presentations. On the observed original 1,444-request extraction panel it scores 89.54% complete correctness, 94.91% field accuracy, 93.75% extracted-value F1, 96.13% missing-field accuracy and 99.93% schema validity. Broader structural diagnostics remain weaker: 183/384 complete requests (47.66%). These are descendant results, not scores of this base. See the [JSONLite technical report](https://huggingface.co/OpenSML/OpenSML-JSONLite-175M/blob/main/TECHNICAL_REPORT.md).

JSONLite's supplied runner targets short English text and flat string-based schemas. Broader JSON tasks, arbitrary domains, chat and multilingual reliability are not established. Check schema validity and source-grounded meaning separately. The original base weights, tokenizer and inference runtime are unchanged.