Sepalith 2B CPT checkpoints

Full-weight continued-pretraining (CPT) checkpoints of openbmb/MiniCPM5-2B-Midtrain on about 472M tokens of permissively licensed R code (10,046 packages), trained for Sepalith, a local-first R next-edit-suggestion model.

These are base checkpoints. They are not tuned for editing or chat.

Folder	Step	Role
`checkpoint-11586/`	11,586	Selected parent for editing SFT
`checkpoint-11649/`	11,649	Final CPT step, preserved alternative

Architecture: LlamaForCausalLM, 42 layers, hidden size 2048, 16 attention heads with 2 key-value heads, vocabulary 130,560, untied embeddings, end-of-generation ids 1 and 130073, 2,516,756,480 parameters, bfloat16.

Held-out causal loss (lower is better), identical cases for both checkpoints:

Context	11,586	11,649
2K (499 cases, 50 packages)	0.94197	0.94435
8K (20 cases, 8 packages)	0.71110	0.71169
16K (6 cases, 3 packages)	0.76017	0.76103

Optimizer state is not published. Training data and provenance live in the scholzmx/sepalith dataset under campaign-20260915/. Released under Apache-2.0, like the base model.

Downloads last month: -; Downloads are not tracked for this model. How to track

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for scholzmx/sepalith-2b-cpt

Base model

openbmb/MiniCPM5-2B-Midtrain

Finetuned

(5)

this model