PSSA 105M distilled from Qwen3-8B

This repo holds checkpoints from a distillation run that continues the 105M PSSA model in sparticle62/pssa-checkpoints-105m with Qwen3-8B as the teacher.

Run result

5,000 local updates completed, exit code 0
final global step: 35,727
final objective EMA: 2.758 at temperature 2.0 (mixed CE and KL, see Method)
throughput: about 1.18 updates/s on one RTX PRO 6000 Blackwell
wall clock: about 71 minutes
10 checkpoints, every 500 local updates

Base model

checkpoint: ck256.pssa from sparticle62/pssa-checkpoints-105m
global step at start: 30,727
config: vocab 2048, latent 3584, state 16, memory key 32, memory capacity 512, chunk 64
parameters: about 107M

Teacher

Qwen/Qwen3-8B, the dense Qwen3 model. The newer hybrid 3.5, 3.6 and 3.8 models are not used.

Corpus

fineweb-edu, 60 percent
open-web-math, 20 percent
codeparrot-clean, 20 percent
about 2.0 GB of UTF-8 text, interleaved so each block carries all three sources

Method

The teacher and student use different byte-level BPE vocabularies, about 151.6k text tokens for the teacher and 2,048 for the student. The trainer forces a student token boundary at every teacher token boundary, then projects the teacher next-token distribution onto the student vocabulary by mapping each teacher token to the first student sub-token of its byte string. Rows inside a teacher token use a one-hot CE target. The loss is 0.5 * CE + 0.5 * KL at temperature 2.0. Optimizer: AdamW, LR 1e-4, gradient clip 1.0, no warmup. The run is 5,000 local updates of 64 tokens.

This is an approximate cross-tokenizer projection, not an exact KL over a shared vocabulary. Each checkpoint embeds the 2,048-token student tokenizer from ck256.pssa.

Evaluation

Held-out slice: the last 1 MB of the corpus (fineweb-edu text), 9,932 tokens, scored with pssa score and the embedded tokenizer.

model	cross-entropy	perplexity	next-token accuracy
base `ck256.pssa` (step 30,727)	4.651	104.7	20.4%
final `ck_035727.pssa` (step 35,727)	4.634	103.0	19.8%

The 5,000-update pass moved held-out cross-entropy by 0.016 nats. That is a small change and within run-to-run noise for a slice this size. The checkpoints load and generate, but the 105M student is still weak on open-domain text and samples are not coherent. More updates at a lower learning rate are needed for a clear gain.

Checkpoints

Files are named ck_<global_step>.pssa. The global step is the base step plus the number of local updates. Each file is a V7 PSSA checkpoint, about 1.69 GB.

file	global step	local updates
`ck_031227.pssa`	31,227	500
`ck_031727.pssa`	31,727	1,000
`ck_032227.pssa`	32,227	1,500
`ck_032727.pssa`	32,727	2,000
`ck_033227.pssa`	33,227	2,500
`ck_033727.pssa`	33,727	3,000
`ck_034227.pssa`	34,227	3,500
`ck_034727.pssa`	34,727	4,000
`ck_035227.pssa`	35,227	4,500
`ck_035727.pssa`	35,727	5,000

How to use

Checkpoints are V7 and embed the 2,048-token tokenizer, so there is no separate tokenizer file. Build the CLI from https://github.com/Sparticle62ops/pssa, then:

# sample from the final checkpoint
./target/release/pssa generate "Once upon a time" -m ck_035727.pssa --max-new-tokens 64

# score a held-out text file
./target/release/pssa score heldout.txt -m ck_035727.pssa --max-tokens 10000

pssa generate restores the tokenizer from the checkpoint. pssa score prints one JSON object with cross-entropy, perplexity, and next-token accuracy.

Code and license

PSSA: https://github.com/Sparticle62ops/pssa (GPL-3.0)

Qwen3-8B: https://huggingface.co/Qwen/Qwen3-8B (Apache-2.0)

Downloads last month: -; Downloads are not tracked for this model. How to track