| --- |
| license: apache-2.0 |
| thumbnail: https://huggingface.co/sparticle62/pssa/resolve/main/logo.png |
| --- |
| # PSSA: a plastic state space architecture |
|
|
| PSSA is a recurrent sequence model written from scratch in Rust. No PyTorch and no autograd framework: the forward pass, the backward pass and the optimizer are all hand written, and every batched or GPU path is checked against a scalar CPU reference twin (max gradient difference 2.98e-8). |
|
|
| Code, training chain and reproduction scripts: https://github.com/Sparticle62ops/pssa |
|
|
| ## Results |
|
|
| PSSA was trained against a parameter matched transformer baseline on cleaned WikiText-103, with the same corpus, the same token budget and the same schedule. The scores below are on a held out slice of 198,939 unseen tokens. |
|
|
| | model | cross entropy | perplexity | next token accuracy | |
| | --- | --- | --- | --- | |
| | PSSA | 3.9971 | 54.44 | 24.08% | |
| | transformer baseline | 4.4285 | 83.81 | 18.02% | |
|
|
| Generating 200 tokens on CPU takes 226 ms with PSSA against 2735 ms with the transformer, about 12x faster. Training runs at roughly 900 tokens per second on a single GPU through the CUDA path, against about 210 on CPU. |
|
|
| ## What is in it |
|
|
| - an affine state space scan as the sequence mixer, no attention |
| - a plastic memory bank with ordered retrieval and protected slots |
| - refractory gating that suppresses repeated writes |
| - a scalar CPU reference implementation kept as a correctness twin for every optimized path |
|
|
| ## Weights and checkpoints |
|
|
| - [NSC64/pssa-100m](https://huggingface.co/NSC64/pssa-100m): a 109M-parameter PyTorch port of this architecture, trained from scratch by NSC64 on 4B tokens of TinyStories and FineWeb-Edu. Ships bf16 safetensors weights, a 16k byte-level BPE tokenizer and a self-contained loader (plain scan on CPU, fused Triton kernel on CUDA). |
| - [sparticle62/pssa-checkpoints-run2](https://huggingface.co/sparticle62/pssa-checkpoints-run2): training checkpoints from the 105M run in progress (raw .pssa resume files, not a packaged model). |
|
|
| ## Status |
|
|
| Research code under active development. The links above hold the published 109M PyTorch port and the in-training 105M checkpoints; the GitHub repository holds the training chain, the evaluation harness and the scripts to reproduce the numbers above. |
|
|
|
|
|
|
|
|
