sparticle62 commited on
Commit
56c40ce
·
verified ·
1 Parent(s): a16b755

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +28 -0
README.md CHANGED
@@ -1,3 +1,31 @@
1
  ---
2
  license: apache-2.0
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: apache-2.0
3
  ---
4
+ # PSSA: a plastic state space architecture
5
+
6
+ PSSA is a recurrent sequence model written from scratch in Rust. No PyTorch and no autograd framework: the forward pass, the backward pass and the optimizer are all hand written, and every batched or GPU path is checked against a scalar CPU reference twin (max gradient difference 2.98e-8).
7
+
8
+ Code, training chain and reproduction scripts: https://github.com/Sparticle62ops/pssa
9
+
10
+ ## Results
11
+
12
+ PSSA was trained against a parameter matched transformer baseline on cleaned WikiText-103, with the same corpus, the same token budget and the same schedule. The scores below are on a held out slice of 198,939 unseen tokens.
13
+
14
+ | model | cross entropy | perplexity | next token accuracy |
15
+ | --- | --- | --- | --- |
16
+ | PSSA | 3.9971 | 54.44 | 24.08% |
17
+ | transformer baseline | 4.4285 | 83.81 | 18.02% |
18
+
19
+ Generating 200 tokens on CPU takes 226 ms with PSSA against 2735 ms with the transformer, about 12x faster. Training runs at roughly 900 tokens per second on a single GPU through the CUDA path, against about 210 on CPU.
20
+
21
+ ## What is in it
22
+
23
+ - an affine state space scan as the sequence mixer, no attention
24
+ - a plastic memory bank with ordered retrieval and protected slots
25
+ - refractory gating that suppresses repeated writes
26
+ - a scalar CPU reference implementation kept as a correctness twin for every optimized path
27
+
28
+ ## Status
29
+
30
+ Research code under active development. Weights are not published in this repository yet; the GitHub repository holds the training chain, the evaluation harness and the scripts to reproduce the numbers above.
31
+