# ZeroPoint: Hierarchical Memory-Slot Network > A sequence-model architecture designed for zeroOS. The language model built on it is called **ZeroX**, and the training framework is **ZeroCG**. (Formerly named ZeroMem.) The document follows the structure of *Attention Is All You Need* (Vaswani et al., 2017); each section begins by noting which section of the paper it corresponds to, for easy cross-reading. > > Code: [`models/zeropoint.py`](../models/zeropoint.py) · > Self-checks: [`tests/test_zeropoint.py`](../tests/test_zeropoint.py) · > Comparison experiment: [`experiments/compare.py`](../experiments/compare.py) → [`compare.md`](../experiments/compare.md) · > Ablation experiment: [`experiments/ablation.py`](../experiments/ablation.py) → [`ablation.md`](../experiments/ablation.md) · > Exact recall: [`experiments/recall.py`](../experiments/recall.py) → [`recall.md`](../experiments/recall.md) · > Archived results: [`experiments/results/`](../experiments/results/) > > Status: **v0 (research prototype)**. --- ## Abstract The Transformer replaced recurrence with self-attention, gaining full parallelism at training time; the price is that, at inference time, generating each character requires looking back at all previous characters, and the KV cache grows linearly with text length. This is not a problem for GPU servers, but it is for **an operating-system kernel with no GPU and memory meted out page by page**. We propose **ZeroPoint**: replacing self-attention with a set of **memory slots that forget at different speeds**. As it reads each character, the model decides **which slot to write it into** and **from which slots to read**; the forgetting speeds of the slots are initialized according to a "register → cache → memory" storage hierarchy and are learned during training. ZeroPoint has two mathematically equivalent algorithms: at training time it uses a **chunked-parallel** form (chunk 32: one matrix multiplication per chunk, with a serial state carry between chunks), and at inference time it **recurs character by character**, with a state size that is fixed and independent of text length, and it needs no positional encoding. On character-level language modeling with **115,240** parameters and 1,558,395 characters, ZeroPoint is statistically indistinguishable from a same-size Transformer (1.5214 ± 0.0134 vs 1.5452 ± 0.0100; difference 0.0238 < 2× seed std = 0.0268 — **we do not claim it is better**). Its inference state is **1/128** of the latter's at the 1024th character (2,048 vs 262,144 floats), and at 4× the training length the loss rises by 0.037 nats/character (0.219 at 64×). ## 1 Introduction 〔Paper §1〕 The starting point of the paper's §1 is: recurrent networks "are inherently sequential" — step t has to wait for step t−1 to finish computing, so there is no parallelism within a single sample. Self-attention shortens the path between any two positions to O(1), so training can be fully parallel. But the other side, which the paper does not discuss, is **inference**: when generating the t-th character, self-attention has to compare against the Key/Value of the previous t−1 characters one by one. These Key/Values must be kept all along (the KV cache). For zeroOS: | | GPU server | zeroOS kernel | |---|---|---| | Memory | tens of GB of VRAM | kernel heap, allocated in 4KB pages | | Compute | tens of thousands of parallel units | one CPU core, x87 floating point | | Execution mode | process a large batch at once | process one character as the user types it | **What we want is the best of both: training almost as parallel as a Transformer (chunked parallel; sequential steps = length/32), inference carrying only a single fixed-size state like an RNN.** ## 2 Background 〔Paper §2〕 The paper's §2 mentions that End-to-End Memory Networks [34] read external memory with "recurrent attention". ZeroPoint is a close relative of this line of work and of the later "linear attention / recurrent models with decay" family; it did not appear out of thin air: | Prior work | Similarity | How ZeroPoint differs | |---|---|---| | Memory networks / Neural Turing Machine | soft-addressable read and write of memory slots | no need for step-by-step serial training; each slot has its own forgetting speed | | Linear attention (Katharopoulos 2020) | parallel / recurrent dual form | the state is M addressable slots, not a D×D outer-product matrix | | RetNet (Sun 2023) | decay λ^(t−u) realizes the dual form and needs no positional encoding | λ is learnable and initialized by storage hierarchy; writes compete for slots via softmax | | Mamba / state-space models | recurrent inference, fixed state | no specialized scan operator is needed; one matrix multiplication suffices to parallelize | What genuinely belongs to ZeroPoint is this combination: **multi-timescale learnable forgetting + write-time competition among slots + a write-strength gate + a parallel form that uses one matrix multiplication**, together with the starting point of "designing for a kernel with no GPU". ## 3 Model Architecture 〔Paper §3〕 ### 3.1 Stacking of Layers 〔Paper §3.1〕 Same as the paper's decoder: N identical layers are stacked, and each layer has two sub-layers — the **memory mixing layer** (replacing multi-head self-attention) and the **position-wise feed-forward network** — with a residual connection and LayerNorm around each sub-layer. There are two differences from the paper: - **LayerNorm is placed before the sub-layer (Pre-LN)**: `x + Sublayer(LayerNorm(x))`. The paper uses Post-LN `LayerNorm(x + Sublayer(x))`; later work such as GPT-2 found that Pre-LN trains more stably and does not depend much on learning-rate warmup. The control Transformer also uses Pre-LN, so both sides are consistent. - **Decoder only (autoregressive)**. The paper is an encoder-decoder for translation; zeroOS's tasks (continuation, intent recognition) do not need an encoder. ### 3.2 Memory Mixing Layer 〔Paper §3.2 Attention〕 #### 3.2.1 Reading and Writing 〔Corresponds to §3.2.1 Scaled Dot-Product Attention〕 The paper's attention is "take a query and compare it with all the keys". ZeroPoint does the reverse: **each character decides where to store itself at write time**, and at read time it deals only with the M slots, never directly with previous characters. Let the vector of the t-th character be `x_t`, the memory slots be `S[1..M]` (each a D-dimensional vector), and the retention rates be `λ[1..M]`: ``` write address w_t = softmax(x_t · Ww) × sigmoid(x_t · Wg) which slots to write × how strongly write value v_t = x_t · Wv update S[m] ← λ[m] · S[m] + w_t[m] · v_t all slots first forget at their own speed, then are written read address a_t = softmax(x_t · Wr) from which slots to read read out r_t = Σ_m a_t[m] · S[m] output LayerNorm(r_t) · Wo ``` - **Write address uses softmax (competition among slots)**: a character either goes into a short-term slot or into a long-term slot; the model has to decide whether "this character is worth remembering long-term". - **Write-strength gate**: uninformative characters such as spaces and newlines can be written almost not at all, without washing out existing memories. - **LayerNorm after reading**: long-term slots may have accumulated dozens of characters while short-term slots have only one or two, so their magnitudes differ greatly. This is the same motivation as dividing by √d_k in the paper's §3.2.1 — do not let the values fall into a range that softmax / subsequent layers handle poorly. - **The slots' initial state S0 is learnable** (not all zeros): otherwise the vectors read out at the beginning of the sequence are close to 0, and the subsequent LayerNorm would amplify the noise. #### 3.2.2 Multiple Heads 〔Corresponds to §3.2.2 Multi-Head Attention〕 As in the paper, the model is split into H heads, each of dimension D = d/H, each with its own set of read/write addresses and M slots; finally they are concatenated and multiplied by Wo. The paper's reasoning applies equally: a single head would average the different "what to remember" together. #### 3.2.3 Retention Rates and the Storage Hierarchy 〔ZeroPoint-specific〕 ZeroPoint's retention rates realize a hierarchical, learnable decay: ``` λ = 0.5 + 0.5 · sigmoid(θ) λ ∈ (0.5, 1), θ learnable ``` At initialization, the **half-lives** of the M slots of each head (how many characters it takes for old content to decay to half) are spread geometrically from 1.5 to 64: ``` slot 0 λ ≈ 0.63 half-life 1.5 characters ← like a register: remember "the character just now" slots 1~14 λ gradually larger ← like a cache: remember "this word" "this sentence" slot 15 λ ≈ 0.989 half-life 64 characters ← like memory: remember "what this passage is about" ``` The paper's §3.5 sinusoidal positional encoding has "wavelengths in a geometric progression from 2π to 10000·2π"; ZeroPoint's geometric spread of half-lives is the same idea: use a set of components at different scales to cover distances from very near to very far. The difference is that the paper encodes scale into the **input**, whereas ZeroPoint encodes scale into the **memory itself**. ### 3.3 Position-wise Feed-Forward Network 〔Paper §3.3〕 Same structure as the paper, `FFN(x) = act(x·W1 + b1)·W2 + b2`, with inner width 4d (the paper's 512 → 2048 is also 4x). The activation is GELU (the paper uses ReLU; after the GPT series, GELU is common), and the control group uses GELU too. ### 3.4 Embeddings and Output 〔Paper §3.4〕 Learnable character embeddings; before the output there is LayerNorm + a linear layer + softmax. We do not adopt the paper's "share weights between input and output embeddings", and neither does the control group. ### 3.5 Positional Information 〔Paper §3.5〕 The paper says: "the model contains no recurrence and no convolution, so positional information must be injected." **ZeroPoint does not need positional encoding.** It is essentially a (parallelizable) recurrent structure: in what the t-th character reads, information written u characters earlier has been multiplied by λ^(t−u). Different slots decay at different speeds, so "how much of a piece of information is left in which slots" itself tells the model how long ago it was. Ablation experiment (G) specifically tests this. ## 4 Why Memory Slots 〔Paper §4, Table 1〕 The paper's Table 1 compares three metrics: computation per layer, minimum number of sequential steps, and maximum path length between any two positions. We add the two columns that matter most for a kernel: computation per character at inference, and state to keep at inference. n = sequence length, d = width, M = number of slots per head (in the experiments M = 16, d = 64). | Layer type | Computation per layer (training) | Sequential steps | Maximum path | Inference: computation per character | Inference: state size | |---|---|---|---|---|---| | Self-attention | O(n²·d) | O(1) | O(1) | O(n·d), slower toward the end | O(n·d), grows with length | | Recurrent network | O(n·d²) | O(n) | O(n) | O(d²) | O(d) | | **ZeroPoint** | O(n·c·(M+d)) (chunked parallel, c = 32) | O(n/c) | **O(1)**, but the signal decays as λ^(t−u) | **O(d²)** | **O(M·d), fixed** | The key is that ZeroPoint has two equivalent algorithms: **Parallel form (for training)**. Expand the recurrence: ``` r_t = Σ_{u ≤ t} A[t,u] · v_u, A[t,u] = Σ_m a_t[m] · w_u[m] · λ[m]^(t−u) ``` Using `λ^(t−u) = λ^t · λ^(−u)`: ``` A = ( a ⊙ λ^t ) @ ( w ⊙ λ^(−u) )ᵀ then zero out the part where u > t (causal) ``` Only one matrix multiplication is needed, and it can process the whole sequence in parallel just like attention's `Q·Kᵀ`. But `λ^(−u)` is at most 2^u, and float32 overflows at u ≈ 120. So what is actually used is **chunked parallel**: the sequence is split into blocks of c = 32 characters; when entering a block, the slot state is `S_in`: ``` within block r_i = Σ_{j ≤ i} A[i,j] · v_j A as above, but i, j are within-block indices; λ^(−j) at most 2^31 across blocks r_i += (a_i ⊙ λ^(i+1)) @ S_in block end S_out = λ^c ⊙ S_in + (w ⊙ λ^(c−1−j))ᵀ @ v ``` Within a block it is matrix multiplication, and only an M × d state is passed between blocks. At training time the computation per layer is O(n·c·(M+d)), and **any sequence length works** (in the self-check, float32 runs 1000 characters without overflow, matching the recurrent form). **Recurrent form (for inference)**. Update `S` character by character; each layer stores only M·d floating-point numbers, regardless of how long the text is. The two forms agree: the self-check compares them bit by bit on **all 64 switch combinations** over 300 characters (crossing several blocks), with float64 error < 1e-8. There is also a fixed reference file `tests/golden/zeropoint_ref.json`; any future implementation (including a C version in the kernel) can check its answers against it. **A trained model can be put directly into the kernel in recurrent form**: no KV cache, a fixed M × d state per layer, no dynamic allocation. About "maximum path O(1)", we must be honest: information written at u can indeed be read at t in one step, but it has already been multiplied by λ^(t−u). Attention can "retrieve verbatim" a character from arbitrarily far away; ZeroPoint can only retrieve information **that has not yet been forgotten and has not been washed out by other writes**. ## 5 Training 〔Paper §5〕 | | Paper | ZeroPoint experiments | |---|---|---| | Data | WMT 2014 English-German, 4.5M sentence pairs, BPE 37000 | Python standard library source, 1.5M characters, character-level, vocabulary 96 | | Batch | about 25,000 tokens | 32 × 64 = 2048 characters | | Hardware | 8 P100s | one Mac's CPU (numpy, the in-house framework zerocg) | | Optimizer | Adam β=(0.9, 0.98), Eq. 3 (4000-step warmup then decay ∝ 1/√step) | AdamW β=(0.9, 0.999), weight decay 0.01; 100-step warmup then cosine decay to 10% | | Regularization | residual dropout 0.1, label smoothing 0.1 | none (small model, few training steps, no overfitting observed) | | Steps | 100K steps (12 hours) | 2000 steps (about 2 minutes per model) | ZeroPoint and the control group use **exactly the same** training settings and the same random seeds (which determine initialization and data order). ## 6 Results 〔Paper §6〕 ### 6.1 Comparison Experiment 〔Paper §6.1, Table 2〕 > Results: [`experiments/compare.md`](../experiments/compare.md) (auto-generated by `compare.py`). ### 6.2 Model Variants (Ablation) 〔Paper §6.2, Table 3〕 > Results: [`experiments/ablation.md`](../experiments/ablation.md) (auto-generated by `ablation.py`). ### 6.3 Length Extrapolation 〔Conjecture in Paper §3.5〕 The paper's reason for choosing sinusoidal positional encoding is that it "may allow the model to extrapolate to sequence lengths longer than the ones encountered during training", but no experiment was given. We train ZeroPoint and the sinusoidal-PE Transformer on 64 characters, then evaluate on 4,096-character text in segments (32 validation sequences × 3 seeds, same chunked-parallel algorithm for both models). - ZeroPoint: loss at 64–255 characters (4× the training length) is 1.4742 ± 0.0341, i.e. **+0.037** above the in-training-length value (0–63: 1.4372); at 1,024–4,095 characters (64×) it is 1.6932 ± 0.0305, i.e. **+0.219 ± 0.063**. **Extrapolation degrades gracefully but does degrade.** - The sinusoidal-PE Transformer jumps to about 4.64 once outside the training length (≈ ln 96 = 4.56, i.e. chance level). This is an 8-sequence × 3-seed measurement; the baseline checkpoints were not retained, so it could not be re-run with 32 sequences. > The earlier 8-sequence values (1.5550 / 1.6164 / 1.6410 / 1.7040) are noise-dominated and are superseded by the 32-sequence numbers above. Raw data: [`experiments/results/`](../experiments/results/). ### 6.4 Exact Recall 〔inconclusive; not an architecture comparison〕 > Results: [`experiments/recall.md`](../experiments/recall.md). The training configuration used by `recall.py` (constant lr 3e-3) traps both architectures at the "guess one of two candidates" local solution (loss ≈ ln 2). Diagnostics show that with a 1e-3 cosine schedule, the Transformer trained on N=2 alone reaches 100% in 8,000 steps. Until the experiment is re-run with the corrected learning rate, the 20,000-step table only shows that the *training setup* was wrong; **it says nothing about whether ZeroPoint or Transformer is better at exact recall**. ## 7 Conclusion and Limitations 〔Paper §7〕 **Limitations** (all honest): - **Very small scale**: tested only on about 115K parameters, 1.5M characters, and a single corpus; the conclusions cannot be extrapolated to large models. - **Exact recall**: with only M slots, information is compressed and mixed. It is a **hypothesis (not measured)** that this hurts verbatim copying of long-ago content; the existing recall experiment is inconclusive because of its learning-rate configuration. - **No encoder-decoder version**: the paper's core application is translation; ZeroPoint currently implements only the decoder. **Next steps**: 1. If exact recall is indeed weak: add a "sliding-window attention" layer (looking only at the most recent 32 characters, with the state still fixed) to make a hybrid structure; 2. Train on zeroOS's intent-recognition task; 3. Implement the recurrent form in C and put it in the kernel (x87 floating point) — this is why ZeroPoint exists.