license: mit
language:
- en
library_name: pytorch
pipeline_tag: text-generation
tags:
- recurrent
- cnn
- vla
- no-attention
datasets:
- HuggingFaceFW/fineweb
- HuggingFaceTB/smollm-corpus
- HuggingFaceFV/finevideo
VLA (recurrent CNN), v485
A language model with no attention and no positional encoding. Its memory is five 2-D registers of 512 × 7 × 7. Every tick, one convolutional network reads the current token together with the registers and writes the registers back. It is the text-trained core of a vision-language-action design aimed at a single chip running at 100 Hz or more with no external components: no off-chip memory, every weight stored on the chip. A CNN makes that possible because it is parameter-efficient: the same small kernels run at every grid position and every tick.
Code, functional spec and figures: github.com/siliconperception/vla
Use
git clone https://github.com/siliconperception/vla && cd vla && pip install -r requirements.txt
for p in "The colors of the rainbow are" \
"The best way to learn a new language is" \
"My grandmother's recipe for apple pie starts with" \
"When you plant a vegetable garden, the first thing to do is" \
"The history of the bicycle began in"; do
python vla.py --pretrained --generate --n 120 --prompt "$p" --device cuda
done
# prompt tuning: fit the starting registers (weights frozen) until the model predicts the prompt,
# then continue from them; prints the cold and the tuned continuation
python vla.py --pretrained --generate --prompt_tune 200 --n 120 --prompt "The colors of the rainbow are" --device cuda
--pretrained downloads model.safetensors and vla_args.json (the settings the weights were trained with)
and restores both. The weights are not a transformers model; load them through vla.py.
Model
| Parameters | 70.9 M: 21.6 M trained, 49.3 M frozen (token embedding and output layer from pythia-31m, a ResNet-50) |
| State | 5 registers × 512 × 7 × 7 = 125,440 values |
| Register windows | learned: 530 / 57 / 15 / 3.8 / 1 ticks (21 s, 2.3 s, 0.61 s, 0.15 s and one tick at 25 Hz) |
| State update | 1×1 mix (5,632 → 384), 8 × (3×3 conv, BatchNorm, ReLU) at 384, 3×3 project (384 → 512), cascade write |
| Readout | 3 decoders (text, voice, image), each a tree of three 3×3 convs and the frozen pythia-31m output layer |
| Tokenizer | pythia-31m BPE (50,304) plus a <NUL> token ("say nothing this tick") |
Training
- 10.18 M steps × 250 rows, one token per row per step, plain SGD.
- Up to step 9.73 M (run v472): learning rate cycling 1e-4 → 1e-2, the gradient stopped at the registers.
- Since then: learning rate 0.04, truncated backpropagation through time over 16 steps, the register projection widened to 3×3 (from its 1×1 weights, the same function at the switch), and learned register windows (from 512 / 64 / 16 / 4). The windows are trained by Adam from a forward-mode sensitivity of the registers to their rates, so their gradient sees every past step, not only the last 16.
- Data across its lineage: FineWeb-Edu (SmolLM corpus), FineWeb sample-10BT, and FineVideo and VALOR (speech, captions and frames); text-only FineWeb in its latest phase.
Evaluation (at the published step, 10,180,000)
| Score | |
|---|---|
| FineWeb held-out (100 documents, nats/token) | 4.527 |
| BabyLM 2026 dev (100 pieces, nats/token) | 4.518 |
| BLiMP (1,005 pairs from all 67 paradigms) | 70.1 % |
| ComVE (999 pairs, common-sense) | 66.3 % |
The BLiMP and ComVE pairs are scored the same way as for these reference models (each sentence read without context; the lower total loss wins):
| Model | Size | BLiMP | ComVE |
|---|---|---|---|
| pythia-14m | 14 M | 69.7 | 64.7 |
| pythia-31m | 31 M | 71.1 | 66.9 |
| babyllama-10m-2024 | trained on 10 M words | 70.0 | 63.7 |
| SmolLM2-135M | 135 M | 80.9 | 79.3 |
| this model | 21.6 M trained | 70.1 | 66.3 |
Limits
A small model: it writes fluent text that is often not factual or consistent. Not instruction-tuned, not safety filtered beyond its training corpora. The voice and image decoders are present but this checkpoint was last trained on text. Training is continuing; this card describes the step above.
License
MIT, Copyright (c) 2024-2026 Silicon Perception Inc. The license covers the software and these weights and grants no patent rights; see PATENTS. Frozen third-party weights inside: EleutherAI pythia-31m embedding and output layer (Apache-2.0), timm/resnet50.a1_in1k (Apache-2.0).