v485 step 10180000 (history squashed: earlier models removed)

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,104 @@
+---
+license: mit
+language:
+- en
+library_name: pytorch
+pipeline_tag: text-generation
+tags:
+- recurrent
+- cnn
+- vla
+- no-attention
+datasets:
+- HuggingFaceFW/fineweb
+- HuggingFaceTB/smollm-corpus
+- HuggingFaceFV/finevideo
+---
+
+# VLA (recurrent CNN), v485
+
+A language model with no attention and no positional encoding. Its memory is five 2-D registers of
+512 × 7 × 7. Every tick, one convolutional network reads the current token together with the registers and
+writes the registers back. It is the text-trained core of a vision-language-action design aimed at a single chip
+running at 100 Hz or more with no external components: no off-chip memory, every weight stored on the chip. A CNN
+makes that possible because it is parameter-efficient: the same small kernels run at every grid position and
+every tick.
+
+Code, functional spec and figures: [github.com/siliconperception/vla](https://github.com/siliconperception/vla)
+
+![One clock tick](https://raw.githubusercontent.com/siliconperception/vla/main/docs/figures/vla_v485.svg)
+
+## Use
+
+```bash
+git clone https://github.com/siliconperception/vla && cd vla && pip install -r requirements.txt
+for p in "The colors of the rainbow are" \
+         "The best way to learn a new language is" \
+         "My grandmother's recipe for apple pie starts with" \
+         "When you plant a vegetable garden, the first thing to do is" \
+         "The history of the bicycle began in"; do
+  python vla.py --pretrained --generate --n 120 --prompt "$p" --device cuda
+done
+
+# prompt tuning: fit the starting registers (weights frozen) until the model predicts the prompt,
+# then continue from them; prints the cold and the tuned continuation
+python vla.py --pretrained --generate --prompt_tune 200 --n 120 --prompt "The colors of the rainbow are" --device cuda
+```
+
+`--pretrained` downloads `model.safetensors` and `vla_args.json` (the settings the weights were trained with)
+and restores both. The weights are not a `transformers` model; load them through `vla.py`.
+
+## Model
+
+| | |
+|---|---|
+| Parameters | 70.9 M: 21.6 M trained, 49.3 M frozen (token embedding and output layer from pythia-31m, a ResNet-50) |
+| State | 5 registers × 512 × 7 × 7 = 125,440 values |
+| Register windows | learned: 530 / 57 / 15 / 3.8 / 1 ticks (21 s, 2.3 s, 0.61 s, 0.15 s and one tick at 25 Hz) |
+| State update | 1×1 mix (5,632 → 384), 8 × (3×3 conv, BatchNorm, ReLU) at 384, 3×3 project (384 → 512), cascade write |
+| Readout | 3 decoders (text, voice, image), each a tree of three 3×3 convs and the frozen pythia-31m output layer |
+| Tokenizer | pythia-31m BPE (50,304) plus a `<NUL>` token ("say nothing this tick") |
+
+## Training
+
+- 10.18 M steps × 250 rows, one token per row per step, plain SGD.
+- Up to step 9.73 M (run v472): learning rate cycling 1e-4 → 1e-2, the gradient stopped at the registers.
+- Since then: learning rate 0.04, truncated backpropagation through time over 16 steps, the register projection
+  widened to 3×3 (from its 1×1 weights, the same function at the switch), and learned register windows (from
+  512 / 64 / 16 / 4). The windows are trained by Adam from a forward-mode sensitivity of the registers to their
+  rates, so their gradient sees every past step, not only the last 16.
+- Data across its lineage: FineWeb-Edu (SmolLM corpus), FineWeb sample-10BT, and FineVideo and VALOR (speech,
+  captions and frames); text-only FineWeb in its latest phase.
+
+## Evaluation (at the published step, 10,180,000)
+
+| | Score |
+|---|---|
+| FineWeb held-out (100 documents, nats/token) | 4.527 |
+| BabyLM 2026 dev (100 pieces, nats/token) | 4.518 |
+| BLiMP (1,005 pairs from all 67 paradigms) | 70.1 % |
+| ComVE (999 pairs, common-sense) | 66.3 % |
+
+The BLiMP and ComVE pairs are scored the same way as for these reference models (each sentence read without
+context; the lower total loss wins):
+
+| Model | Size | BLiMP | ComVE |
+|---|---|---|---|
+| pythia-14m | 14 M | 69.7 | 64.7 |
+| pythia-31m | 31 M | 71.1 | 66.9 |
+| babyllama-10m-2024 | trained on 10 M words | 70.0 | 63.7 |
+| SmolLM2-135M | 135 M | 80.9 | 79.3 |
+| **this model** | 21.6 M trained | 70.1 | 66.3 |
+
+## Limits
+
+A small model: it writes fluent text that is often not factual or consistent. Not instruction-tuned, not safety
+filtered beyond its training corpora. The voice and image decoders are present but this checkpoint was last
+trained on text. Training is continuing; this card describes the step above.
+
+## License
+
+MIT, Copyright (c) 2024-2026 Silicon Perception Inc. The license covers the software and these weights and grants
+no patent rights; see [PATENTS](https://github.com/siliconperception/vla/blob/main/PATENTS). Frozen third-party
+weights inside: EleutherAI [pythia-31m](https://huggingface.co/EleutherAI/pythia-31m) embedding and output
+layer (Apache-2.0), [timm/resnet50.a1_in1k](https://huggingface.co/timm/resnet50.a1_in1k) (Apache-2.0).

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:41c390304a470328e064c9bf3005ed2a2f6926c338b7a88760029b0e4917df1e
+size 386889272

vla_args.json

@@ -0,0 +1,167 @@
+{
+ "saved_args": {
+  "context": 7,
+  "n_hidden": 512,
+  "c_context": 384,
+  "depth": 8,
+  "kernel": 3,
+  "learn_win": true,
+  "win_lr": 0.0003,
+  "linout_kernel": 3,
+  "tokenizer": "pythia31m",
+  "ctx_ema": "512,64,16,4,1",
+  "ctx_read_norm": 1.0,
+  "dec_tree_kernels": "3,3,3",
+  "dec_width": 256,
+  "bn_frozen": false,
+  "bn_momentum": 0.1,
+  "init": "zero",
+  "reset_scale": 0.0,
+  "continuous": "",
+  "image_encoder": "resnet50",
+  "audio_encoder": "logmel",
+  "fifo3": true,
+  "streams": 3,
+  "dataset": "smolfwedu",
+  "mix": "fineweb:1",
+  "full_crops": 0.5,
+  "imagine": null,
+  "imagine_window": 32,
+  "imagine_min": 10,
+  "imagine_size": 256,
+  "imagine_batch": 16,
+  "image": null,
+  "shards": 14,
+  "shard_offset": 1,
+  "streaming": false,
+  "shuffle": true,
+  "workers": 12,
+  "batch": 250,
+  "lr_max": 0.04,
+  "lr_min": 0.0004,
+  "lr_period": 20000,
+  "schedule": "coswarm",
+  "lr_final": 0.001,
+  "decay_steps": 200000,
+  "decay_start": null,
+  "eval_at_start": true,
+  "profile": 0,
+  "bptt": 16,
+  "lr_restart": false,
+  "momentum": 0.0,
+  "weight_decay": 0.0,
+  "run_steps": 3840000,
+  "seed": 2958979956,
+  "device": "cuda",
+  "load": "checkpoint/v485_v484_learnwin512_fw_bptt16.pt",
+  "save": "checkpoint/v485_v484_learnwin512_fw_bptt16.pt",
+  "checkpoint": 20000,
+  "log": "log/exp.v485_v484_learnwin512_fw_bptt16",
+  "monitor": 100,
+  "gen_interval": 2000,
+  "gen_score": "EleutherAI/pythia-70m",
+  "prompt": "",
+  "temperature": 1.0,
+  "gen_images": 8,
+  "gen_vis": false,
+  "sample_vis": false,
+  "eval_interval": 20000,
+  "eval_corpora": "fineweb,qaipt",
+  "eval_docs": 100,
+  "eval_bands": "1024,8192",
+  "eval_info": 100,
+  "eval_gen": 50,
+  "eval_gen_judge": "Qwen/Qwen2.5-1.5B",
+  "eval_gen_prompt": 128,
+  "eval_gen_len": 128,
+  "eval_cloze": 400,
+  "eval_pairs": 1000,
+  "eval_hold": 100,
+  "doc_log": 100,
+  "doc_log_len": 384,
+  "doc_keep": 10,
+  "doc_log_file": null,
+  "eval_recent": 20,
+  "weight_ema": "",
+  "use_ema": 0,
+  "doc_score": null,
+  "state_pca": 0,
+  "state_pca_stride": 16,
+  "doc_ablate": "",
+  "step_tokens": 0,
+  "doc_tokens": false,
+  "eval_wp": 0,
+  "eval_wp_len": 256,
+  "eval_qa": 100,
+  "eval_rare": 10,
+  "eval_reach": 100,
+  "eval_reach_period": 256,
+  "generate": false,
+  "n": 200,
+  "free_run": 0,
+  "eval_free": 10000,
+  "free_rows": 8,
+  "free_window": 2000,
+  "free_chunk": 128,
+  "free_save": null,
+  "prompt_tune": 0,
+  "prompt_tune_lr": 0.01,
+  "prompt_tune_cap": "1",
+  "prompt_tune_batch": 10,
+  "prompt_warm": 256,
+  "prompt_tune_save": null,
+  "clock_hz": 25.0,
+  "ctx_tau": "20.48,2.56,0.64,0.16",
+  "vision_hz": 2.0,
+  "max_lag": 2.0,
+  "nul_weight": 1.0,
+  "clip": 0.0,
+  "loss_norm": "words",
+  "inner_voice": false,
+  "min_p": 0.009220976663785505,
+  "caption_feed": 0.0,
+  "text_feed": 0.0,
+  "top_k": 0,
+  "audio_hz": 25.0,
+  "speech_delay": 0.12,
+  "speech_align": true,
+  "align_min": 0.5,
+  "write_surprise": null,
+  "voice_drop": 0.0,
+  "fluency_target": 100.0,
+  "fluency_gain": 0.5,
+  "video": null,
+  "video_rate": 4.0,
+  "video_sound": "mimi",
+  "video_seconds": 0.0,
+  "video_save": null,
+  "teach": 0,
+  "teach_ask": "data",
+  "teach_teacher": "Qwen/Qwen2.5-1.5B-Instruct",
+  "teach_lr": 0.001,
+  "teach_steps": 1,
+  "teach_control": 100,
+  "teach_skip": 1000,
+  "teach_para": true,
+  "teach_json": null,
+  "eval": false,
+  "eval_json": null,
+  "vis": false,
+  "delay": 0.1,
+  "cmap": "viridis",
+  "push": false,
+  "hub_repo": "siliconperception/vla",
+  "private": false,
+  "push_dir": null,
+  "card": null,
+  "hub_tag": null,
+  "pretrained": null,
+  "revision": null,
+  "verbose": false
+ },
+ "step": 10180000,
+ "examples": 2441355,
+ "examples_by": {
+  "fineweb": 2298981
+ }
+}