jgalego commited on
Commit
95230dc
·
verified ·
1 Parent(s): 5bb5aee

Upload folder using huggingface_hub

Browse files
Files changed (4) hide show
  1. README.md +40 -2
  2. assets/checks.png +0 -0
  3. assets/layers.png +0 -0
  4. assets/logo.svg +1 -0
README.md CHANGED
@@ -12,6 +12,8 @@ tags:
12
 
13
  # olmputer-tiny
14
 
 
 
15
  A RISC-V computer (RV32IM + Zba) compiled into the layers of a stock `OlmoHybridForCausalLM`. No weight is trained: a compiler computes all of them. The code ROM, data ROM, RAM and registers live in the recurrent state of the Gated DeltaNet layers, and the model runs a program by greedy decoding: every token it generates is the next token of the CPU's execution trace.
16
 
17
  This is the standalone half of [Olmputer](https://huggingface.co/jgalego/olmputer), which puts the same layers in front of [allenai/Olmo-Hybrid-Instruct-DPO-7B](https://huggingface.co/allenai/Olmo-Hybrid-Instruct-DPO-7B) so that OLMo's `calc` tool calls run on the CPU inside its weights. It ports [Qwenputer](https://github.com/divagr18/qwenputer), which does this with Qwen3.5-4B, to OLMo-Hybrid's head shapes. The compiler is the `chipformer` package of [JGalego/olmputer](https://github.com/JGalego/olmputer).
@@ -29,6 +31,23 @@ This is the standalone half of [Olmputer](https://huggingface.co/jgalego/olmpute
29
  | Parameters | 3.13B, almost all zero |
30
  | Vocabulary | the trace tokens (values in 15-bit chunks, jumps, stores, putchar, ...) plus PAD |
31
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
32
  Qwen3.5-4B's heads are 128 × 128, two value heads per key head; OLMo's are 96 × 192, one each. The changes that keep every read exact:
33
 
34
  - A state row's ballast sits in 3 of its 192 columns (Qwen: 2 of 128), so the gated output norm divides by the same power of 2.
@@ -43,12 +62,31 @@ For each program, the machine's trace from a reference RISC-V emulator is the ta
43
  - the residual stream after the chip layers equals the circuit's IR evaluator on every dimension at every position (bf16; traces up to 60k tokens);
44
  - free-running greedy `generate` from the program's start reproduces the reference trace token for token.
45
 
46
- Pending.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
47
 
48
  ## Usage
49
 
50
  ```bash
51
- pip install "chipformer @ https://github.com/JGalego/olmputer/archive/f1599461ea6a693771ff84ea19e2e48551f72034.tar.gz#subdirectory=experimental/riscv" torch==2.14.1 transformers==5.19.0
52
  ```
53
 
54
  ```python
 
12
 
13
  # olmputer-tiny
14
 
15
+ <img src="assets/logo.svg" alt="Olmputer logo: a pixel computer" width="384">
16
+
17
  A RISC-V computer (RV32IM + Zba) compiled into the layers of a stock `OlmoHybridForCausalLM`. No weight is trained: a compiler computes all of them. The code ROM, data ROM, RAM and registers live in the recurrent state of the Gated DeltaNet layers, and the model runs a program by greedy decoding: every token it generates is the next token of the CPU's execution trace.
18
 
19
  This is the standalone half of [Olmputer](https://huggingface.co/jgalego/olmputer), which puts the same layers in front of [allenai/Olmo-Hybrid-Instruct-DPO-7B](https://huggingface.co/allenai/Olmo-Hybrid-Instruct-DPO-7B) so that OLMo's `calc` tool calls run on the CPU inside its weights. It ports [Qwenputer](https://github.com/divagr18/qwenputer), which does this with Qwen3.5-4B, to OLMo-Hybrid's head shapes. The compiler is the `chipformer` package of [JGalego/olmputer](https://github.com/JGalego/olmputer).
 
31
  | Parameters | 3.13B, almost all zero |
32
  | Vocabulary | the trace tokens (values in 15-bit chunks, jumps, stores, putchar, ...) plus PAD |
33
 
34
+ Each generated token runs once through the 11 layers, and each layer does one part of executing the instruction:
35
+
36
+ ```mermaid
37
+ flowchart TB
38
+ IN["a trace token in"] --> L1["L1: the PC anchor and counters (record position, commits since the last jump)"]
39
+ L1 --> L2["L2: the PC word and the link address"]
40
+ L2 --> L3["L3: fetch: the code ROM, 23 heads of 2,944 words each"]
41
+ L3 --> L4["L4: decode; read 3 register ports"]
42
+ L4 --> L5["L5: the bignum digit file; adder and multiplier stages"]
43
+ L5 --> L6["L6 to L8: the RAM, 86 heads of 96 words; loads, stores, compares"]
44
+ L6 --> L9["L9: the data ROM; divider digits"]
45
+ L9 --> L10["L10 to L11: pick the result and write the next token's code"]
46
+ L10 --> OUT["the next trace token"]
47
+ ```
48
+
49
+ ![Heads per chip layer, by the memory they hold](assets/layers.png)
50
+
51
  Qwen3.5-4B's heads are 128 × 128, two value heads per key head; OLMo's are 96 × 192, one each. The changes that keep every read exact:
52
 
53
  - A state row's ballast sits in 3 of its 192 columns (Qwen: 2 of 128), so the gated output norm divides by the same power of 2.
 
62
  - the residual stream after the chip layers equals the circuit's IR evaluator on every dimension at every position (bf16; traces up to 60k tokens);
63
  - free-running greedy `generate` from the program's start reproduces the reference trace token for token.
64
 
65
+ ![Trace tokens generated free-running, per run](assets/checks.png)
66
+
67
+ | Run | Kernels | Programs | Exact | Trace tokens |
68
+ |---|---|---|---|---|
69
+ | c-sieve-gmpmul | torch | 2 | 2 | 66,825 |
70
+ | short-fla | fla | 55 | 55 | 26,677 |
71
+ | short-torch | torch | 55 | 55 | 26,677 |
72
+
73
+
74
+ A run, from an ELF to the HALT token:
75
+
76
+ ```mermaid
77
+ flowchart LR
78
+ ELF["program.elf"] --> SNAP["snapshot: code ROM, data ROM, RAM, registers, PC"]
79
+ SNAP -->|"written into the Gated DeltaNet states"| M["olmo-tiny"]
80
+ GO["GO"] --> M
81
+ M -->|"greedy: the next trace token"| T["trace token"]
82
+ T -->|"fed back in"| M
83
+ T -->|"HALT"| END["done; putchar output is in the BYTE tokens"]
84
+ ```
85
 
86
  ## Usage
87
 
88
  ```bash
89
+ pip install "chipformer @ https://github.com/JGalego/olmputer/archive/ff1f044bbd7aab21aa2e0da0dfe0a90fcf170be4.tar.gz#subdirectory=experimental/riscv" torch==2.14.1 transformers==5.19.0
90
  ```
91
 
92
  ```python
assets/checks.png ADDED
assets/layers.png ADDED
assets/logo.svg ADDED