jgalego commited on
Commit
fa0f2cf
·
verified ·
1 Parent(s): 8a4ac0c

Upload folder using huggingface_hub

Browse files
Files changed (5) hide show
  1. README.md +30 -49
  2. assets/conversation.png +0 -0
  3. assets/eval.png +0 -0
  4. assets/layers.png +0 -0
  5. example.py +7 -7
README.md CHANGED
@@ -15,50 +15,42 @@ tags:
15
 
16
  <img src="assets/logo.svg" alt="Olmputer logo: a pixel computer" width="384">
17
 
18
- [allenai/Olmo-Hybrid-Instruct-DPO-7B](https://huggingface.co/allenai/Olmo-Hybrid-Instruct-DPO-7B) with a RISC-V computer (RV32IM + Zba) compiled into 11 extra layers in front of its own. When OLMo calls its `calc` tool, the CPU runs the calculation inside the forward passes and writes the exact result back as OLMo's own tokens. No weight is trained: a compiler computes the chip layers, and OLMo's weights change only by zero-padding to the wider hidden size and a rescale of its norm gains.
19
 
20
- It ports [Qwenputer](https://github.com/divagr18/qwenputer) (the same machine inside Qwen3.5-4B) to OLMo-Hybrid. The compiler is the `chipformer` package of [JGalego/olmputer](https://github.com/JGalego/olmputer); [olmputer-tiny](https://huggingface.co/jgalego/olmputer-tiny) is the chip layers alone.
 
 
21
 
22
  > [!NOTE]
23
- > Built on Ai2's [OLMo-Hybrid](https://huggingface.co/allenai/Olmo-Hybrid-Instruct-DPO-7B); not affiliated with or endorsed by Ai2.
24
 
25
  ## How it works
26
 
27
- ![A calc call played out: OLMo writes the call, the CPU runs it, the result comes back as OLMo's own tokens](assets/call.gif)
 
 
28
 
29
- - **The machine.** 256 KB of code ROM, 128 KB of read-only data and 32 KB of RAM, all held in the Gated DeltaNet layers' recurrent state; about 1.2 trace tokens per instruction. The firmware is a bignum calculator: `+ - * / // % **` and `gcd` on integers of up to 62 digits, exact fractions and decimals.
30
- - **The bridge.** The text OLMo writes inside `<function_calls>…</function_calls>` is copied into a buffer the firmware reads. At `</function_calls>` the circuit takes over: it runs the call and writes the environment turn, `<|im_end|>\n<|im_start|>environment\n…<|im_end|>\n<|im_start|>assistant\n`, then hands the stream back to OLMo.
31
- - **Numbers as OLMo's tokens.** OLMo splits numbers into groups of up to three digits. Read one digit per token, it garbles them, so the firmware prints numbers as those groups. With the result of a product echoed as OLMo's own digit groups, OLMo's answer held the exact value 33/40 times; one digit per token, 0/40.
32
 
33
- ![Base OLMo reading a calc result](assets/digits.png)
34
- - **Chip tokens are transparent.** The CPU's trace tokens run through the chip layers only; OLMo's layers, caches and positions see the text alone. OLMo's full-attention layers have no output gate to mute chip tokens with, so this decoding is required: plain `generate()` does not run the machine.
35
 
36
- ### A calc call
37
 
38
- ![One calc call in four steps: the call, the hidden trace, the echo, the value](assets/call.png)
39
 
40
- ![One conversation, each token filled by who wrote it](assets/conversation.png)
41
 
42
  ### Inside the chip layers
43
 
44
- Each trace token runs once through the 11 chip layers. Their Gated DeltaNet heads hold the machine's memory, a layer's worth at a time:
45
 
46
- ![Heads per chip layer, by the memory they hold](assets/layers.png)
47
 
48
  ## Checks
49
 
50
- **Text parity.** Next-token distributions of the merged model against base OLMo on held-out prompts, against the base's own noise floor (its sdpa and eager attention), and greedy continuations.
51
-
52
- ![KL per prompt against the noise floor](assets/parity.png)
53
-
54
- | | |
55
- |---|---|
56
- | KL(base ‖ merged), mean | 5.87e-04 |
57
- | KL(base sdpa ‖ base eager), mean (noise floor) | 5.66e-04 |
58
- | Greedy continuations identical to base | 12/16 |
59
- | Base sdpa vs base eager, greedy identical (noise floor) | 11/16 |
60
 
61
- **The machine in the merged weights.** Programs run through the merged model (chip tokens through the chip layers, echoed text through every layer); every token the circuit owns must be the reference emulator's.
62
 
63
  | Program | Positions | Circuit-owned | Wrong |
64
  |---|---|---|---|
@@ -68,28 +60,17 @@ Each trace token runs once through the 11 chip layers. Their Gated DeltaNet head
68
  | rv32um-div | 142 | 142 | 0 |
69
 
70
 
71
- **Conversations.** OLMo answers questions with the calc tool; every circuit token is checked against the reference as it is generated, and the whole stream is then replayed through the circuit.
72
-
73
- 24/24 conversations clean. With the chip layers on chipformer's Triton executor (the circuit's arithmetic in fp32, checked equal to the circuit on every value), the same 24 conversations are token for token identical and take 6.5 s each instead of 157 s through the HF modules.
74
 
75
  ## Arithmetic
76
 
77
- Greedy decoding, 40 problems per set, scored by exact rational match of the final boxed answer: Olmputer, base OLMo whose calc calls a host answers (the same firmware on a reference machine, so its responses are byte for byte Olmputer's), and base OLMo without tools. The sets are a seeded synthetic set and four modules of DeepMind's mathematics dataset (extrapolation split).
78
-
79
- ![Exact answers per set](assets/eval.png)
80
-
81
- | Set | Olmputer | OLMo + host calc tool | OLMo, no tool |
82
- |---|---|---|---|
83
- | arith | 40/40 | 40/40 | 25/40 |
84
- | dmx_mul_big | 40/40 | 40/40 | 14/40 |
85
- | dmx_div_big | 37/40 | 37/40 | 11/40 |
86
- | dmx_mul_div_multiple_longer | 24/40 | 22/40 | 16/40 |
87
- | dmx_add_or_sub_big | 38/40 | 38/40 | 21/40 |
88
 
 
89
 
90
  ## Usage
91
 
92
- [`example.py`](example.py) asks Olmputer a question with the calc tool (the first argument) and prints the conversation; every token the machine writes is checked against a reference RISC-V emulator as it is generated, and the stream is replayed through the circuit at the end. It is a [uv script](https://docs.astral.sh/uv/guides/scripts/) that declares its own dependencies:
93
 
94
  ```bash
95
  uv run https://huggingface.co/jgalego/olmputer/resolve/main/example.py "What is 2**64 - 1?"
@@ -105,14 +86,14 @@ uv run https://huggingface.co/jgalego/olmputer/resolve/main/example.py "What is
105
  # "transformers==5.19.0",
106
  # ]
107
  # ///
108
- """Ask Olmputer a question: OLMo calls its calc tool and the RISC-V computer in its weights runs it.
109
-
110
- OLMo writes the call; at </function_calls> the chip layers run the calc firmware and write the
111
- environment turn with the exact value; OLMo reads it and answers. The CPU's trace tokens stay in
112
- the chip layers (chipformer's RvChat; plain generate() does not run the machine), on chipformer's
113
- Triton executor when there is a GPU. Every token the machine writes is checked against a
114
- reference RISC-V emulator as it is generated, and the whole stream is replayed through the circuit
115
- at the end. The question is the first argument.
116
  """
117
 
118
  import sys
@@ -147,4 +128,4 @@ print(f"\n{RESULT['text_tokens']} text tokens, {RESULT['chip_tokens']} trace tok
147
  print("replayed through the circuit:", replay(BOT, IDS, RESULT["tokens"], FIRMWARE))
148
  ```
149
 
150
- The checkpoint is about 32 GB in bf16, so it needs a GPU with 40 GB or more. Decoding goes through chipformer's `RvChat`, which keeps the CPU's trace tokens in the chip layers (on its Triton executor on a GPU): plain `generate()` does not run the machine.
 
15
 
16
  <img src="assets/logo.svg" alt="Olmputer logo: a pixel computer" width="384">
17
 
18
+ Olmputer is [allenai/Olmo-Hybrid-Instruct-DPO-7B](https://huggingface.co/allenai/Olmo-Hybrid-Instruct-DPO-7B) with a small RISC-V computer built into its weights. When OLMo calls its `calc` tool, the computer runs the calculation inside the forward pass and writes the exact result back into the conversation, in the same tokens OLMo would use itself.
19
 
20
+ Nothing here is trained. A compiler works out the weights of 11 new layers that implement the CPU (RV32IM with the Zba extension), and they sit in front of OLMo's own 32 layers. OLMo's weights stay as they were, apart from zero-padding to the wider hidden size and a rescale of the norm gains to make up for it.
21
+
22
+ The idea and the CPU come from [Qwenputer](https://github.com/divagr18/qwenputer), which does the same inside Qwen3.5-4B. Olmputer ports it to OLMo-Hybrid. The compiler is the `chipformer` package in [JGalego/olmputer](https://github.com/JGalego/olmputer), and [olmputer-tiny](https://huggingface.co/jgalego/olmputer-tiny) is the computer on its own, without OLMo.
23
 
24
  > [!NOTE]
25
+ > Built on Ai2's [OLMo-Hybrid](https://huggingface.co/allenai/Olmo-Hybrid-Instruct-DPO-7B). Not affiliated with or endorsed by Ai2.
26
 
27
  ## How it works
28
 
29
+ ![A calc call played out: OLMo writes the call, the CPU runs it, and the result comes back as OLMo's own tokens](assets/call.gif)
30
+
31
+ The computer's memory lives in the recurrent state of the new layers' Gated DeltaNet heads. It has 256 KB of code ROM, 128 KB of read-only data and 32 KB of RAM. The program it runs is a bignum calculator that handles `+ - * / // % **` and `gcd` on integers of up to 62 digits, as well as exact fractions and decimals.
32
 
33
+ The CPU works by writing an execution trace, one token at a time, at about 1.2 tokens per instruction. These trace tokens only pass through the chip layers. OLMo's own layers, caches and positions never see them, so from OLMo's side the computer is invisible. This takes a custom decoding loop. OLMo's full-attention layers have no output gate that could silence the trace tokens, so plain `generate()` does not run the computer (see [Usage](#usage)).
 
 
34
 
35
+ When OLMo writes a tool call, the text between `<function_calls>` and `</function_calls>` is copied into a buffer that the calculator reads. At `</function_calls>` the circuit takes over. It runs the calculation, writes the whole environment turn with the result, and then hands the conversation back to OLMo for its answer.
 
36
 
37
+ ![A calc conversation as a dialogue: the user's question, OLMo's call, the CPU's environment turn with the result split into OLMo's digit tokens, and OLMo's answer](assets/conversation.png)
38
 
39
+ The CPU writes numbers the way OLMo's tokenizer does, in groups of up to three digits. This matters more than you might expect. In a test where base OLMo asked calc for a product and then had to give the result in its answer, it got the exact value 33 times out of 40 when the number came in its own digit groups, and 0 times when it came one digit per token. The conversation above took 2,717 trace tokens, none of which OLMo saw.
40
 
41
+ ![One calc call in four steps: the call, the hidden trace, the echo and the value](assets/call.png)
42
 
43
  ### Inside the chip layers
44
 
45
+ Every trace token passes once through the 11 chip layers. Each row below is one layer and each cell one of its 30 heads, tinted by the part of the machine's memory it holds. The last two layers hold no memory, and only their MLPs do any work.
46
 
47
+ ![The 11 chip layers as rows of 30 heads, each head tinted by the memory it holds](assets/layers.png)
48
 
49
  ## Checks
50
 
51
+ **Text parity.** Adding the chip layers should not change what OLMo says. To check this, the merged model's next-token distributions are compared with base OLMo's on held-out prompts. The yardstick is base OLMo against itself with two attention implementations (sdpa and eager), which differ only by numerical noise. The merge's mean KL divergence from base OLMo is 5.87e-04, against a noise floor of 5.66e-04. Greedy continuations match base OLMo on 12 of 16 prompts, while the two base implementations agree on 11.
 
 
 
 
 
 
 
 
 
52
 
53
+ **The computer inside the merged model.** A few programs run through the full merged model, with trace tokens going through the chip layers and the echoed text through every layer. Every token the circuit owns has to match the reference emulator's.
54
 
55
  | Program | Positions | Circuit-owned | Wrong |
56
  |---|---|---|---|
 
60
  | rv32um-div | 142 | 142 | 0 |
61
 
62
 
63
+ **Conversations.** OLMo answers questions using the calc tool. Each token the computer writes is checked against the reference emulator as it is generated, and the whole conversation is then replayed through the circuit. 24 of 24 conversations came out clean. There is also a Triton executor in chipformer, which does the chip layers' arithmetic in fp32 and has been checked to match the circuit on every value. On it, the same conversations come out token for token identical, and each takes 6.5 s instead of 157 s.
 
 
64
 
65
  ## Arithmetic
66
 
67
+ Three setups answer the same problems. The first is Olmputer. The second is base OLMo with a calc tool that the host answers by running the same calculator on a reference emulator, so its replies are byte for byte the ones Olmputer writes. The third is base OLMo with no tool at all. There are five sets of 40 problems. `arith` is a seeded synthetic set, and the other four are modules from the extrapolation split of DeepMind's mathematics dataset. Decoding is greedy, and an answer counts only if the final boxed value is exactly the right number.
 
 
 
 
 
 
 
 
 
 
68
 
69
+ ![Exact answers out of 40 per set, for Olmputer, base OLMo with a host calc tool and base OLMo without one](assets/eval.png)
70
 
71
  ## Usage
72
 
73
+ [`example.py`](example.py) asks Olmputer a question, given as its first argument, and prints the conversation. Every token the computer writes is checked against a reference RISC-V emulator as it is generated, and at the end the whole conversation is replayed through the circuit. It is a [uv script](https://docs.astral.sh/uv/guides/scripts/), so it brings its own dependencies:
74
 
75
  ```bash
76
  uv run https://huggingface.co/jgalego/olmputer/resolve/main/example.py "What is 2**64 - 1?"
 
86
  # "transformers==5.19.0",
87
  # ]
88
  # ///
89
+ """Ask Olmputer a question and print the conversation.
90
+
91
+ OLMo writes a calc call. At </function_calls> the chip layers run the calculator firmware and
92
+ write the environment turn with the exact value, and OLMo reads it and answers. The CPU's trace
93
+ tokens stay inside the chip layers, so this decodes with chipformer's RvChat rather than
94
+ generate(), on its Triton executor when there is a GPU. Every token the computer writes is checked
95
+ against a reference RISC-V emulator as it is generated, and the whole conversation is replayed
96
+ through the circuit at the end. The question is the first argument.
97
  """
98
 
99
  import sys
 
128
  print("replayed through the circuit:", replay(BOT, IDS, RESULT["tokens"], FIRMWARE))
129
  ```
130
 
131
+ The checkpoint is about 32 GB in bf16, so you need a GPU with at least 40 GB of memory. Decoding goes through chipformer's `RvChat`, which keeps the trace tokens inside the chip layers and runs them on the Triton executor when there is a GPU.
assets/conversation.png CHANGED
assets/eval.png CHANGED
assets/layers.png CHANGED
example.py CHANGED
@@ -7,14 +7,14 @@
7
  # "transformers==5.19.0",
8
  # ]
9
  # ///
10
- """Ask Olmputer a question: OLMo calls its calc tool and the RISC-V computer in its weights runs it.
11
 
12
- OLMo writes the call; at </function_calls> the chip layers run the calc firmware and write the
13
- environment turn with the exact value; OLMo reads it and answers. The CPU's trace tokens stay in
14
- the chip layers (chipformer's RvChat; plain generate() does not run the machine), on chipformer's
15
- Triton executor when there is a GPU. Every token the machine writes is checked against a
16
- reference RISC-V emulator as it is generated, and the whole stream is replayed through the circuit
17
- at the end. The question is the first argument.
18
  """
19
 
20
  import sys
 
7
  # "transformers==5.19.0",
8
  # ]
9
  # ///
10
+ """Ask Olmputer a question and print the conversation.
11
 
12
+ OLMo writes a calc call. At </function_calls> the chip layers run the calculator firmware and
13
+ write the environment turn with the exact value, and OLMo reads it and answers. The CPU's trace
14
+ tokens stay inside the chip layers, so this decodes with chipformer's RvChat rather than
15
+ generate(), on its Triton executor when there is a GPU. Every token the computer writes is checked
16
+ against a reference RISC-V emulator as it is generated, and the whole conversation is replayed
17
+ through the circuit at the end. The question is the first argument.
18
  """
19
 
20
  import sys