Text Generation
Transformers
Safetensors
olmo_hybrid
risc-v
compiled-weights
olmo-hybrid
gated-deltanet
tool-use
conversational
Instructions to use jgalego/olmputer with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use jgalego/olmputer with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="jgalego/olmputer") messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("jgalego/olmputer") model = AutoModelForCausalLM.from_pretrained("jgalego/olmputer", device_map="auto") messages = [ {"role": "user", "content": "Who are you?"}, ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use jgalego/olmputer with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "jgalego/olmputer" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "jgalego/olmputer", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/jgalego/olmputer
- SGLang
How to use jgalego/olmputer with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "jgalego/olmputer" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "jgalego/olmputer", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "jgalego/olmputer" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "jgalego/olmputer", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use jgalego/olmputer with Docker Model Runner:
docker model run hf.co/jgalego/olmputer
Upload folder using huggingface_hub
Browse files- README.md +49 -13
- example.py +49 -0
README.md
CHANGED
|
@@ -89,26 +89,62 @@ Greedy decoding, 40 problems per set, scored by exact rational match of the fina
|
|
| 89 |
|
| 90 |
## Usage
|
| 91 |
|
|
|
|
|
|
|
| 92 |
```bash
|
| 93 |
-
|
| 94 |
```
|
| 95 |
|
| 96 |
```python
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 97 |
import urllib.request
|
|
|
|
| 98 |
|
| 99 |
-
|
| 100 |
-
|
| 101 |
from chipformer.rv import bridge, elf, ref
|
| 102 |
-
from chipformer.rv.chat import RvChat
|
| 103 |
-
|
| 104 |
-
firmware = "calc-olmo.elf"
|
| 105 |
-
urllib.request.urlretrieve("https://raw.githubusercontent.com/JGalego/olmputer/defd633c754fedd4c4ab7ff1cd9d109a9a5effcf/experimental/riscv/tests/data/rv/calc-olmo.elf", firmware)
|
| 106 |
|
| 107 |
-
|
| 108 |
-
|
| 109 |
-
|
| 110 |
-
|
| 111 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 112 |
```
|
| 113 |
|
| 114 |
-
The
|
|
|
|
| 89 |
|
| 90 |
## Usage
|
| 91 |
|
| 92 |
+
[`example.py`](example.py) asks Olmputer a question with the calc tool (the first argument) and prints the conversation; every token the machine writes is checked against a reference RISC-V emulator as it is generated, and the stream is replayed through the circuit at the end. It is a [uv script](https://docs.astral.sh/uv/guides/scripts/) that declares its own dependencies:
|
| 93 |
+
|
| 94 |
```bash
|
| 95 |
+
uv run https://huggingface.co/jgalego/olmputer/resolve/main/example.py "What is 2**64 - 1?"
|
| 96 |
```
|
| 97 |
|
| 98 |
```python
|
| 99 |
+
# /// script
|
| 100 |
+
# requires-python = ">=3.12"
|
| 101 |
+
# dependencies = [
|
| 102 |
+
# "chipformer @ https://github.com/JGalego/olmputer/archive/98639dff4c84a18542e3ca12a286a4df563b3a27.tar.gz#subdirectory=experimental/riscv", # pylint: disable=line-too-long
|
| 103 |
+
# "huggingface_hub",
|
| 104 |
+
# "torch==2.14.1",
|
| 105 |
+
# "transformers==5.19.0",
|
| 106 |
+
# ]
|
| 107 |
+
# ///
|
| 108 |
+
"""Ask Olmputer a question: OLMo calls its calc tool and the RISC-V computer in its weights runs it.
|
| 109 |
+
|
| 110 |
+
OLMo writes the call; at </function_calls> the chip layers run the calc firmware and write the
|
| 111 |
+
environment turn with the exact value; OLMo reads it and answers. The CPU's trace tokens stay in
|
| 112 |
+
the chip layers (chipformer's RvChat; plain generate() does not run the machine), on chipformer's
|
| 113 |
+
Triton executor when there is a GPU. Every token the machine writes is checked against a
|
| 114 |
+
reference RISC-V emulator as it is generated, and the whole stream is replayed through the circuit
|
| 115 |
+
at the end. The question is the first argument.
|
| 116 |
+
"""
|
| 117 |
+
|
| 118 |
+
import sys
|
| 119 |
import urllib.request
|
| 120 |
+
from pathlib import Path
|
| 121 |
|
| 122 |
+
import torch
|
|
|
|
| 123 |
from chipformer.rv import bridge, elf, ref
|
| 124 |
+
from chipformer.rv.chat import RvChat, replay
|
| 125 |
+
from huggingface_hub import snapshot_download
|
|
|
|
|
|
|
| 126 |
|
| 127 |
+
QUESTION = sys.argv[1] if len(sys.argv) > 1 else "Use calc: what is 123456789 * 987654321?"
|
| 128 |
+
FIRMWARE = Path("calc-olmo.elf")
|
| 129 |
+
if not FIRMWARE.exists():
|
| 130 |
+
urllib.request.urlretrieve("https://raw.githubusercontent.com/JGalego/olmputer/"
|
| 131 |
+
"98639dff4c84a18542e3ca12a286a4df563b3a27/experimental/riscv/"
|
| 132 |
+
"tests/data/rv/calc-olmo.elf", FIRMWARE)
|
| 133 |
+
|
| 134 |
+
BOT = RvChat(snapshot_download("jgalego/olmputer"), fast=False,
|
| 135 |
+
executor=torch.cuda.is_available())
|
| 136 |
+
IDS = BOT.prompt_ids([{"role": "user", "content": QUESTION}])
|
| 137 |
+
MACHINE, REFERENCE = (ref.Machine.from_program(elf.program(FIRMWARE)) for _ in range(2))
|
| 138 |
+
bridge.park(MACHINE) # the firmware waits for a call
|
| 139 |
+
bridge.park(REFERENCE)
|
| 140 |
+
LOCK = bridge.Session(BOT.T, REFERENCE)
|
| 141 |
+
LOCK.history(IDS)
|
| 142 |
+
RESULT = BOT.generate(IDS, MACHINE, 600, lock=LOCK)
|
| 143 |
+
|
| 144 |
+
print(BOT.text_of(RESULT["tokens"]))
|
| 145 |
+
print(f"\n{RESULT['text_tokens']} text tokens, {RESULT['chip_tokens']} trace tokens, "
|
| 146 |
+
f"{RESULT['seconds']:.1f} s; checked against the reference: {RESULT['lock_error'] is None}")
|
| 147 |
+
print("replayed through the circuit:", replay(BOT, IDS, RESULT["tokens"], FIRMWARE))
|
| 148 |
```
|
| 149 |
|
| 150 |
+
The checkpoint is about 32 GB in bf16, so it needs a GPU with 40 GB or more. Decoding goes through chipformer's `RvChat`, which keeps the CPU's trace tokens in the chip layers (on its Triton executor on a GPU): plain `generate()` does not run the machine.
|
example.py
ADDED
|
@@ -0,0 +1,49 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# /// script
|
| 2 |
+
# requires-python = ">=3.12"
|
| 3 |
+
# dependencies = [
|
| 4 |
+
# "chipformer @ https://github.com/JGalego/olmputer/archive/98639dff4c84a18542e3ca12a286a4df563b3a27.tar.gz#subdirectory=experimental/riscv", # pylint: disable=line-too-long
|
| 5 |
+
# "huggingface_hub",
|
| 6 |
+
# "torch==2.14.1",
|
| 7 |
+
# "transformers==5.19.0",
|
| 8 |
+
# ]
|
| 9 |
+
# ///
|
| 10 |
+
"""Ask Olmputer a question: OLMo calls its calc tool and the RISC-V computer in its weights runs it.
|
| 11 |
+
|
| 12 |
+
OLMo writes the call; at </function_calls> the chip layers run the calc firmware and write the
|
| 13 |
+
environment turn with the exact value; OLMo reads it and answers. The CPU's trace tokens stay in
|
| 14 |
+
the chip layers (chipformer's RvChat; plain generate() does not run the machine), on chipformer's
|
| 15 |
+
Triton executor when there is a GPU. Every token the machine writes is checked against a
|
| 16 |
+
reference RISC-V emulator as it is generated, and the whole stream is replayed through the circuit
|
| 17 |
+
at the end. The question is the first argument.
|
| 18 |
+
"""
|
| 19 |
+
|
| 20 |
+
import sys
|
| 21 |
+
import urllib.request
|
| 22 |
+
from pathlib import Path
|
| 23 |
+
|
| 24 |
+
import torch
|
| 25 |
+
from chipformer.rv import bridge, elf, ref
|
| 26 |
+
from chipformer.rv.chat import RvChat, replay
|
| 27 |
+
from huggingface_hub import snapshot_download
|
| 28 |
+
|
| 29 |
+
QUESTION = sys.argv[1] if len(sys.argv) > 1 else "Use calc: what is 123456789 * 987654321?"
|
| 30 |
+
FIRMWARE = Path("calc-olmo.elf")
|
| 31 |
+
if not FIRMWARE.exists():
|
| 32 |
+
urllib.request.urlretrieve("https://raw.githubusercontent.com/JGalego/olmputer/"
|
| 33 |
+
"98639dff4c84a18542e3ca12a286a4df563b3a27/experimental/riscv/"
|
| 34 |
+
"tests/data/rv/calc-olmo.elf", FIRMWARE)
|
| 35 |
+
|
| 36 |
+
BOT = RvChat(snapshot_download("jgalego/olmputer"), fast=False,
|
| 37 |
+
executor=torch.cuda.is_available())
|
| 38 |
+
IDS = BOT.prompt_ids([{"role": "user", "content": QUESTION}])
|
| 39 |
+
MACHINE, REFERENCE = (ref.Machine.from_program(elf.program(FIRMWARE)) for _ in range(2))
|
| 40 |
+
bridge.park(MACHINE) # the firmware waits for a call
|
| 41 |
+
bridge.park(REFERENCE)
|
| 42 |
+
LOCK = bridge.Session(BOT.T, REFERENCE)
|
| 43 |
+
LOCK.history(IDS)
|
| 44 |
+
RESULT = BOT.generate(IDS, MACHINE, 600, lock=LOCK)
|
| 45 |
+
|
| 46 |
+
print(BOT.text_of(RESULT["tokens"]))
|
| 47 |
+
print(f"\n{RESULT['text_tokens']} text tokens, {RESULT['chip_tokens']} trace tokens, "
|
| 48 |
+
f"{RESULT['seconds']:.1f} s; checked against the reference: {RESULT['lock_error'] is None}")
|
| 49 |
+
print("replayed through the circuit:", replay(BOT, IDS, RESULT["tokens"], FIRMWARE))
|