File size: 8,688 Bytes
7594205
edf2c2e
 
 
 
57a01e3
edf2c2e
 
abe6d35
 
 
57a01e3
edf2c2e
 
abe6d35
 
 
7594205
edf2c2e
abe6d35
 
57a01e3
abe6d35
57a01e3
abe6d35
57a01e3
abe6d35
57a01e3
 
 
 
 
 
 
abe6d35
57a01e3
edf2c2e
 
abe6d35
 
57a01e3
 
 
abe6d35
 
 
 
 
 
57a01e3
 
abe6d35
edf2c2e
 
57a01e3
 
 
 
 
 
 
 
 
 
 
 
 
abe6d35
 
57a01e3
abe6d35
57a01e3
 
 
 
abe6d35
57a01e3
abe6d35
 
 
 
 
57a01e3
abe6d35
57a01e3
edf2c2e
57a01e3
edf2c2e
 
57a01e3
abe6d35
 
57a01e3
abe6d35
 
 
 
57a01e3
abe6d35
 
 
 
 
 
 
57a01e3
 
 
 
 
abe6d35
 
57a01e3
abe6d35
 
57a01e3
 
 
 
 
abe6d35
 
 
 
 
 
 
57a01e3
abe6d35
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
57a01e3
abe6d35
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
57a01e3
abe6d35
 
 
edf2c2e
 
57a01e3
abe6d35
 
 
 
 
57a01e3
 
 
abe6d35
 
 
 
 
 
57a01e3
abe6d35
57a01e3
abe6d35
57a01e3
 
 
 
abe6d35
57a01e3
abe6d35
57a01e3
 
abe6d35
 
57a01e3
abe6d35
57a01e3
 
 
abe6d35
57a01e3
abe6d35
57a01e3
abe6d35
57a01e3
abe6d35
57a01e3
 
 
abe6d35
57a01e3
abe6d35
57a01e3
 
 
 
 
 
 
 
 
abe6d35
 
57a01e3
abe6d35
 
57a01e3
 
 
 
 
 
abe6d35
 
57a01e3
abe6d35
 
 
57a01e3
abe6d35
 
57a01e3
abe6d35
 
 
 
 
edf2c2e
 
57a01e3
edf2c2e
 
57a01e3
 
 
edf2c2e
 
abe6d35
edf2c2e
 
57a01e3
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
abe6d35
edf2c2e
57a01e3
edf2c2e
abe6d35
edf2c2e
57a01e3
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
---
language:
- en
library_name: pytorch
pipeline_tag: text-generation
inference: false
tags:
- text-generation
- pytorch
- custom-architecture
- native-bf16
- bfloat16
- inference-only
- local-inference
- no-quantization
- gradio
license: apache-2.0
---

# LuluV2 Native-bf16

LuluV2 is a local-first text-generation model from OpenMachine. This release is packaged as a **custom PyTorch runtime with native bfloat16 weights**. It is not a standard Transformers `AutoModelForCausalLM` repository, and it is not meant to be loaded through `pipeline(...)`, `AutoModelForCausalLM.from_pretrained(...)`, llama.cpp, Ollama, GGUF, GPTQ, AWQ, or any other quantized conversion path.

Clone the repo, keep the checkpoint as `LULUV2-bf16.pt`, and run it directly.

## Key points

- **Architecture:** native RPT/VWM-style LuluV2 architecture, not a normal decoder-only Transformers layout.
- **Runtime:** custom PyTorch inference code included in this repository.
- **Precision:** native `bfloat16` checkpoint.
- **Quantization:** no quantization step is required or recommended for the intended release path.
- **Mode:** inference-only public package.
- **Tokenizer:** local tokenizer files are included under `tokenizer/`.
- **Status:** experimental research/runtime release; some inference paths are still Python-heavy and not fully optimized.

## What is included

```text
.
β”œβ”€β”€ LULUV2-bf16.pt                  # native bf16 checkpoint, required
β”œβ”€β”€ README.md                       # model card and run guide
β”œβ”€β”€ app.py                          # local Gradio chat UI
β”œβ”€β”€ run_inference.py                # command-line generation runner
β”œβ”€β”€ luluv2_inference_runtime.py     # custom architecture + checkpoint loader
β”œβ”€β”€ luluv2_live_inference.py        # streaming generation engine
β”œβ”€β”€ luluv2_optimized_engine.py      # optimized local engine variant
β”œβ”€β”€ config.json                     # HF metadata only; not an AutoModel config
β”œβ”€β”€ generation_config.json          # default generation settings
β”œβ”€β”€ requirements.txt
β”œβ”€β”€ run_chat.sh                     # Linux/macOS launcher
β”œβ”€β”€ run_chat.ps1                    # Windows PowerShell launcher
└── tokenizer/                      # local tokenizer files
```

## What this is not

This repository is **not**:

- a standard Transformers model package
- a hosted inference-widget-first package
- a training repository
- a dataset release
- a quantized model release
- a GGUF / llama.cpp / Ollama release

Use the included runtime scripts.

## Requirements

Recommended environment:

- Python 3.10 or newer
- PyTorch 2.1 or newer
- CUDA GPU with bf16 support for best performance
- Local disk space for the checkpoint

Install dependencies:

```bash
pip install -r requirements.txt
```

The repo is local-first. The runtime uses the included checkpoint and tokenizer folder. It does not download external model weights at startup.

## Quick start: chat UI

From the cloned repository:

```bash
python app.py --inbrowser
```

Equivalent explicit command:

```bash
python app.py \
  --ckpt ./LULUV2-bf16.pt \
  --model-py ./luluv2_inference_runtime.py \
  --tokenizer-dir ./tokenizer \
  --device cuda \
  --dtype bf16 \
  --max-context 32768 \
  --inbrowser
```

The UI starts a local Gradio chat app. By default it binds to `127.0.0.1` on port `7862`.

### Launcher scripts

Linux/macOS:

```bash
bash ./run_chat.sh
```

Windows PowerShell:

```powershell
.\run_chat.ps1
```

## Quick start: command line

```bash
python run_inference.py \
  --ckpt ./LULUV2-bf16.pt \
  --tokenizer-dir ./tokenizer \
  --device cuda \
  --dtype bf16 \
  --prompt "Write a short introduction to LuluV2."
```

Generation controls:

```bash
python run_inference.py \
  --ckpt ./LULUV2-bf16.pt \
  --prompt "Explain why native bf16 inference matters." \
  --max-new-tokens 700 \
  --temperature 0.65 \
  --top-p 0.90 \
  --top-k 40
```

## Python usage

```python
import torch
from luluv2_live_inference import LULUV2LiveEngine, GenerationConfig

engine = LULUV2LiveEngine(
    ckpt_path="./LULUV2-bf16.pt",
    model_py="./luluv2_inference_runtime.py",
    tokenizer_dir="./tokenizer",
    device="cuda",
    dtype="bf16",
    local_files_only=True,
    no_config_download=True,
)

cfg = GenerationConfig(
    max_new_tokens=512,
    temperature=0.65,
    top_p=0.90,
    top_k=40,
)

history = []
system_prompt = "You are LuluV2, a helpful local AI assistant."

for partial_text in engine.generate_stream(
    "What are you?",
    history,
    system_prompt,
    cfg,
):
    print(partial_text, end="", flush=True)

print()
torch.cuda.empty_cache()
```

## Native bf16: no quantization step

LuluV2 is intended to run from the native bf16 checkpoint:

```text
LULUV2-bf16.pt
```

Start with:

```bash
--dtype bf16
```

Do not quantize the model before using this package. The intended baseline is direct native-bf16 execution.

Fallback modes are available for hardware compatibility:

```bash
--dtype fp16
--dtype fp32
```

These are compatibility modes, not separate quantized releases.

## Hardware notes

- **CUDA / NVIDIA:** recommended path. Use `--device cuda --dtype bf16` when your GPU supports bf16.
- **Windows / PC:** use the same CUDA command above. If memory is tight, reduce `--max-context` first.
- **macOS:** CPU mode is the safer compatibility path at the moment. MPS may produce unstable logits in some environments, so use `--device cpu --dtype fp32` if MPS causes issues.
- **CPU:** supported for testing and compatibility, but generation will be much slower than GPU inference.

CPU test command:

```bash
python app.py --ckpt ./LULUV2-bf16.pt --device cpu --dtype fp32 --max-context 4096 --inbrowser
```

If CUDA runs out of memory, reduce context length:

```bash
python app.py --ckpt ./LULUV2-bf16.pt --device cuda --dtype bf16 --max-context 8192 --inbrowser
```

## Context length

The runtime exposes `--max-context` and the architecture may support long context settings. Start with a practical context such as `4096` or `8192`, then increase if your hardware has enough memory and the runtime remains stable.

Example:

```bash
python app.py --ckpt ./LULUV2-bf16.pt --max-context 4096 --inbrowser
```

## Checkpoint format

The runtime expects a PyTorch checkpoint file named:

```text
LULUV2-bf16.pt
```

The checkpoint should contain a model state dictionary under the `model` key. Optional fields may include runtime metadata and second-pass refinement weights.

If your checkpoint has a different filename, pass it explicitly:

```bash
python app.py --ckpt /path/to/your-checkpoint.pt --tokenizer-dir ./tokenizer --inbrowser
```

## Troubleshooting

### `FileNotFoundError: LULUV2-bf16.pt`

The checkpoint is missing or has a different name. Put `LULUV2-bf16.pt` in the repository root or pass the full path with `--ckpt`.

### `Checkpoint missing model state dict`

The file passed to `--ckpt` is not the expected LuluV2 PyTorch checkpoint. Make sure you are using the native `.pt` checkpoint, not a config file, tokenizer file, or unrelated export.

### CUDA out of memory

Lower the context window and/or generated token count:

```bash
python app.py --ckpt ./LULUV2-bf16.pt --max-context 4096 --inbrowser
```

### Older GPU does not support bf16

Try fp16:

```bash
python app.py --ckpt ./LULUV2-bf16.pt --device cuda --dtype fp16 --inbrowser
```

### Mac MPS issues

Use CPU mode for compatibility:

```bash
python app.py --ckpt ./LULUV2-bf16.pt --device cpu --dtype fp32 --inbrowser
```

## Model status

This is an early public inference release. The model has been tested for chat behavior, including English and multilingual prompts, but evaluation is still ongoing. The public package is intentionally stripped to the files needed for local inference.

Fine-tuning code and additional technical notes are planned separately. A mobile-oriented native runtime is also being explored.

## Feedback

Try it locally, inspect the runtime, break things, and report what is missing. Useful feedback includes:

- hardware and OS
- command used
- context length
- dtype
- error logs
- examples of good or bad generations

## Safety and intended use

LuluV2 is a local text-generation model. It can produce incorrect, incomplete, or unsafe outputs. Do not treat generations as verified facts for medical, legal, financial, security, or other high-stakes decisions. Use human review where accuracy matters.

## License and notices

This repository is marked as Apache-2.0. Before publishing the final weights, make sure the repository includes any notices, attribution, or license text required by the checkpoint, training sources, tokenizer, or dependencies.