File size: 2,672 Bytes
0cd1141 ae23feb 0cd1141 ae23feb | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 | ---
library_name: transformers
pipeline_tag: image-text-to-text
tags:
- molmo2
- tiny-random
- testing
base_model: allenai/MolmoWeb-4B
---
# tiny-random-molmo2
A tiny Molmo2 (`allenai/MolmoWeb-4B` architecture) with random weights, used for the
OpenVINO GenAI `VLMPipeline` and `genai-cli export` tests. It produces meaningless text and is
meant only for testing.
- Text model: 3 layers, hidden size 128, 4 attention heads / 2 KV heads, head_dim 32, full Qwen vocabulary
- Vision (ViT): 3 layers, hidden size 64, 2 heads, head_dim 32; adapter uses the last two ViT layers
- 39.6M parameters, float32
- Tokenizer, processor configs, chat template and remote code (`*_molmo2.py`) are copied unchanged
from `allenai/MolmoWeb-4B`
Molmo2 is remote code: load it with `trust_remote_code=True` and `transformers>=4.57,<4.58`
(`einops` and `torchvision` are also required).
## Generation script
```python
# uv run --with transformers==4.57.6 --with einops --with torchvision python make_tiny_molmo2.py <MolmoWeb-4B snapshot> <out>
import math
import shutil
import sys
from pathlib import Path
import torch
from transformers import AutoConfig, AutoModelForImageTextToText
src, out = Path(sys.argv[1]), Path(sys.argv[2])
torch.manual_seed(0)
cfg = AutoConfig.from_pretrained(src, trust_remote_code=True)
t, v, a = cfg.text_config, cfg.vit_config, cfg.adapter_config
t.hidden_size, t.num_attention_heads, t.num_key_value_heads, t.head_dim = 128, 4, 2, 32
t.num_hidden_layers, t.intermediate_size = 3, 256
v.hidden_size, v.intermediate_size, v.num_hidden_layers = 64, 128, 3
v.num_attention_heads = v.num_key_value_heads = 2
v.head_dim = 32
a.hidden_size, a.num_attention_heads, a.num_key_value_heads, a.head_dim = 64, 2, 2, 32
a.intermediate_size, a.text_hidden_size, a.vit_layers = 128, t.hidden_size, [-1, -2]
model = AutoModelForImageTextToText.from_config(cfg, trust_remote_code=True).float()
with torch.no_grad():
for name, p in model.named_parameters():
if name.endswith("norm.weight") or "ln_f" in name or "_norm" in name:
continue
if p.ndim == 2: # linear / embedding / positional embedding
std = 1.0 if ("wte" in name or "positional_embedding" in name) else 1.0 / math.sqrt(p.shape[1])
if name == "lm_head.weight":
std = 4.0 / math.sqrt(p.shape[1])
p.normal_(0.0, std)
elif name.endswith("bias"):
p.normal_(0.0, 0.02)
model.save_pretrained(out, safe_serialization=True)
for f in src.iterdir():
if f.is_file() and f.suffix != ".safetensors" and f.name not in ("config.json", "model.safetensors.index.json"):
shutil.copy(f, out / f.name)
```
|