File size: 6,988 Bytes
9e4b6ba
 
e3f7919
 
 
 
 
 
 
 
 
 
 
 
 
 
 
9e4b6ba
e3f7919
 
 
3ae22f3
e3f7919
3ae22f3
 
 
 
e3f7919
3ae22f3
 
 
 
e3f7919
 
 
 
 
3ae22f3
e3f7919
 
 
 
 
 
 
3ae22f3
e3f7919
3ae22f3
e3f7919
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3ae22f3
e3f7919
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3ae22f3
e3f7919
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3ae22f3
e3f7919
 
 
 
 
 
 
 
 
3ae22f3
e3f7919
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3ae22f3
e3f7919
 
 
 
 
 
 
 
 
 
3ae22f3
e3f7919
 
3ae22f3
e3f7919
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
---
license: apache-2.0
language:
  - ar
  - en
library_name: transformers
pipeline_tag: image-text-to-text
base_model:
  - inception42/Jais-3-0.9B
  - google/siglip2-base-patch16-naflex
tags:
  - arabic
  - vision
  - image-captioning
  - image-understanding
  - siglip2
  - jais
---

# dhad-1-vision-preview (رؤية)

An **Arabic image-captioning model**: give it a picture, it writes a short Arabic description.

A frozen SigLIP2 vision encoder turns the image into 576 patch features; a small trained
**connector** maps them to 144 soft tokens (64 in low-token mode); the tokens are spliced into a
frozen Jais-3-0.9B language model through two learned marker vectors, and the model writes the
caption.

**Preview.** Only stage 1 (the connector) was finished. Its output is caption-style text and it
did not go through a formal release gate. An unfinished stage-2 snapshot (a vision LoRA that
answers free-form questions about an image, stopped at step 675 of 1,435) is included as an
extra, clearly labelled `UNFINISHED`.

## What is in this repository

| File | What |
|---|---|
| `connector.pt` | **The model.** A `torch.save` dict with `connector` (7,080,961 params) and `markers` (2 × 1536 = 3,072 params). |
| `MANIFEST.json` | Provenance: sources, sha256, parameter counts, training recipe, gate notes. |
| `requirements.txt` | Runtime Python packages. |
| `LICENSE` | Apache License 2.0 text. |
| `code/connector.py` | The connector architecture (pixel-shuffle → MLP → L2 × learnable scale). |
| `code/graft.py` | Splices the soft tokens into the frozen Jais trunk (`inputs_embeds`) and generates. |
| `code/infer.py` | `describe(image, prompt)` helper + CLI. |
| `code/encoder.py` | Runs the frozen SigLIP2 encoder on raw images → `[576, 768]` features. |
| `stage2-step675/` | UNFINISHED stage-2 snapshot: its own `connector.pt` + `vision_lora/` (rank-32 LoRA, 27,983,872 params). See `stage2-step675/UNFINISHED.md`. |

The base models are not re-uploaded: load `inception42/Jais-3-0.9B` (Apache-2.0) and
`google/siglip2-base-patch16-naflex` (Apache-2.0) yourself.

## Parameter count

The full runtime is **1,171,213,057 parameters** (≈1.17B): the frozen Jais trunk
(1,078,285,824) + the frozen SigLIP2 vision tower used (85,843,200) + the connector and markers
(7,084,033). Only the 28 MB connector file is new; the rest is public base weights. With the
unfinished stage-2 LoRA it would be 1,199,196,929.

## How it was trained (stage 1)

- **Trainable:** connector + 2 marker vectors only (7.08M). The SigLIP2 encoder and the Jais trunk
  are frozen end-to-end.
- **Data:** 29,800 train + 200 held-out pairs of (image, Arabic caption) from
  `FreedomIntelligence/ALLaVA-4V` images with `FreedomIntelligence/ALLaVA-4V-Arabic` captions
  (machine translation of GPT-4V captions). Captions trimmed to their first 1–2 sentences.
  See `MANIFEST.json`.
- **Recipe:** cosine LR 1e-3, 64 samples/step, 2 epochs (930 steps), bf16, gradient
  checkpointing, 30 % of steps in 64-token mode.
- **Probe at step 930:** caption loss on the true image 2.4456 vs shuffled image 3.0071
  (**+18.7 % gain**). Owner-tested and accepted 2026-11-14; **no formal gate was run**.

## Usage

The weights are a raw state dict, so you also need the `code/` folder (shipped here) and the two
base models (downloaded automatically on first run).

### 1. Download this repository

```bash
hf download dhadlab/dhad-vision --local-dir dhad-1-vision-preview-1.2b
cd dhad-1-vision-preview-1.2b
```

Or download it from the **Files** tab of this page.

### 2. Install

```bash
pip install -r requirements.txt
```

### 3. Describe an image

```bash
python code/infer.py photo.jpg
python code/infer.py photo.jpg "صف هذه الصورة بالتفصيل"
```

Real output from a test photo:

> الصورة تظهر مجموعة من أشجار التفاح في بيئة خريفية. الأشجار مغطاة بأوراق بنية داكنة، مما يشير إلى أنها في أواخر الموسم.

### Python

```python
import sys
sys.path.insert(0, "code")
import infer

g = infer.load_graft(device="cuda")                 # loads Jais + connector.pt (once)
print(infer.describe("photo.jpg", "صِف هذه الصورة", graft=g))
print(infer.describe("b.jpg", graft=g))             # reuse the loaded models
```

`infer.describe` also accepts a path, a PIL image, raw bytes, or cached SigLIP2 features
(`[576, 768]` fp16). Defaults: SigLIP2 encoder on CPU (1–2 s per image), 96 new tokens, Arabic
prompt `صِف هذه الصورة`. Environment options: `DEVICE=cuda` (encoder on GPU), `LOW=1`
(64-token mode), `CONNECTOR=<dir>`, `JAIS_PATH` / `SIGLIP2_PATH` (local base-model copies).

Expect caption-style output, not chat: this model **describes** an image; it does not reliably
answer questions about it (that was the unfinished stage 2).

## Intended use and limits

- Arabic (and some English) **image captioning / description** for research and demos.
- **Not** an OCR engine (glyph probe was weak), **not** a reliable visual question answerer, and
  **not** grounded truth. Captions may be inaccurate, generic, or hallucinated — the training
  captions themselves are GPT-4V output, not native ground truth.
- No safety/NSFW filtering was applied to training images.
- Stage 1 is a captioner; free-form image Q&A and image-follow-up turns belong to the unfinished
  stage-2 snapshot and are not supported by the released weights.

## Licence and data provenance

**Apache License 2.0.** The connector weights and the code in this repository are released under
the Apache License 2.0 (see `LICENSE`).

Training-data provenance (informational): the connector was trained on ALLaVA-4V captions and
LAION-crawled images. ALLaVA-4V's card states CC-BY-NC-4.0; its uploaders do not own the image
rights, per-image licences are unknown, and the captions are ChatGPT/GPT-4V-derived. We do not
ship any images or captions — only the trained connector — but if you redistribute or build on
this model, review those upstream terms yourself.

- Base trunk: `inception42/Jais-3-0.9B` — Apache-2.0 (declared in its model card).
- Encoder: `google/siglip2-base-patch16-naflex` — Apache-2.0. Not re-uploaded; cite
  [SigLIP 2, arXiv:2502.14786](https://huggingface.co/papers/2502.14786).
- Training images/captions: `FreedomIntelligence/ALLaVA-4V` (CC-BY-NC-4.0 card) and
  `FreedomIntelligence/ALLaVA-4V-Arabic` (Apache-2.0 card, derived from the GPT-4V captions).

See `NOTICE.md` for the full third-party list.

## Files and checksums

`MANIFEST.json` holds the sha256 of every shipped file (recompute with `sha256sum`). Connector:
`82c611cadf8e694d23b5edea89297a5647c61417f45e11a168f5d9cafc63dd22` (28,339,217 B).

## Citation

```bibtex
@misc{dhad1visionpreview,
  title  = {dhad-1-vision-preview: an Arabic image-captioning connector},
  author = {Dhad},
  year   = {2026},
  note   = {Preview release, stage 1 only. https://huggingface.co/dhadlab/dhad-vision}
}
```