dhad-vision / README.md
Alshami's picture
model card: focus on this model
3ae22f3 verified
|
Raw History Blame Contribute Delete
6.99 kB
---
license: apache-2.0
language:
- ar
- en
library_name: transformers
pipeline_tag: image-text-to-text
base_model:
- inception42/Jais-3-0.9B
- google/siglip2-base-patch16-naflex
tags:
- arabic
- vision
- image-captioning
- image-understanding
- siglip2
- jais
---
# dhad-1-vision-preview (رؤية)
An **Arabic image-captioning model**: give it a picture, it writes a short Arabic description.
A frozen SigLIP2 vision encoder turns the image into 576 patch features; a small trained
**connector** maps them to 144 soft tokens (64 in low-token mode); the tokens are spliced into a
frozen Jais-3-0.9B language model through two learned marker vectors, and the model writes the
caption.
**Preview.** Only stage 1 (the connector) was finished. Its output is caption-style text and it
did not go through a formal release gate. An unfinished stage-2 snapshot (a vision LoRA that
answers free-form questions about an image, stopped at step 675 of 1,435) is included as an
extra, clearly labelled `UNFINISHED`.
## What is in this repository
| File | What |
|---|---|
| `connector.pt` | **The model.** A `torch.save` dict with `connector` (7,080,961 params) and `markers` (2 × 1536 = 3,072 params). |
| `MANIFEST.json` | Provenance: sources, sha256, parameter counts, training recipe, gate notes. |
| `requirements.txt` | Runtime Python packages. |
| `LICENSE` | Apache License 2.0 text. |
| `code/connector.py` | The connector architecture (pixel-shuffle → MLP → L2 × learnable scale). |
| `code/graft.py` | Splices the soft tokens into the frozen Jais trunk (`inputs_embeds`) and generates. |
| `code/infer.py` | `describe(image, prompt)` helper + CLI. |
| `code/encoder.py` | Runs the frozen SigLIP2 encoder on raw images → `[576, 768]` features. |
| `stage2-step675/` | UNFINISHED stage-2 snapshot: its own `connector.pt` + `vision_lora/` (rank-32 LoRA, 27,983,872 params). See `stage2-step675/UNFINISHED.md`. |
The base models are not re-uploaded: load `inception42/Jais-3-0.9B` (Apache-2.0) and
`google/siglip2-base-patch16-naflex` (Apache-2.0) yourself.
## Parameter count
The full runtime is **1,171,213,057 parameters** (≈1.17B): the frozen Jais trunk
(1,078,285,824) + the frozen SigLIP2 vision tower used (85,843,200) + the connector and markers
(7,084,033). Only the 28 MB connector file is new; the rest is public base weights. With the
unfinished stage-2 LoRA it would be 1,199,196,929.
## How it was trained (stage 1)
- **Trainable:** connector + 2 marker vectors only (7.08M). The SigLIP2 encoder and the Jais trunk
are frozen end-to-end.
- **Data:** 29,800 train + 200 held-out pairs of (image, Arabic caption) from
`FreedomIntelligence/ALLaVA-4V` images with `FreedomIntelligence/ALLaVA-4V-Arabic` captions
(machine translation of GPT-4V captions). Captions trimmed to their first 1–2 sentences.
See `MANIFEST.json`.
- **Recipe:** cosine LR 1e-3, 64 samples/step, 2 epochs (930 steps), bf16, gradient
checkpointing, 30 % of steps in 64-token mode.
- **Probe at step 930:** caption loss on the true image 2.4456 vs shuffled image 3.0071
(**+18.7 % gain**). Owner-tested and accepted 2026-11-14; **no formal gate was run**.
## Usage
The weights are a raw state dict, so you also need the `code/` folder (shipped here) and the two
base models (downloaded automatically on first run).
### 1. Download this repository
```bash
hf download dhadlab/dhad-vision --local-dir dhad-1-vision-preview-1.2b
cd dhad-1-vision-preview-1.2b
```
Or download it from the **Files** tab of this page.
### 2. Install
```bash
pip install -r requirements.txt
```
### 3. Describe an image
```bash
python code/infer.py photo.jpg
python code/infer.py photo.jpg "صف هذه الصورة بالتفصيل"
```
Real output from a test photo:
> الصورة تظهر مجموعة من أشجار التفاح في بيئة خريفية. الأشجار مغطاة بأوراق بنية داكنة، مما يشير إلى أنها في أواخر الموسم.
### Python
```python
import sys
sys.path.insert(0, "code")
import infer
g = infer.load_graft(device="cuda") # loads Jais + connector.pt (once)
print(infer.describe("photo.jpg", "صِف هذه الصورة", graft=g))
print(infer.describe("b.jpg", graft=g)) # reuse the loaded models
```
`infer.describe` also accepts a path, a PIL image, raw bytes, or cached SigLIP2 features
(`[576, 768]` fp16). Defaults: SigLIP2 encoder on CPU (1–2 s per image), 96 new tokens, Arabic
prompt `صِف هذه الصورة`. Environment options: `DEVICE=cuda` (encoder on GPU), `LOW=1`
(64-token mode), `CONNECTOR=<dir>`, `JAIS_PATH` / `SIGLIP2_PATH` (local base-model copies).
Expect caption-style output, not chat: this model **describes** an image; it does not reliably
answer questions about it (that was the unfinished stage 2).
## Intended use and limits
- Arabic (and some English) **image captioning / description** for research and demos.
- **Not** an OCR engine (glyph probe was weak), **not** a reliable visual question answerer, and
**not** grounded truth. Captions may be inaccurate, generic, or hallucinated — the training
captions themselves are GPT-4V output, not native ground truth.
- No safety/NSFW filtering was applied to training images.
- Stage 1 is a captioner; free-form image Q&A and image-follow-up turns belong to the unfinished
stage-2 snapshot and are not supported by the released weights.
## Licence and data provenance
**Apache License 2.0.** The connector weights and the code in this repository are released under
the Apache License 2.0 (see `LICENSE`).
Training-data provenance (informational): the connector was trained on ALLaVA-4V captions and
LAION-crawled images. ALLaVA-4V's card states CC-BY-NC-4.0; its uploaders do not own the image
rights, per-image licences are unknown, and the captions are ChatGPT/GPT-4V-derived. We do not
ship any images or captions — only the trained connector — but if you redistribute or build on
this model, review those upstream terms yourself.
- Base trunk: `inception42/Jais-3-0.9B` — Apache-2.0 (declared in its model card).
- Encoder: `google/siglip2-base-patch16-naflex` — Apache-2.0. Not re-uploaded; cite
[SigLIP 2, arXiv:2502.14786](https://huggingface.co/papers/2502.14786).
- Training images/captions: `FreedomIntelligence/ALLaVA-4V` (CC-BY-NC-4.0 card) and
`FreedomIntelligence/ALLaVA-4V-Arabic` (Apache-2.0 card, derived from the GPT-4V captions).
See `NOTICE.md` for the full third-party list.
## Files and checksums
`MANIFEST.json` holds the sha256 of every shipped file (recompute with `sha256sum`). Connector:
`82c611cadf8e694d23b5edea89297a5647c61417f45e11a168f5d9cafc63dd22` (28,339,217 B).
## Citation
```bibtex
@misc{dhad1visionpreview,
title = {dhad-1-vision-preview: an Arabic image-captioning connector},
author = {Dhad},
year = {2026},
note = {Preview release, stage 1 only. https://huggingface.co/dhadlab/dhad-vision}
}
```