Image-Text-to-Text
Transformers
Safetensors
Arabic
English
arabic
vision
image-captioning
image-understanding
siglip2
jais
Instructions to use dhadlab/dhad-vision with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use dhadlab/dhad-vision with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="dhadlab/dhad-vision")# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("dhadlab/dhad-vision", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use dhadlab/dhad-vision with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "dhadlab/dhad-vision" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "dhadlab/dhad-vision", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/dhadlab/dhad-vision
- SGLang
How to use dhadlab/dhad-vision with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "dhadlab/dhad-vision" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "dhadlab/dhad-vision", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "dhadlab/dhad-vision" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "dhadlab/dhad-vision", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use dhadlab/dhad-vision with Docker Model Runner:
docker model run hf.co/dhadlab/dhad-vision
File size: 6,988 Bytes
9e4b6ba e3f7919 9e4b6ba e3f7919 3ae22f3 e3f7919 3ae22f3 e3f7919 3ae22f3 e3f7919 3ae22f3 e3f7919 3ae22f3 e3f7919 3ae22f3 e3f7919 3ae22f3 e3f7919 3ae22f3 e3f7919 3ae22f3 e3f7919 3ae22f3 e3f7919 3ae22f3 e3f7919 3ae22f3 e3f7919 3ae22f3 e3f7919 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 | ---
license: apache-2.0
language:
- ar
- en
library_name: transformers
pipeline_tag: image-text-to-text
base_model:
- inception42/Jais-3-0.9B
- google/siglip2-base-patch16-naflex
tags:
- arabic
- vision
- image-captioning
- image-understanding
- siglip2
- jais
---
# dhad-1-vision-preview (رؤية)
An **Arabic image-captioning model**: give it a picture, it writes a short Arabic description.
A frozen SigLIP2 vision encoder turns the image into 576 patch features; a small trained
**connector** maps them to 144 soft tokens (64 in low-token mode); the tokens are spliced into a
frozen Jais-3-0.9B language model through two learned marker vectors, and the model writes the
caption.
**Preview.** Only stage 1 (the connector) was finished. Its output is caption-style text and it
did not go through a formal release gate. An unfinished stage-2 snapshot (a vision LoRA that
answers free-form questions about an image, stopped at step 675 of 1,435) is included as an
extra, clearly labelled `UNFINISHED`.
## What is in this repository
| File | What |
|---|---|
| `connector.pt` | **The model.** A `torch.save` dict with `connector` (7,080,961 params) and `markers` (2 × 1536 = 3,072 params). |
| `MANIFEST.json` | Provenance: sources, sha256, parameter counts, training recipe, gate notes. |
| `requirements.txt` | Runtime Python packages. |
| `LICENSE` | Apache License 2.0 text. |
| `code/connector.py` | The connector architecture (pixel-shuffle → MLP → L2 × learnable scale). |
| `code/graft.py` | Splices the soft tokens into the frozen Jais trunk (`inputs_embeds`) and generates. |
| `code/infer.py` | `describe(image, prompt)` helper + CLI. |
| `code/encoder.py` | Runs the frozen SigLIP2 encoder on raw images → `[576, 768]` features. |
| `stage2-step675/` | UNFINISHED stage-2 snapshot: its own `connector.pt` + `vision_lora/` (rank-32 LoRA, 27,983,872 params). See `stage2-step675/UNFINISHED.md`. |
The base models are not re-uploaded: load `inception42/Jais-3-0.9B` (Apache-2.0) and
`google/siglip2-base-patch16-naflex` (Apache-2.0) yourself.
## Parameter count
The full runtime is **1,171,213,057 parameters** (≈1.17B): the frozen Jais trunk
(1,078,285,824) + the frozen SigLIP2 vision tower used (85,843,200) + the connector and markers
(7,084,033). Only the 28 MB connector file is new; the rest is public base weights. With the
unfinished stage-2 LoRA it would be 1,199,196,929.
## How it was trained (stage 1)
- **Trainable:** connector + 2 marker vectors only (7.08M). The SigLIP2 encoder and the Jais trunk
are frozen end-to-end.
- **Data:** 29,800 train + 200 held-out pairs of (image, Arabic caption) from
`FreedomIntelligence/ALLaVA-4V` images with `FreedomIntelligence/ALLaVA-4V-Arabic` captions
(machine translation of GPT-4V captions). Captions trimmed to their first 1–2 sentences.
See `MANIFEST.json`.
- **Recipe:** cosine LR 1e-3, 64 samples/step, 2 epochs (930 steps), bf16, gradient
checkpointing, 30 % of steps in 64-token mode.
- **Probe at step 930:** caption loss on the true image 2.4456 vs shuffled image 3.0071
(**+18.7 % gain**). Owner-tested and accepted 2026-11-14; **no formal gate was run**.
## Usage
The weights are a raw state dict, so you also need the `code/` folder (shipped here) and the two
base models (downloaded automatically on first run).
### 1. Download this repository
```bash
hf download dhadlab/dhad-vision --local-dir dhad-1-vision-preview-1.2b
cd dhad-1-vision-preview-1.2b
```
Or download it from the **Files** tab of this page.
### 2. Install
```bash
pip install -r requirements.txt
```
### 3. Describe an image
```bash
python code/infer.py photo.jpg
python code/infer.py photo.jpg "صف هذه الصورة بالتفصيل"
```
Real output from a test photo:
> الصورة تظهر مجموعة من أشجار التفاح في بيئة خريفية. الأشجار مغطاة بأوراق بنية داكنة، مما يشير إلى أنها في أواخر الموسم.
### Python
```python
import sys
sys.path.insert(0, "code")
import infer
g = infer.load_graft(device="cuda") # loads Jais + connector.pt (once)
print(infer.describe("photo.jpg", "صِف هذه الصورة", graft=g))
print(infer.describe("b.jpg", graft=g)) # reuse the loaded models
```
`infer.describe` also accepts a path, a PIL image, raw bytes, or cached SigLIP2 features
(`[576, 768]` fp16). Defaults: SigLIP2 encoder on CPU (1–2 s per image), 96 new tokens, Arabic
prompt `صِف هذه الصورة`. Environment options: `DEVICE=cuda` (encoder on GPU), `LOW=1`
(64-token mode), `CONNECTOR=<dir>`, `JAIS_PATH` / `SIGLIP2_PATH` (local base-model copies).
Expect caption-style output, not chat: this model **describes** an image; it does not reliably
answer questions about it (that was the unfinished stage 2).
## Intended use and limits
- Arabic (and some English) **image captioning / description** for research and demos.
- **Not** an OCR engine (glyph probe was weak), **not** a reliable visual question answerer, and
**not** grounded truth. Captions may be inaccurate, generic, or hallucinated — the training
captions themselves are GPT-4V output, not native ground truth.
- No safety/NSFW filtering was applied to training images.
- Stage 1 is a captioner; free-form image Q&A and image-follow-up turns belong to the unfinished
stage-2 snapshot and are not supported by the released weights.
## Licence and data provenance
**Apache License 2.0.** The connector weights and the code in this repository are released under
the Apache License 2.0 (see `LICENSE`).
Training-data provenance (informational): the connector was trained on ALLaVA-4V captions and
LAION-crawled images. ALLaVA-4V's card states CC-BY-NC-4.0; its uploaders do not own the image
rights, per-image licences are unknown, and the captions are ChatGPT/GPT-4V-derived. We do not
ship any images or captions — only the trained connector — but if you redistribute or build on
this model, review those upstream terms yourself.
- Base trunk: `inception42/Jais-3-0.9B` — Apache-2.0 (declared in its model card).
- Encoder: `google/siglip2-base-patch16-naflex` — Apache-2.0. Not re-uploaded; cite
[SigLIP 2, arXiv:2502.14786](https://huggingface.co/papers/2502.14786).
- Training images/captions: `FreedomIntelligence/ALLaVA-4V` (CC-BY-NC-4.0 card) and
`FreedomIntelligence/ALLaVA-4V-Arabic` (Apache-2.0 card, derived from the GPT-4V captions).
See `NOTICE.md` for the full third-party list.
## Files and checksums
`MANIFEST.json` holds the sha256 of every shipped file (recompute with `sha256sum`). Connector:
`82c611cadf8e694d23b5edea89297a5647c61417f45e11a168f5d9cafc63dd22` (28,339,217 B).
## Citation
```bibtex
@misc{dhad1visionpreview,
title = {dhad-1-vision-preview: an Arabic image-captioning connector},
author = {Dhad},
year = {2026},
note = {Preview release, stage 1 only. https://huggingface.co/dhadlab/dhad-vision}
}
```
|