Image-Text-to-Text
Transformers
Safetensors
Arabic
English
arabic
vision
image-captioning
image-understanding
siglip2
jais
Instructions to use dhadlab/dhad-vision with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use dhadlab/dhad-vision with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="dhadlab/dhad-vision")# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("dhadlab/dhad-vision", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use dhadlab/dhad-vision with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "dhadlab/dhad-vision" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "dhadlab/dhad-vision", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/dhadlab/dhad-vision
- SGLang
How to use dhadlab/dhad-vision with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "dhadlab/dhad-vision" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "dhadlab/dhad-vision", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "dhadlab/dhad-vision" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "dhadlab/dhad-vision", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use dhadlab/dhad-vision with Docker Model Runner:
docker model run hf.co/dhadlab/dhad-vision
|
Download README.md from dhadlab/dhad-vision: direct link, hf CLI and curl.
- Browser
- Download file 6.99 kB
-
https://huggingface.co/dhadlab/dhad-vision/resolve/main/README.md
- Command line
-
hf download hf://dhadlab/dhad-vision/README.md
-
curl -L -o README.md https://huggingface.co/dhadlab/dhad-vision/resolve/main/README.md
6.99 kB
| license: apache-2.0 | |
| language: | |
| - ar | |
| - en | |
| library_name: transformers | |
| pipeline_tag: image-text-to-text | |
| base_model: | |
| - inception42/Jais-3-0.9B | |
| - google/siglip2-base-patch16-naflex | |
| tags: | |
| - arabic | |
| - vision | |
| - image-captioning | |
| - image-understanding | |
| - siglip2 | |
| - jais | |
| # dhad-1-vision-preview (رؤية) | |
| An **Arabic image-captioning model**: give it a picture, it writes a short Arabic description. | |
| A frozen SigLIP2 vision encoder turns the image into 576 patch features; a small trained | |
| **connector** maps them to 144 soft tokens (64 in low-token mode); the tokens are spliced into a | |
| frozen Jais-3-0.9B language model through two learned marker vectors, and the model writes the | |
| caption. | |
| **Preview.** Only stage 1 (the connector) was finished. Its output is caption-style text and it | |
| did not go through a formal release gate. An unfinished stage-2 snapshot (a vision LoRA that | |
| answers free-form questions about an image, stopped at step 675 of 1,435) is included as an | |
| extra, clearly labelled `UNFINISHED`. | |
| ## What is in this repository | |
| | File | What | | |
| |---|---| | |
| | `connector.pt` | **The model.** A `torch.save` dict with `connector` (7,080,961 params) and `markers` (2 × 1536 = 3,072 params). | | |
| | `MANIFEST.json` | Provenance: sources, sha256, parameter counts, training recipe, gate notes. | | |
| | `requirements.txt` | Runtime Python packages. | | |
| | `LICENSE` | Apache License 2.0 text. | | |
| | `code/connector.py` | The connector architecture (pixel-shuffle → MLP → L2 × learnable scale). | | |
| | `code/graft.py` | Splices the soft tokens into the frozen Jais trunk (`inputs_embeds`) and generates. | | |
| | `code/infer.py` | `describe(image, prompt)` helper + CLI. | | |
| | `code/encoder.py` | Runs the frozen SigLIP2 encoder on raw images → `[576, 768]` features. | | |
| | `stage2-step675/` | UNFINISHED stage-2 snapshot: its own `connector.pt` + `vision_lora/` (rank-32 LoRA, 27,983,872 params). See `stage2-step675/UNFINISHED.md`. | | |
| The base models are not re-uploaded: load `inception42/Jais-3-0.9B` (Apache-2.0) and | |
| `google/siglip2-base-patch16-naflex` (Apache-2.0) yourself. | |
| ## Parameter count | |
| The full runtime is **1,171,213,057 parameters** (≈1.17B): the frozen Jais trunk | |
| (1,078,285,824) + the frozen SigLIP2 vision tower used (85,843,200) + the connector and markers | |
| (7,084,033). Only the 28 MB connector file is new; the rest is public base weights. With the | |
| unfinished stage-2 LoRA it would be 1,199,196,929. | |
| ## How it was trained (stage 1) | |
| - **Trainable:** connector + 2 marker vectors only (7.08M). The SigLIP2 encoder and the Jais trunk | |
| are frozen end-to-end. | |
| - **Data:** 29,800 train + 200 held-out pairs of (image, Arabic caption) from | |
| `FreedomIntelligence/ALLaVA-4V` images with `FreedomIntelligence/ALLaVA-4V-Arabic` captions | |
| (machine translation of GPT-4V captions). Captions trimmed to their first 1–2 sentences. | |
| See `MANIFEST.json`. | |
| - **Recipe:** cosine LR 1e-3, 64 samples/step, 2 epochs (930 steps), bf16, gradient | |
| checkpointing, 30 % of steps in 64-token mode. | |
| - **Probe at step 930:** caption loss on the true image 2.4456 vs shuffled image 3.0071 | |
| (**+18.7 % gain**). Owner-tested and accepted 2026-11-14; **no formal gate was run**. | |
| ## Usage | |
| The weights are a raw state dict, so you also need the `code/` folder (shipped here) and the two | |
| base models (downloaded automatically on first run). | |
| ### 1. Download this repository | |
| ```bash | |
| hf download dhadlab/dhad-vision --local-dir dhad-1-vision-preview-1.2b | |
| cd dhad-1-vision-preview-1.2b | |
| ``` | |
| Or download it from the **Files** tab of this page. | |
| ### 2. Install | |
| ```bash | |
| pip install -r requirements.txt | |
| ``` | |
| ### 3. Describe an image | |
| ```bash | |
| python code/infer.py photo.jpg | |
| python code/infer.py photo.jpg "صف هذه الصورة بالتفصيل" | |
| ``` | |
| Real output from a test photo: | |
| > الصورة تظهر مجموعة من أشجار التفاح في بيئة خريفية. الأشجار مغطاة بأوراق بنية داكنة، مما يشير إلى أنها في أواخر الموسم. | |
| ### Python | |
| ```python | |
| import sys | |
| sys.path.insert(0, "code") | |
| import infer | |
| g = infer.load_graft(device="cuda") # loads Jais + connector.pt (once) | |
| print(infer.describe("photo.jpg", "صِف هذه الصورة", graft=g)) | |
| print(infer.describe("b.jpg", graft=g)) # reuse the loaded models | |
| ``` | |
| `infer.describe` also accepts a path, a PIL image, raw bytes, or cached SigLIP2 features | |
| (`[576, 768]` fp16). Defaults: SigLIP2 encoder on CPU (1–2 s per image), 96 new tokens, Arabic | |
| prompt `صِف هذه الصورة`. Environment options: `DEVICE=cuda` (encoder on GPU), `LOW=1` | |
| (64-token mode), `CONNECTOR=<dir>`, `JAIS_PATH` / `SIGLIP2_PATH` (local base-model copies). | |
| Expect caption-style output, not chat: this model **describes** an image; it does not reliably | |
| answer questions about it (that was the unfinished stage 2). | |
| ## Intended use and limits | |
| - Arabic (and some English) **image captioning / description** for research and demos. | |
| - **Not** an OCR engine (glyph probe was weak), **not** a reliable visual question answerer, and | |
| **not** grounded truth. Captions may be inaccurate, generic, or hallucinated — the training | |
| captions themselves are GPT-4V output, not native ground truth. | |
| - No safety/NSFW filtering was applied to training images. | |
| - Stage 1 is a captioner; free-form image Q&A and image-follow-up turns belong to the unfinished | |
| stage-2 snapshot and are not supported by the released weights. | |
| ## Licence and data provenance | |
| **Apache License 2.0.** The connector weights and the code in this repository are released under | |
| the Apache License 2.0 (see `LICENSE`). | |
| Training-data provenance (informational): the connector was trained on ALLaVA-4V captions and | |
| LAION-crawled images. ALLaVA-4V's card states CC-BY-NC-4.0; its uploaders do not own the image | |
| rights, per-image licences are unknown, and the captions are ChatGPT/GPT-4V-derived. We do not | |
| ship any images or captions — only the trained connector — but if you redistribute or build on | |
| this model, review those upstream terms yourself. | |
| - Base trunk: `inception42/Jais-3-0.9B` — Apache-2.0 (declared in its model card). | |
| - Encoder: `google/siglip2-base-patch16-naflex` — Apache-2.0. Not re-uploaded; cite | |
| [SigLIP 2, arXiv:2502.14786](https://huggingface.co/papers/2502.14786). | |
| - Training images/captions: `FreedomIntelligence/ALLaVA-4V` (CC-BY-NC-4.0 card) and | |
| `FreedomIntelligence/ALLaVA-4V-Arabic` (Apache-2.0 card, derived from the GPT-4V captions). | |
| See `NOTICE.md` for the full third-party list. | |
| ## Files and checksums | |
| `MANIFEST.json` holds the sha256 of every shipped file (recompute with `sha256sum`). Connector: | |
| `82c611cadf8e694d23b5edea89297a5647c61417f45e11a168f5d9cafc63dd22` (28,339,217 B). | |
| ## Citation | |
| ```bibtex | |
| @misc{dhad1visionpreview, | |
| title = {dhad-1-vision-preview: an Arabic image-captioning connector}, | |
| author = {Dhad}, | |
| year = {2026}, | |
| note = {Preview release, stage 1 only. https://huggingface.co/dhadlab/dhad-vision} | |
| } | |
| ``` | |