--- license: apache-2.0 language: - ar - en library_name: transformers pipeline_tag: image-text-to-text base_model: - inception42/Jais-3-0.9B - google/siglip2-base-patch16-naflex tags: - arabic - vision - image-captioning - image-understanding - siglip2 - jais --- # dhad-1-vision-preview (رؤية) An **Arabic image-captioning model**: give it a picture, it writes a short Arabic description. A frozen SigLIP2 vision encoder turns the image into 576 patch features; a small trained **connector** maps them to 144 soft tokens (64 in low-token mode); the tokens are spliced into a frozen Jais-3-0.9B language model through two learned marker vectors, and the model writes the caption. **Preview.** Only stage 1 (the connector) was finished. Its output is caption-style text and it did not go through a formal release gate. An unfinished stage-2 snapshot (a vision LoRA that answers free-form questions about an image, stopped at step 675 of 1,435) is included as an extra, clearly labelled `UNFINISHED`. ## What is in this repository | File | What | |---|---| | `connector.pt` | **The model.** A `torch.save` dict with `connector` (7,080,961 params) and `markers` (2 × 1536 = 3,072 params). | | `MANIFEST.json` | Provenance: sources, sha256, parameter counts, training recipe, gate notes. | | `requirements.txt` | Runtime Python packages. | | `LICENSE` | Apache License 2.0 text. | | `code/connector.py` | The connector architecture (pixel-shuffle → MLP → L2 × learnable scale). | | `code/graft.py` | Splices the soft tokens into the frozen Jais trunk (`inputs_embeds`) and generates. | | `code/infer.py` | `describe(image, prompt)` helper + CLI. | | `code/encoder.py` | Runs the frozen SigLIP2 encoder on raw images → `[576, 768]` features. | | `stage2-step675/` | UNFINISHED stage-2 snapshot: its own `connector.pt` + `vision_lora/` (rank-32 LoRA, 27,983,872 params). See `stage2-step675/UNFINISHED.md`. | The base models are not re-uploaded: load `inception42/Jais-3-0.9B` (Apache-2.0) and `google/siglip2-base-patch16-naflex` (Apache-2.0) yourself. ## Parameter count The full runtime is **1,171,213,057 parameters** (≈1.17B): the frozen Jais trunk (1,078,285,824) + the frozen SigLIP2 vision tower used (85,843,200) + the connector and markers (7,084,033). Only the 28 MB connector file is new; the rest is public base weights. With the unfinished stage-2 LoRA it would be 1,199,196,929. ## How it was trained (stage 1) - **Trainable:** connector + 2 marker vectors only (7.08M). The SigLIP2 encoder and the Jais trunk are frozen end-to-end. - **Data:** 29,800 train + 200 held-out pairs of (image, Arabic caption) from `FreedomIntelligence/ALLaVA-4V` images with `FreedomIntelligence/ALLaVA-4V-Arabic` captions (machine translation of GPT-4V captions). Captions trimmed to their first 1–2 sentences. See `MANIFEST.json`. - **Recipe:** cosine LR 1e-3, 64 samples/step, 2 epochs (930 steps), bf16, gradient checkpointing, 30 % of steps in 64-token mode. - **Probe at step 930:** caption loss on the true image 2.4456 vs shuffled image 3.0071 (**+18.7 % gain**). Owner-tested and accepted 2026-11-14; **no formal gate was run**. ## Usage The weights are a raw state dict, so you also need the `code/` folder (shipped here) and the two base models (downloaded automatically on first run). ### 1. Download this repository ```bash hf download dhadlab/dhad-vision --local-dir dhad-1-vision-preview-1.2b cd dhad-1-vision-preview-1.2b ``` Or download it from the **Files** tab of this page. ### 2. Install ```bash pip install -r requirements.txt ``` ### 3. Describe an image ```bash python code/infer.py photo.jpg python code/infer.py photo.jpg "صف هذه الصورة بالتفصيل" ``` Real output from a test photo: > الصورة تظهر مجموعة من أشجار التفاح في بيئة خريفية. الأشجار مغطاة بأوراق بنية داكنة، مما يشير إلى أنها في أواخر الموسم. ### Python ```python import sys sys.path.insert(0, "code") import infer g = infer.load_graft(device="cuda") # loads Jais + connector.pt (once) print(infer.describe("photo.jpg", "صِف هذه الصورة", graft=g)) print(infer.describe("b.jpg", graft=g)) # reuse the loaded models ``` `infer.describe` also accepts a path, a PIL image, raw bytes, or cached SigLIP2 features (`[576, 768]` fp16). Defaults: SigLIP2 encoder on CPU (1–2 s per image), 96 new tokens, Arabic prompt `صِف هذه الصورة`. Environment options: `DEVICE=cuda` (encoder on GPU), `LOW=1` (64-token mode), `CONNECTOR=`, `JAIS_PATH` / `SIGLIP2_PATH` (local base-model copies). Expect caption-style output, not chat: this model **describes** an image; it does not reliably answer questions about it (that was the unfinished stage 2). ## Intended use and limits - Arabic (and some English) **image captioning / description** for research and demos. - **Not** an OCR engine (glyph probe was weak), **not** a reliable visual question answerer, and **not** grounded truth. Captions may be inaccurate, generic, or hallucinated — the training captions themselves are GPT-4V output, not native ground truth. - No safety/NSFW filtering was applied to training images. - Stage 1 is a captioner; free-form image Q&A and image-follow-up turns belong to the unfinished stage-2 snapshot and are not supported by the released weights. ## Licence and data provenance **Apache License 2.0.** The connector weights and the code in this repository are released under the Apache License 2.0 (see `LICENSE`). Training-data provenance (informational): the connector was trained on ALLaVA-4V captions and LAION-crawled images. ALLaVA-4V's card states CC-BY-NC-4.0; its uploaders do not own the image rights, per-image licences are unknown, and the captions are ChatGPT/GPT-4V-derived. We do not ship any images or captions — only the trained connector — but if you redistribute or build on this model, review those upstream terms yourself. - Base trunk: `inception42/Jais-3-0.9B` — Apache-2.0 (declared in its model card). - Encoder: `google/siglip2-base-patch16-naflex` — Apache-2.0. Not re-uploaded; cite [SigLIP 2, arXiv:2502.14786](https://huggingface.co/papers/2502.14786). - Training images/captions: `FreedomIntelligence/ALLaVA-4V` (CC-BY-NC-4.0 card) and `FreedomIntelligence/ALLaVA-4V-Arabic` (Apache-2.0 card, derived from the GPT-4V captions). See `NOTICE.md` for the full third-party list. ## Files and checksums `MANIFEST.json` holds the sha256 of every shipped file (recompute with `sha256sum`). Connector: `82c611cadf8e694d23b5edea89297a5647c61417f45e11a168f5d9cafc63dd22` (28,339,217 B). ## Citation ```bibtex @misc{dhad1visionpreview, title = {dhad-1-vision-preview: an Arabic image-captioning connector}, author = {Dhad}, year = {2026}, note = {Preview release, stage 1 only. https://huggingface.co/dhadlab/dhad-vision} } ```