Image-Text-to-Text
Transformers
Safetensors
English
vision
vlm
vision-language-model
image-text
instruction-tuning
dpo
lora
custom-architecture
conversational
Instructions to use mkd-ai-dev/keural-VLM with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use mkd-ai-dev/keural-VLM with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="mkd-ai-dev/keural-VLM") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("mkd-ai-dev/keural-VLM", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use mkd-ai-dev/keural-VLM with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "mkd-ai-dev/keural-VLM" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "mkd-ai-dev/keural-VLM", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/mkd-ai-dev/keural-VLM
- SGLang
How to use mkd-ai-dev/keural-VLM with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "mkd-ai-dev/keural-VLM" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "mkd-ai-dev/keural-VLM", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "mkd-ai-dev/keural-VLM" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "mkd-ai-dev/keural-VLM", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use mkd-ai-dev/keural-VLM with Docker Model Runner:
docker model run hf.co/mkd-ai-dev/keural-VLM
|
Download README.md from mkd-ai-dev/keural-VLM: direct link, hf CLI and curl.
- Browser
- Download file 8.76 kB
-
https://huggingface.co/mkd-ai-dev/keural-VLM/resolve/main/README.md
- Command line
-
hf download hf://mkd-ai-dev/keural-VLM/README.md
-
curl -L -o README.md https://huggingface.co/mkd-ai-dev/keural-VLM/resolve/main/README.md
8.76 kB
| language: en | |
| license: other | |
| library_name: transformers | |
| tags: | |
| - vision | |
| - vlm | |
| - vision-language-model | |
| - image-text | |
| - instruction-tuning | |
| - dpo | |
| - lora | |
| - custom-architecture | |
| pipeline_tag: image-text-to-text | |
| <div align="center"> | |
| # Keural VLM β Vision-Language Model <sub>(PoC Β· V0.1)</sub> | |
| **##Keural VLM is a proof-of-concept vision-language model developed to explore lightweight multimodal learning using a custom vision encoder. Unlike many existing VLMs, the model does not rely on a pretrained CLIP backbone. Instead, the visual encoder is trained from scratch and connected to Mistral-7B-Instruct through a lightweight projection module. | |
| .** | |
| <p> | |
| <img src="https://img.shields.io/badge/MKD_Co.,_Ltd.-Keural-6d28d9?style=for-the-badge" alt="MKD"> | |
| <img src="https://img.shields.io/badge/status-V0.1_PoC-16a34a?style=for-the-badge" alt="Status"> | |
| </p> | |
| <p> | |
| <img src="https://img.shields.io/badge/vision_encoder-24.7M-8b5cf6" alt="Params"> | |
| <img src="https://img.shields.io/badge/LLM-Mistral--7B--Instruct-f59e0b" alt="LLM"> | |
| <img src="https://img.shields.io/badge/PyTorch-2.x-ee4c2c?logo=pytorch&logoColor=white" alt="PyTorch"> | |
| <img src="https://img.shields.io/badge/_Transformers-integrated-fcd34d" alt="Transformers"> | |
| <img src="https://img.shields.io/badge/PEFT-LoRA_+_DPO-38bdf8" alt="PEFT"> | |
| <img src="https://img.shields.io/badge/trained_on-1Γ_RTX_5090-22d3ee" alt="Hardware"> | |
| </p> | |
| <br> | |
| <img src="assets/keural_vision_encoder.gif" alt="Keural VLM architecture animation" width="98%"> | |
| <br> | |
| *Image β CNN Stem β Adaptive Token Budget β Spatial Transformer β LevelAware Projector β Mistral-7B β Answer* | |
| </div> | |
| --- | |
| ## Highlights | |
| | | | | |
| |---|---| | |
| | **From scratch** | 24.7M vision encoder β **12.4Γ smaller** than LLaVA's CLIP encoder (307M) | | |
| | **Adaptive Token Budget (ATB)** | Token count is a runtime knob β dense regions get more tokens, blank regions fewer | | |
| | **Hierarchical Concept Tokenization (HCT)** | Every token carries a semantic level: global / region / detail | | |
| | **3-phase pipeline** | Vision pretraining β projector alignment β SFT + DPO | | |
| | **DPO improved every benchmark** | e.g. VQAv2 **+30.7pp**, ScienceQA **+14.0pp** | | |
| --- | |
| ## What This Is | |
| A complete Vision-Language Model (VLM) proof-of-concept built entirely from scratch β no CLIP, no pretrained vision backbone. | |
| | Component | Details | | |
| |---|---| | |
| | **Vision Encoder** | 24.7M params, trained from scratch on CC3M + CC12M (~15M pairs) | | |
| | **Projector** | `LevelAwareProjector` (384 β 2048 β 4096) | | |
| | **LLM** | `Mistral-7B-Instruct-v0.3` (4-bit NF4 QLoRA) | | |
| | **SFT** | LLaVA-Instruct-150K, 30,000 steps | | |
| | **DPO** | RLHF-V dataset, 5,733 pairs, 3,000 steps | | |
| --- | |
| ## Architecture | |
| ``` | |
| Image (256Γ256) | |
| β | |
| CNN Stem β ATB Tokenizer β Spatial Transformer (12 layers, embed_dim=384) | |
| β | |
| KeuralEncoderOutput {tokens, level_ids, spatial_metadata, saliency_scores, pooled} | |
| β | |
| LevelAwareProjector (384 β 2048 β 4096) | |
| β | |
| Visual Tokens (N_vis Γ 4096) | |
| β | |
| Mistral-7B-Instruct-v0.3 + SFT LoRA + DPO LoRA | |
| β | |
| Text Response | |
| ``` | |
| <div align="center"> | |
| <img src="assets/keural_vlm_ieee.png" alt="Keural VLM detailed architecture" width="92%"> | |
| </div> | |
| ### Key Innovations | |
| **Adaptive Token Budget (ATB) Tokenization** β token count is a runtime parameter; dense regions get more tokens, blank regions fewer. | |
| ```python | |
| out = encoder(image, token_budget=64) # fast / cheap | |
| out = encoder(image, token_budget=256) # default | |
| out = encoder(image, token_budget=1024) # full fidelity | |
| ``` | |
| **Hierarchical Concept Tokenization (HCT)** β every token carries a semantic level tag. | |
| ```python | |
| out = encoder(image) | |
| print(out.level_ids) # {0=global, 1=region, 2=detail} | |
| ``` | |
| --- | |
| ## Training Pipeline | |
| <table> | |
| <tr><th>Phase</th><th>What trains</th><th>Data</th><th>Steps</th><th>Result</th></tr> | |
| <tr><td><b>1 Β· Vision Encoder</b> </td><td>Encoder from scratch</td><td>CC3M + CC12M (~15.3M)</td><td>~75,000</td><td>24.7M params Β· 1Γ RTX 5090</td></tr> | |
| <tr><td><b>2A Β· Projector Align</b> </td><td>Projector + LLM LoRA (r=64), encoder frozen</td><td>LLaVA-Instruct-150K</td><td>10,000</td><td>β</td></tr> | |
| <tr><td><b>2B Β· SFT</b> </td><td>LLM LoRA (r=64, Ξ±=128)</td><td>LLaVA-Instruct-150K</td><td>30,000</td><td>final loss <b>1.022</b></td></tr> | |
| <tr><td><b>2B Β· DPO</b> </td><td>DPO LoRA (r=16, Ξ±=32)</td><td>RLHF-V (5,733 pairs)</td><td>3,000</td><td>loss <b>0.235</b> Β· reward acc <b>95%</b> Β· margin <b>2.11</b></td></tr> | |
| </table> | |
| --- | |
| ## Benchmark Results | |
| Evaluated on 1,000 samples each (where applicable). The vision encoder is **12.4Γ smaller** than LLaVA's CLIP encoder (307M). | |
| | Benchmark | Keural SFT-30K | Keural SFT+DPO | LLaVA 1.5 <sub>(307M enc)</sub> | LLaVA 1.6 <sub>(307M enc)</sub> | | |
| |---|:---:|:---:|:---:|:---:| | |
| | VQAv2 Accuracy | 12.9% | **43.6%** | 78.5% | 81.8% | | |
| | POPE F1 | 66.9% | **67.0%** | 85.9% | 86.5% | | |
| | MME Total Score | 704.3 | **838.8** | 1510.7 | 1519.3 | | |
| | TextVQA Accuracy | 0.8% | **6.6%** | 58.2% | 64.9% | | |
| | ScienceQA Accuracy | 39.7% | **53.7%** | 66.8% | 70.6% | | |
| > **POPE F1 (67.0%)** is the standout β within 19pp of LLaVA 1.6 using a **12Γ smaller** encoder. | |
| > **DPO improved every benchmark**, most dramatically VQAv2 (+30.7pp) and ScienceQA (+14.0pp). | |
| > **TextVQA** is low by design β no OCR training. EasyOCR integration in the GUI bridges this gap. | |
| <div align="center"> | |
| <img src="assets/fig_all_benchmarks.png" alt="All benchmarks" width="90%"> | |
| </div> | |
| | | | | |
| |:---:|:---:| | |
| | <img src="assets/fig_vqav2.png" width="100%"> | <img src="assets/fig_pope.png" width="100%"> | | |
| | <img src="assets/fig_mme.png" width="100%"> | <img src="assets/fig_scienceqa.png" width="100%"> | | |
| --- | |
| ## Qualitative & Saliency | |
| The ATB tokenizer concentrates tokens on salient regions. Left β right: original Β· saliency heatmap Β· token placement. | |
| <div align="center"> | |
| <img src="assets/cat_saliency.png" alt="Saliency β cat" width="80%"> | |
| <img src="assets/qualitative_examples.png" alt="Qualitative examples" width="90%"> | |
| </div> | |
| --- | |
| ## DPO Training Curves | |
| | Metric | Step 0 | Step 3000 | | |
| |---|:---:|:---:| | |
| | Loss | 0.694 | **0.235** | | |
| | Reward Accuracy | ~50% | **95%** | | |
| | Reward Margin | 0.0 | **2.11** | | |
| <div align="center"> | |
| <img src="assets/dpo_training_curves.png" alt="DPO training curves" width="80%"> | |
| </div> | |
| --- | |
| ## Repository Structure | |
| ``` | |
| keural-vlm-poc/ | |
| βββ vision_encoder/ # Keural encoder weights (config + safetensors) | |
| βββ sft_adapter/ # SFT LoRA (30K steps) | |
| βββ dpo_adapter/ # DPO LoRA (3K steps, RLHF-V) β stacks on SFT | |
| βββ assets/ # figures, diagrams, animation | |
| βββ tokenizer.json β¦ # Mistral tokenizer + chat template | |
| ``` | |
| For inference, load: **Vision Encoder β Projector β Mistral-7B + SFT LoRA + DPO LoRA**. | |
| --- | |
| ## Usage | |
| ```python | |
| import torch | |
| from transformers import AutoModel, AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig | |
| from peft import PeftModel | |
| from alignment.projectors import LevelAwareProjector | |
| device = "cuda" | |
| # 1. Vision encoder (frozen) | |
| encoder = AutoModel.from_pretrained( | |
| "mkd-hika/keural-vision-encoder-poc", | |
| trust_remote_code=True, torch_dtype=torch.bfloat16, | |
| ).to(device).eval() | |
| # 2. Projector | |
| projector = LevelAwareProjector(encoder_dim=384, hidden_dim=2048, llm_dim=4096) | |
| projector.load_state_dict(torch.load("projector.pt", map_location=device)) | |
| projector = projector.to(device, dtype=torch.bfloat16).eval() | |
| # 3. LLM + SFT LoRA + DPO LoRA | |
| bnb_cfg = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=torch.bfloat16, | |
| bnb_4bit_use_double_quant=True, bnb_4bit_quant_type="nf4") | |
| base_llm = AutoModelForCausalLM.from_pretrained( | |
| "mistralai/Mistral-7B-Instruct-v0.3", | |
| quantization_config=bnb_cfg, device_map="auto", torch_dtype=torch.bfloat16, | |
| ) | |
| llm = PeftModel.from_pretrained(base_llm, "sft_adapter") # SFT LoRA | |
| llm = PeftModel.from_pretrained(llm, "dpo_adapter") # DPO LoRA | |
| llm.eval() | |
| ``` | |
| --- | |
| ## Roadmap | |
| | Phase | Encoder Params | Hardware | Status | | |
| |---|:---:|:---:|:---:| | |
| | **PoC** (this model) | 24.7M | 1Γ RTX 5090 | Complete (SFT + DPO) | | |
| | Mid-level | ~183.5 | 8Γ H100 80 GB | Planned | | |
| | Commercial | ~1.1B | 64Γ H100 80 GB | Future | | |
| --- | |
| ## Citation | |
| ```bibtex | |
| @misc{keural_vlm_2026, | |
| title = {Keural VLM: Vision-Language Model with Content-Adaptive Encoding via Saliency-Guided Token Budgets}, | |
| author = {Hika Barki and MKD Co., Ltd.}, | |
| year = {2026}, | |
| }, | |
| } | |
| ``` | |
| --- | |
| ## π License | |
| See [LICENSE](LICENSE) for details. | |
| *Training data: CC3M, CC12M, LLaVA-Instruct-150K, RLHF-V β respective data licenses apply.* | |
| <div align="center"> | |
| **MKD Co., Ltd. β 2026** | |
| </div> | |