mkd-ai-dev
/

keural-VLM / README.md
mkd-hossain's picture mkd-hika's picture
Duplicate from mkd-ai/keural-VLM
3058604
|
Raw History Blame Contribute Delete
8.76 kB
---
language: en
license: other
library_name: transformers
tags:
- vision
- vlm
- vision-language-model
- image-text
- instruction-tuning
- dpo
- lora
- custom-architecture
pipeline_tag: image-text-to-text
---
<div align="center">
# Keural VLM β€” Vision-Language Model <sub>(PoC Β· V0.1)</sub>
**##Keural VLM is a proof-of-concept vision-language model developed to explore lightweight multimodal learning using a custom vision encoder. Unlike many existing VLMs, the model does not rely on a pretrained CLIP backbone. Instead, the visual encoder is trained from scratch and connected to Mistral-7B-Instruct through a lightweight projection module.
.**
<p>
<img src="https://img.shields.io/badge/MKD_Co.,_Ltd.-Keural-6d28d9?style=for-the-badge" alt="MKD">
<img src="https://img.shields.io/badge/status-V0.1_PoC-16a34a?style=for-the-badge" alt="Status">
</p>
<p>
<img src="https://img.shields.io/badge/vision_encoder-24.7M-8b5cf6" alt="Params">
<img src="https://img.shields.io/badge/LLM-Mistral--7B--Instruct-f59e0b" alt="LLM">
<img src="https://img.shields.io/badge/PyTorch-2.x-ee4c2c?logo=pytorch&logoColor=white" alt="PyTorch">
<img src="https://img.shields.io/badge/_Transformers-integrated-fcd34d" alt="Transformers">
<img src="https://img.shields.io/badge/PEFT-LoRA_+_DPO-38bdf8" alt="PEFT">
<img src="https://img.shields.io/badge/trained_on-1Γ—_RTX_5090-22d3ee" alt="Hardware">
</p>
<br>
<img src="assets/keural_vision_encoder.gif" alt="Keural VLM architecture animation" width="98%">
<br>
*Image β†’ CNN Stem β†’ Adaptive Token Budget β†’ Spatial Transformer β†’ LevelAware Projector β†’ Mistral-7B β†’ Answer*
</div>
---
## Highlights
| | |
|---|---|
| **From scratch** | 24.7M vision encoder β€” **12.4Γ— smaller** than LLaVA's CLIP encoder (307M) |
| **Adaptive Token Budget (ATB)** | Token count is a runtime knob β€” dense regions get more tokens, blank regions fewer |
| **Hierarchical Concept Tokenization (HCT)** | Every token carries a semantic level: global / region / detail |
| **3-phase pipeline** | Vision pretraining β†’ projector alignment β†’ SFT + DPO |
| **DPO improved every benchmark** | e.g. VQAv2 **+30.7pp**, ScienceQA **+14.0pp** |
---
## What This Is
A complete Vision-Language Model (VLM) proof-of-concept built entirely from scratch β€” no CLIP, no pretrained vision backbone.
| Component | Details |
|---|---|
| **Vision Encoder** | 24.7M params, trained from scratch on CC3M + CC12M (~15M pairs) |
| **Projector** | `LevelAwareProjector` (384 β†’ 2048 β†’ 4096) |
| **LLM** | `Mistral-7B-Instruct-v0.3` (4-bit NF4 QLoRA) |
| **SFT** | LLaVA-Instruct-150K, 30,000 steps |
| **DPO** | RLHF-V dataset, 5,733 pairs, 3,000 steps |
---
## Architecture
```
Image (256Γ—256)
↓
CNN Stem β†’ ATB Tokenizer β†’ Spatial Transformer (12 layers, embed_dim=384)
↓
KeuralEncoderOutput {tokens, level_ids, spatial_metadata, saliency_scores, pooled}
↓
LevelAwareProjector (384 β†’ 2048 β†’ 4096)
↓
Visual Tokens (N_vis Γ— 4096)
↓
Mistral-7B-Instruct-v0.3 + SFT LoRA + DPO LoRA
↓
Text Response
```
<div align="center">
<img src="assets/keural_vlm_ieee.png" alt="Keural VLM detailed architecture" width="92%">
</div>
### Key Innovations
**Adaptive Token Budget (ATB) Tokenization** β€” token count is a runtime parameter; dense regions get more tokens, blank regions fewer.
```python
out = encoder(image, token_budget=64) # fast / cheap
out = encoder(image, token_budget=256) # default
out = encoder(image, token_budget=1024) # full fidelity
```
**Hierarchical Concept Tokenization (HCT)** β€” every token carries a semantic level tag.
```python
out = encoder(image)
print(out.level_ids) # {0=global, 1=region, 2=detail}
```
---
## Training Pipeline
<table>
<tr><th>Phase</th><th>What trains</th><th>Data</th><th>Steps</th><th>Result</th></tr>
<tr><td><b>1 Β· Vision Encoder</b> </td><td>Encoder from scratch</td><td>CC3M + CC12M (~15.3M)</td><td>~75,000</td><td>24.7M params Β· 1Γ— RTX 5090</td></tr>
<tr><td><b>2A Β· Projector Align</b> </td><td>Projector + LLM LoRA (r=64), encoder frozen</td><td>LLaVA-Instruct-150K</td><td>10,000</td><td>β€”</td></tr>
<tr><td><b>2B Β· SFT</b> </td><td>LLM LoRA (r=64, Ξ±=128)</td><td>LLaVA-Instruct-150K</td><td>30,000</td><td>final loss <b>1.022</b></td></tr>
<tr><td><b>2B Β· DPO</b> </td><td>DPO LoRA (r=16, Ξ±=32)</td><td>RLHF-V (5,733 pairs)</td><td>3,000</td><td>loss <b>0.235</b> Β· reward acc <b>95%</b> Β· margin <b>2.11</b></td></tr>
</table>
---
## Benchmark Results
Evaluated on 1,000 samples each (where applicable). The vision encoder is **12.4Γ— smaller** than LLaVA's CLIP encoder (307M).
| Benchmark | Keural SFT-30K | Keural SFT+DPO | LLaVA 1.5 <sub>(307M enc)</sub> | LLaVA 1.6 <sub>(307M enc)</sub> |
|---|:---:|:---:|:---:|:---:|
| VQAv2 Accuracy | 12.9% | **43.6%** | 78.5% | 81.8% |
| POPE F1 | 66.9% | **67.0%** | 85.9% | 86.5% |
| MME Total Score | 704.3 | **838.8** | 1510.7 | 1519.3 |
| TextVQA Accuracy | 0.8% | **6.6%** | 58.2% | 64.9% |
| ScienceQA Accuracy | 39.7% | **53.7%** | 66.8% | 70.6% |
> **POPE F1 (67.0%)** is the standout β€” within 19pp of LLaVA 1.6 using a **12Γ— smaller** encoder.
> **DPO improved every benchmark**, most dramatically VQAv2 (+30.7pp) and ScienceQA (+14.0pp).
> **TextVQA** is low by design β€” no OCR training. EasyOCR integration in the GUI bridges this gap.
<div align="center">
<img src="assets/fig_all_benchmarks.png" alt="All benchmarks" width="90%">
</div>
| | |
|:---:|:---:|
| <img src="assets/fig_vqav2.png" width="100%"> | <img src="assets/fig_pope.png" width="100%"> |
| <img src="assets/fig_mme.png" width="100%"> | <img src="assets/fig_scienceqa.png" width="100%"> |
---
## Qualitative & Saliency
The ATB tokenizer concentrates tokens on salient regions. Left β†’ right: original Β· saliency heatmap Β· token placement.
<div align="center">
<img src="assets/cat_saliency.png" alt="Saliency β€” cat" width="80%">
<img src="assets/qualitative_examples.png" alt="Qualitative examples" width="90%">
</div>
---
## DPO Training Curves
| Metric | Step 0 | Step 3000 |
|---|:---:|:---:|
| Loss | 0.694 | **0.235** |
| Reward Accuracy | ~50% | **95%** |
| Reward Margin | 0.0 | **2.11** |
<div align="center">
<img src="assets/dpo_training_curves.png" alt="DPO training curves" width="80%">
</div>
---
## Repository Structure
```
keural-vlm-poc/
β”œβ”€β”€ vision_encoder/ # Keural encoder weights (config + safetensors)
β”œβ”€β”€ sft_adapter/ # SFT LoRA (30K steps)
β”œβ”€β”€ dpo_adapter/ # DPO LoRA (3K steps, RLHF-V) β€” stacks on SFT
β”œβ”€β”€ assets/ # figures, diagrams, animation
└── tokenizer.json … # Mistral tokenizer + chat template
```
For inference, load: **Vision Encoder β†’ Projector β†’ Mistral-7B + SFT LoRA + DPO LoRA**.
---
## Usage
```python
import torch
from transformers import AutoModel, AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel
from alignment.projectors import LevelAwareProjector
device = "cuda"
# 1. Vision encoder (frozen)
encoder = AutoModel.from_pretrained(
"mkd-hika/keural-vision-encoder-poc",
trust_remote_code=True, torch_dtype=torch.bfloat16,
).to(device).eval()
# 2. Projector
projector = LevelAwareProjector(encoder_dim=384, hidden_dim=2048, llm_dim=4096)
projector.load_state_dict(torch.load("projector.pt", map_location=device))
projector = projector.to(device, dtype=torch.bfloat16).eval()
# 3. LLM + SFT LoRA + DPO LoRA
bnb_cfg = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_use_double_quant=True, bnb_4bit_quant_type="nf4")
base_llm = AutoModelForCausalLM.from_pretrained(
"mistralai/Mistral-7B-Instruct-v0.3",
quantization_config=bnb_cfg, device_map="auto", torch_dtype=torch.bfloat16,
)
llm = PeftModel.from_pretrained(base_llm, "sft_adapter") # SFT LoRA
llm = PeftModel.from_pretrained(llm, "dpo_adapter") # DPO LoRA
llm.eval()
```
---
## Roadmap
| Phase | Encoder Params | Hardware | Status |
|---|:---:|:---:|:---:|
| **PoC** (this model) | 24.7M | 1Γ— RTX 5090 | Complete (SFT + DPO) |
| Mid-level | ~183.5 | 8Γ— H100 80 GB | Planned |
| Commercial | ~1.1B | 64Γ— H100 80 GB | Future |
---
## Citation
```bibtex
@misc{keural_vlm_2026,
title = {Keural VLM: Vision-Language Model with Content-Adaptive Encoding via Saliency-Guided Token Budgets},
author = {Hika Barki and MKD Co., Ltd.},
year = {2026},
},
}
```
---
## πŸ“„ License
See [LICENSE](LICENSE) for details.
*Training data: CC3M, CC12M, LLaVA-Instruct-150K, RLHF-V β€” respective data licenses apply.*
<div align="center">
**MKD Co., Ltd. β€” 2026**
</div>