Blame for README.md

---
language: en
license: other
library_name: transformers
tags:
  - vision
  - vlm
  - vision-language-model
  - image-text
  - instruction-tuning
  - dpo
  - lora
  - custom-architecture
pipeline_tag: image-text-to-text
---

<div align="center">

#  Keural VLM — Vision-Language Model <sub>(PoC · V0.1)</sub>

**##Keural VLM is a proof-of-concept vision-language model developed to explore lightweight multimodal learning using a custom vision encoder. Unlike many existing VLMs, the model does not rely on a pretrained CLIP backbone. Instead, the visual encoder is trained from scratch and connected to Mistral-7B-Instruct through a lightweight projection module.
.**

<p>
  <img src="https://img.shields.io/badge/MKD_Co.,_Ltd.-Keural-6d28d9?style=for-the-badge" alt="MKD">
  <img src="https://img.shields.io/badge/status-V0.1_PoC-16a34a?style=for-the-badge" alt="Status">
</p>

<p>
  <img src="https://img.shields.io/badge/vision_encoder-24.7M-8b5cf6" alt="Params">
  <img src="https://img.shields.io/badge/LLM-Mistral--7B--Instruct-f59e0b" alt="LLM">
  <img src="https://img.shields.io/badge/PyTorch-2.x-ee4c2c?logo=pytorch&logoColor=white" alt="PyTorch">
  <img src="https://img.shields.io/badge/_Transformers-integrated-fcd34d" alt="Transformers">
  <img src="https://img.shields.io/badge/PEFT-LoRA_+_DPO-38bdf8" alt="PEFT">
  <img src="https://img.shields.io/badge/trained_on-1×_RTX_5090-22d3ee" alt="Hardware">
</p>

<br>

<img src="assets/keural_vision_encoder.gif" alt="Keural VLM architecture animation" width="98%">

<br>

*Image → CNN Stem → Adaptive Token Budget → Spatial Transformer → LevelAware Projector → Mistral-7B → Answer*

</div>

---

##  Highlights

| | |
|---|---|
|  **From scratch** | 24.7M vision encoder — **12.4× smaller** than LLaVA's CLIP encoder (307M) |
|  **Adaptive Token Budget (ATB)** | Token count is a runtime knob — dense regions get more tokens, blank regions fewer |
|  **Hierarchical Concept Tokenization (HCT)** | Every token carries a semantic level: global / region / detail |
|  **3-phase pipeline** | Vision pretraining → projector alignment → SFT + DPO |
|  **DPO improved every benchmark** | e.g. VQAv2 **+30.7pp**, ScienceQA **+14.0pp** |

---

##  What This Is

A complete Vision-Language Model (VLM) proof-of-concept built entirely from scratch — no CLIP, no pretrained vision backbone.

| Component | Details |
|---|---|
| **Vision Encoder** | 24.7M params, trained from scratch on CC3M + CC12M (~15M pairs) |
| **Projector** | `LevelAwareProjector` (384 → 2048 → 4096) |
| **LLM** | `Mistral-7B-Instruct-v0.3` (4-bit NF4 QLoRA) |
| **SFT** | LLaVA-Instruct-150K, 30,000 steps |
| **DPO** | RLHF-V dataset, 5,733 pairs, 3,000 steps |

---

##  Architecture

```
Image (256×256)
    ↓
CNN Stem  →  ATB Tokenizer  →  Spatial Transformer (12 layers, embed_dim=384)
    ↓
KeuralEncoderOutput  {tokens, level_ids, spatial_metadata, saliency_scores, pooled}
    ↓
LevelAwareProjector  (384 → 2048 → 4096)
    ↓
Visual Tokens  (N_vis × 4096)
    ↓
Mistral-7B-Instruct-v0.3  +  SFT LoRA  +  DPO LoRA
    ↓
Text Response
```

<div align="center">
<img src="assets/keural_vlm_ieee.png" alt="Keural VLM detailed architecture" width="92%">
</div>

### Key Innovations

**Adaptive Token Budget (ATB) Tokenization** — token count is a runtime parameter; dense regions get more tokens, blank regions fewer.

```python
out = encoder(image, token_budget=64)    # fast / cheap
out = encoder(image, token_budget=256)   # default
out = encoder(image, token_budget=1024)  # full fidelity
```

**Hierarchical Concept Tokenization (HCT)** — every token carries a semantic level tag.

```python
out = encoder(image)
print(out.level_ids)  # {0=global, 1=region, 2=detail}
```

---

##  Training Pipeline

<table>
<tr><th>Phase</th><th>What trains</th><th>Data</th><th>Steps</th><th>Result</th></tr>
<tr><td><b>1 · Vision Encoder</b> </td><td>Encoder from scratch</td><td>CC3M + CC12M (~15.3M)</td><td>~75,000</td><td>24.7M params · 1× RTX 5090</td></tr>
<tr><td><b>2A · Projector Align</b> </td><td>Projector + LLM LoRA (r=64), encoder frozen</td><td>LLaVA-Instruct-150K</td><td>10,000</td><td>—</td></tr>
<tr><td><b>2B · SFT</b> </td><td>LLM LoRA (r=64, α=128)</td><td>LLaVA-Instruct-150K</td><td>30,000</td><td>final loss <b>1.022</b></td></tr>
<tr><td><b>2B · DPO</b> </td><td>DPO LoRA (r=16, α=32)</td><td>RLHF-V (5,733 pairs)</td><td>3,000</td><td>loss <b>0.235</b> · reward acc <b>95%</b> · margin <b>2.11</b></td></tr>
</table>

---

##  Benchmark Results

Evaluated on 1,000 samples each (where applicable). The vision encoder is **12.4× smaller** than LLaVA's CLIP encoder (307M).

| Benchmark | Keural SFT-30K | Keural SFT+DPO | LLaVA 1.5 <sub>(307M enc)</sub> | LLaVA 1.6 <sub>(307M enc)</sub> |
|---|:---:|:---:|:---:|:---:|
| VQAv2 Accuracy | 12.9% | **43.6%** | 78.5% | 81.8% |
| POPE F1 | 66.9% | **67.0%** | 85.9% | 86.5% |
| MME Total Score | 704.3 | **838.8** | 1510.7 | 1519.3 |
| TextVQA Accuracy | 0.8% | **6.6%** | 58.2% | 64.9% |
| ScienceQA Accuracy | 39.7% | **53.7%** | 66.8% | 70.6% |

>  **POPE F1 (67.0%)** is the standout — within 19pp of LLaVA 1.6 using a **12× smaller** encoder.
>  **DPO improved every benchmark**, most dramatically VQAv2 (+30.7pp) and ScienceQA (+14.0pp).
>  **TextVQA** is low by design — no OCR training. EasyOCR integration in the GUI bridges this gap.

<div align="center">
<img src="assets/fig_all_benchmarks.png" alt="All benchmarks" width="90%">
</div>

| | |
|:---:|:---:|
| <img src="assets/fig_vqav2.png" width="100%"> | <img src="assets/fig_pope.png" width="100%"> |
| <img src="assets/fig_mme.png" width="100%"> | <img src="assets/fig_scienceqa.png" width="100%"> |

---

##  Qualitative & Saliency

The ATB tokenizer concentrates tokens on salient regions. Left → right: original · saliency heatmap · token placement.

<div align="center">
<img src="assets/cat_saliency.png" alt="Saliency — cat" width="80%">
<img src="assets/qualitative_examples.png" alt="Qualitative examples" width="90%">
</div>

---

##  DPO Training Curves

| Metric | Step 0 | Step 3000 |
|---|:---:|:---:|
| Loss | 0.694 | **0.235** |
| Reward Accuracy | ~50% | **95%** |
| Reward Margin | 0.0 | **2.11** |

<div align="center">
<img src="assets/dpo_training_curves.png" alt="DPO training curves" width="80%">
</div>

---

##  Repository Structure

```
keural-vlm-poc/
├── vision_encoder/     # Keural encoder weights (config + safetensors)
├── sft_adapter/        # SFT LoRA (30K steps)
├── dpo_adapter/        # DPO LoRA (3K steps, RLHF-V) — stacks on SFT
├── assets/             # figures, diagrams, animation
└── tokenizer.json …    # Mistral tokenizer + chat template
```

For inference, load: **Vision Encoder → Projector → Mistral-7B + SFT LoRA + DPO LoRA**.

---

##  Usage

```python
import torch
from transformers import AutoModel, AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel
from alignment.projectors import LevelAwareProjector

device = "cuda"

# 1. Vision encoder (frozen)
encoder = AutoModel.from_pretrained(
    "mkd-hika/keural-vision-encoder-poc",
    trust_remote_code=True, torch_dtype=torch.bfloat16,
).to(device).eval()

# 2. Projector
projector = LevelAwareProjector(encoder_dim=384, hidden_dim=2048, llm_dim=4096)
projector.load_state_dict(torch.load("projector.pt", map_location=device))
projector = projector.to(device, dtype=torch.bfloat16).eval()

# 3. LLM + SFT LoRA + DPO LoRA
bnb_cfg = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=torch.bfloat16,
                             bnb_4bit_use_double_quant=True, bnb_4bit_quant_type="nf4")
base_llm = AutoModelForCausalLM.from_pretrained(
    "mistralai/Mistral-7B-Instruct-v0.3",
    quantization_config=bnb_cfg, device_map="auto", torch_dtype=torch.bfloat16,
)
llm = PeftModel.from_pretrained(base_llm, "sft_adapter")        # SFT LoRA
llm = PeftModel.from_pretrained(llm, "dpo_adapter")             # DPO LoRA
llm.eval()
```

---

##  Roadmap

| Phase | Encoder Params | Hardware | Status |
|---|:---:|:---:|:---:|
| **PoC** (this model) | 24.7M | 1× RTX 5090 |  Complete (SFT + DPO) |
| Mid-level | ~183.5 | 8× H100 80 GB |  Planned |
| Commercial | ~1.1B | 64× H100 80 GB |  Future |

---

##  Citation

```bibtex
@misc{keural_vlm_2026,
  title  = {Keural VLM: Vision-Language Model with Content-Adaptive Encoding via Saliency-Guided Token Budgets},
  author = {Hika Barki and MKD Co., Ltd.},
  year   = {2026},
  },
}
```

---

## 📄 License

See [LICENSE](LICENSE) for details.

*Training data: CC3M, CC12M, LLaVA-Instruct-150K, RLHF-V — respective data licenses apply.*

<div align="center">

**MKD Co., Ltd. — 2026**

</div>