--- language: en license: mit library_name: transformers tags: - vision - vision-encoder - image-feature-extraction - image-text - contrastive-learning - custom-architecture - vlm pipeline_tag: image-feature-extraction ---

# 👁️ Keural Vision Encoder _{(PoC · V0.1)} **A 24.7M-parameter vision encoder trained _from scratch_ — no pretrained backbone, no CLIP weights. Structured tokens that plug into any LLM.**

Where the Keural encoder sits in the VLM pipeline

*The encoder (CNN Stem → ATB → Spatial Transformer) feeds the LevelAware Projector → Mistral-7B in the full [Keural VLM](https://huggingface.co/mkd-hika/keural-vlm-poc).*

--- ## ✨ Key Innovations **🎯 Adaptive Token Budget (ATB) Tokenization** — token count is a runtime parameter. Tokens are allocated to information-dense regions: a blank wall gets fewer, a dense document gets more. ```python out = encoder(image, token_budget=64) # cheap out = encoder(image, token_budget=256) # default out = encoder(image, token_budget=1024) # full fidelity ``` **🪜 Hierarchical Concept Tokenization (HCT)** — every token carries a semantic level tag: global (whole-image), region (object-scale), or detail (fine-grained). ```python out = encoder(image) print(out.level_ids) # tensor of {0=global, 1=region, 2=detail} ``` --- ## ⚙️ Model Specifications | Property | Value | |---|---| | Parameters | **24.7M** (trained from scratch) | | Architecture | CNN Stem + ATB Tokenizer + 12-layer Spatial Transformer | | Embedding dim | 384 | | Token budget | 256 (default) · runtime-adjustable | | Token split | 5% global / 25% region / 70% detail | | Precision | bfloat16 | | Training data | CC3M + CC12M (~6.9M image-text pairs) | | Hardware | 1× RTX 5090 (32 GB VRAM) | --- ## 🎨 Saliency & Token Placement The ATB tokenizer concentrates tokens on salient regions. Left → right: original · saliency heatmap · token placement (global / region / detail).

--- ## 🧪 Training **Phase 1 — Vision Encoder Pretraining ✅ COMPLETE.** Trained from scratch for ~75,000 steps on CC3M + CC12M (~6.9M image–text pairs), SigLIP-style contrastive objective, 1× RTX 5090. The frozen encoder is then integrated into the full VLM via `LevelAwareProjector` (384 → 2048 → 4096) → **Mistral-7B-Instruct-v0.3** (4-bit NF4 QLoRA), fine-tuned with SFT (LLaVA-Instruct-150K, 30K steps).

--- ## 📊 Benchmark Results — SFT-30K Downstream VLM benchmarks using this encoder (frozen) + projector + Mistral-7B at the **SFT-30K checkpoint** (supervised fine-tuning, 30,000 steps; **before** DPO). Evaluated on 1,000 samples each where applicable. The Keural encoder is **12.4× smaller** than LLaVA's CLIP encoder (307M). | Benchmark | Keural SFT-30K | LLaVA 1.5 _{(307M enc)} | LLaVA 1.6 _{(307M enc)} | |---|:---:|:---:|:---:| | VQAv2 Accuracy | **12.9%** | 78.5% | 81.8% | | POPE F1 | **66.9%** | 85.9% | 86.5% | | MME Total Score | **704.3** | 1510.7 | 1519.3 | | TextVQA Accuracy | **0.8%** | 58.2% | 64.9% | | ScienceQA Accuracy _(img) | **39.7%** | 66.8% | 70.6% | > These are the **SFT-30K** numbers (measured from `outputs/eval/sft_30k/`). Applying > DPO alignment (RLHF-V, 3K steps) improves every metric — e.g. VQAv2 → 43.6%, > ScienceQA → 53.7%, MME → 838.8. Full **SFT+DPO** results, the comparison chart, and > the complete VLM live on **[mkd-hika/keural-vlm-poc](https://huggingface.co/mkd-hika/keural-vlm-poc)**. --- ## ⚡ Adaptive Token Budget — latency vs budget Encode latency vs token budget — trade visual detail for speed at runtime, no retraining.

--- ## 📐 Output Contract Every forward pass returns a `KeuralEncoderOutput`: | Field | Shape | Description | |---|---|---| | `tokens` | `(B, N, 384)` | Token vectors | | `attention_mask` | `(B, N)` | 1 = real token, 0 = padding | | `level_ids` | `(B, N)` | 0=global, 1=region, 2=detail | | `spatial_metadata` | `(B, N, 4)` | Per-token (cx, cy, w, h) in [0,1] | | `saliency_scores` | `(B, N)` | Allocator confidence per token | | `pooled` | `(B, 384)` | Single-vector image summary | | `token_budget_used` | `(B,)` | Actual tokens used per image | --- ## 🚀 Usage ```python import torch from architecture.encoder import KeuralVisionEncoder from keural_config import KeuralConfig cfg = KeuralConfig.from_yaml("configs/keural_tiny_poc.yaml") model = KeuralVisionEncoder.from_pretrained("mkd-hika/keural-vision-encoder-poc") model.eval() from PIL import Image from torchvision import transforms transform = transforms.Compose([ transforms.Resize((256, 256)), transforms.ToTensor(), transforms.Normalize([0.5], [0.5]), ]) image = Image.open("image.jpg").convert("RGB") x = transform(image).unsqueeze(0) # (1, 3, 256, 256) with torch.no_grad(): out = model(x) print(out.tokens.shape) # (1, 256, 384) print(out.level_ids[0, :8]) # [0, 0, 1, 1, 1, 2, 2, 2] print(out.pooled.shape) # (1, 384) ``` --- ## 🗺️ Roadmap | Phase | Params | Hardware | Status | |---|:---:|:---:|:---:| | **PoC** (this model) | 24.7M | 1× RTX 5090 | ✅ Phase 1 complete | | Mid-level | ~230M | 8× H100 80 GB | 🔜 Planned | | Commercial | ~1.1B | 64× H100 80 GB | 🔮 Future | The mid-level model will use knowledge distillation from **SigLIP-400M** (natural images) and **InternViT-300M** (documents/OCR). Teachers are discarded after training — not part of the final model. --- ## 📝 Citation ```bibtex @misc{keural_vision_encoder_2026, title = {Keural Vision Encoder: Content-Adaptive Vision Encoding via Saliency-Guided Token Budgets}, author = {Barki, Hika and MKD Co., Ltd.}, year = {2026}, note = {V0.1 — Phase 1 complete}, } ``` --- ## 📄 License Code: MIT. Model weights trained on CC3M + CC12M — data licenses apply.

**MKD Co., Ltd. — 2026**