Image Feature Extraction
Transformers
Safetensors
English
keural_vision
vision
vision-encoder
image-text
contrastive-learning
custom-architecture
vlm
Instructions to use mkd-dev/keural-vision-encoder-poc with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use mkd-dev/keural-vision-encoder-poc with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-feature-extraction", model="mkd-dev/keural-vision-encoder-poc")# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("mkd-dev/keural-vision-encoder-poc", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Commit ·
96faa83
0
Parent(s):
Duplicate from mkd-hika/keural-vision-encoder-poc
Browse files- .gitattributes +51 -0
- README.md +220 -0
- assets/keural_vision_encoder.gif +3 -0
- assets/token_budget_curve.png +3 -0
- assets/training_loss_curves.png +3 -0
- checkpoint-37000/config.json +133 -0
- checkpoint-37000/model.safetensors +3 -0
- checkpoint-37000/training_state.pt +3 -0
- checkpoint-44000/config.json +133 -0
- checkpoint-44000/model.safetensors +3 -0
- checkpoint-44000/training_state.pt +3 -0
- config.json +133 -0
- model.safetensors +3 -0
- preprocessor_config.json +15 -0
- saliency_viz/step_28k/cat_saliency.png +3 -0
- saliency_viz/step_28k/dog_saliency.png +3 -0
- saliency_viz/step_28k/horse_saliency.png +3 -0
- saliency_viz/step_35k/cat_saliency.png +3 -0
- saliency_viz/step_35k/dog_saliency.png +3 -0
- saliency_viz/step_35k/horse_saliency.png +3 -0
- saliency_viz/step_36k/cat_saliency.png +3 -0
- saliency_viz/step_36k/dog_saliency.png +3 -0
- saliency_viz/step_36k/horse_saliency.png +3 -0
- saliency_viz/step_36k_cpu/cat_saliency.png +3 -0
- saliency_viz/step_36k_cpu/dog_saliency.png +3 -0
- saliency_viz/step_36k_cpu/horse_saliency.png +3 -0
- training_state.pt +3 -0
.gitattributes
ADDED
|
@@ -0,0 +1,51 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
*.7z filter=lfs diff=lfs merge=lfs -text
|
| 2 |
+
*.arrow filter=lfs diff=lfs merge=lfs -text
|
| 3 |
+
*.bin filter=lfs diff=lfs merge=lfs -text
|
| 4 |
+
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
| 5 |
+
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
| 6 |
+
*.ftz filter=lfs diff=lfs merge=lfs -text
|
| 7 |
+
*.gz filter=lfs diff=lfs merge=lfs -text
|
| 8 |
+
*.h5 filter=lfs diff=lfs merge=lfs -text
|
| 9 |
+
*.joblib filter=lfs diff=lfs merge=lfs -text
|
| 10 |
+
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
| 11 |
+
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
| 12 |
+
*.model filter=lfs diff=lfs merge=lfs -text
|
| 13 |
+
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
| 14 |
+
*.npy filter=lfs diff=lfs merge=lfs -text
|
| 15 |
+
*.npz filter=lfs diff=lfs merge=lfs -text
|
| 16 |
+
*.onnx filter=lfs diff=lfs merge=lfs -text
|
| 17 |
+
*.ot filter=lfs diff=lfs merge=lfs -text
|
| 18 |
+
*.parquet filter=lfs diff=lfs merge=lfs -text
|
| 19 |
+
*.pb filter=lfs diff=lfs merge=lfs -text
|
| 20 |
+
*.pickle filter=lfs diff=lfs merge=lfs -text
|
| 21 |
+
*.pkl filter=lfs diff=lfs merge=lfs -text
|
| 22 |
+
*.pt filter=lfs diff=lfs merge=lfs -text
|
| 23 |
+
*.pth filter=lfs diff=lfs merge=lfs -text
|
| 24 |
+
*.rar filter=lfs diff=lfs merge=lfs -text
|
| 25 |
+
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
| 26 |
+
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
| 27 |
+
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
| 28 |
+
*.tar filter=lfs diff=lfs merge=lfs -text
|
| 29 |
+
*.tflite filter=lfs diff=lfs merge=lfs -text
|
| 30 |
+
*.tgz filter=lfs diff=lfs merge=lfs -text
|
| 31 |
+
*.wasm filter=lfs diff=lfs merge=lfs -text
|
| 32 |
+
*.xz filter=lfs diff=lfs merge=lfs -text
|
| 33 |
+
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
+
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
+
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
+
saliency_viz/step_28k/cat_saliency.png filter=lfs diff=lfs merge=lfs -text
|
| 37 |
+
saliency_viz/step_28k/dog_saliency.png filter=lfs diff=lfs merge=lfs -text
|
| 38 |
+
saliency_viz/step_28k/horse_saliency.png filter=lfs diff=lfs merge=lfs -text
|
| 39 |
+
saliency_viz/step_35k/cat_saliency.png filter=lfs diff=lfs merge=lfs -text
|
| 40 |
+
saliency_viz/step_35k/dog_saliency.png filter=lfs diff=lfs merge=lfs -text
|
| 41 |
+
saliency_viz/step_35k/horse_saliency.png filter=lfs diff=lfs merge=lfs -text
|
| 42 |
+
saliency_viz/step_36k/cat_saliency.png filter=lfs diff=lfs merge=lfs -text
|
| 43 |
+
saliency_viz/step_36k/dog_saliency.png filter=lfs diff=lfs merge=lfs -text
|
| 44 |
+
saliency_viz/step_36k/horse_saliency.png filter=lfs diff=lfs merge=lfs -text
|
| 45 |
+
saliency_viz/step_36k_cpu/cat_saliency.png filter=lfs diff=lfs merge=lfs -text
|
| 46 |
+
saliency_viz/step_36k_cpu/dog_saliency.png filter=lfs diff=lfs merge=lfs -text
|
| 47 |
+
saliency_viz/step_36k_cpu/horse_saliency.png filter=lfs diff=lfs merge=lfs -text
|
| 48 |
+
assets/keural_vision_encoder.gif filter=lfs diff=lfs merge=lfs -text
|
| 49 |
+
assets/training_loss_curves.png filter=lfs diff=lfs merge=lfs -text
|
| 50 |
+
assets/benchmark_results.png filter=lfs diff=lfs merge=lfs -text
|
| 51 |
+
assets/token_budget_curve.png filter=lfs diff=lfs merge=lfs -text
|
README.md
ADDED
|
@@ -0,0 +1,220 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
language: en
|
| 3 |
+
license: mit
|
| 4 |
+
library_name: transformers
|
| 5 |
+
tags:
|
| 6 |
+
- vision
|
| 7 |
+
- vision-encoder
|
| 8 |
+
- image-feature-extraction
|
| 9 |
+
- image-text
|
| 10 |
+
- contrastive-learning
|
| 11 |
+
- custom-architecture
|
| 12 |
+
- vlm
|
| 13 |
+
pipeline_tag: image-feature-extraction
|
| 14 |
+
---
|
| 15 |
+
|
| 16 |
+
<div align="center">
|
| 17 |
+
|
| 18 |
+
# 👁️ Keural Vision Encoder <sub>(PoC · V0.1)</sub>
|
| 19 |
+
|
| 20 |
+
**A 24.7M-parameter vision encoder trained _from scratch_ — no pretrained backbone, no CLIP weights. Structured tokens that plug into any LLM.**
|
| 21 |
+
|
| 22 |
+
<p>
|
| 23 |
+
<img src="https://img.shields.io/badge/MKD_Co.,_Ltd.-Keural-6d28d9?style=for-the-badge" alt="MKD">
|
| 24 |
+
<img src="https://img.shields.io/badge/status-V0.1_PoC-16a34a?style=for-the-badge" alt="Status">
|
| 25 |
+
</p>
|
| 26 |
+
|
| 27 |
+
<p>
|
| 28 |
+
<img src="https://img.shields.io/badge/params-24.7M-8b5cf6" alt="Params">
|
| 29 |
+
<img src="https://img.shields.io/badge/from-scratch-ec4899" alt="From scratch">
|
| 30 |
+
<img src="https://img.shields.io/badge/embed_dim-384-f59e0b" alt="Dim">
|
| 31 |
+
<img src="https://img.shields.io/badge/token_budget-adaptive-22d3ee" alt="ATB">
|
| 32 |
+
<img src="https://img.shields.io/badge/PyTorch-2.x-ee4c2c?logo=pytorch&logoColor=white" alt="PyTorch">
|
| 33 |
+
<img src="https://img.shields.io/badge/trained_on-1×_RTX_5090-38bdf8" alt="Hardware">
|
| 34 |
+
</p>
|
| 35 |
+
|
| 36 |
+
<br>
|
| 37 |
+
|
| 38 |
+
<img src="assets/keural_vision_encoder.gif" alt="Where the Keural encoder sits in the VLM pipeline" width="98%">
|
| 39 |
+
|
| 40 |
+
<br>
|
| 41 |
+
|
| 42 |
+
*The encoder (CNN Stem → ATB → Spatial Transformer) feeds the LevelAware Projector → Mistral-7B in the full [Keural VLM](https://huggingface.co/mkd-hika/keural-vlm-poc).*
|
| 43 |
+
|
| 44 |
+
</div>
|
| 45 |
+
|
| 46 |
+
---
|
| 47 |
+
|
| 48 |
+
## ✨ Key Innovations
|
| 49 |
+
|
| 50 |
+
**🎯 Adaptive Token Budget (ATB) Tokenization** — token count is a runtime parameter. Tokens are allocated to information-dense regions: a blank wall gets fewer, a dense document gets more.
|
| 51 |
+
|
| 52 |
+
```python
|
| 53 |
+
out = encoder(image, token_budget=64) # cheap
|
| 54 |
+
out = encoder(image, token_budget=256) # default
|
| 55 |
+
out = encoder(image, token_budget=1024) # full fidelity
|
| 56 |
+
```
|
| 57 |
+
|
| 58 |
+
**🪜 Hierarchical Concept Tokenization (HCT)** — every token carries a semantic level tag: global (whole-image), region (object-scale), or detail (fine-grained).
|
| 59 |
+
|
| 60 |
+
```python
|
| 61 |
+
out = encoder(image)
|
| 62 |
+
print(out.level_ids) # tensor of {0=global, 1=region, 2=detail}
|
| 63 |
+
```
|
| 64 |
+
|
| 65 |
+
---
|
| 66 |
+
|
| 67 |
+
## ⚙️ Model Specifications
|
| 68 |
+
|
| 69 |
+
| Property | Value |
|
| 70 |
+
|---|---|
|
| 71 |
+
| Parameters | **24.7M** (trained from scratch) |
|
| 72 |
+
| Architecture | CNN Stem + ATB Tokenizer + 12-layer Spatial Transformer |
|
| 73 |
+
| Embedding dim | 384 |
|
| 74 |
+
| Token budget | 256 (default) · runtime-adjustable |
|
| 75 |
+
| Token split | 5% global / 25% region / 70% detail |
|
| 76 |
+
| Precision | bfloat16 |
|
| 77 |
+
| Training data | CC3M + CC12M (~6.9M image-text pairs) |
|
| 78 |
+
| Hardware | 1× RTX 5090 (32 GB VRAM) |
|
| 79 |
+
|
| 80 |
+
---
|
| 81 |
+
|
| 82 |
+
## 🎨 Saliency & Token Placement
|
| 83 |
+
|
| 84 |
+
The ATB tokenizer concentrates tokens on salient regions. Left → right: original · saliency heatmap · token placement (global / region / detail).
|
| 85 |
+
|
| 86 |
+
<div align="center">
|
| 87 |
+
<img src="saliency_viz/step_36k/cat_saliency.png" alt="Saliency and token placement" width="88%">
|
| 88 |
+
</div>
|
| 89 |
+
|
| 90 |
+
---
|
| 91 |
+
|
| 92 |
+
## 🧪 Training
|
| 93 |
+
|
| 94 |
+
**Phase 1 — Vision Encoder Pretraining ✅ COMPLETE.** Trained from scratch for
|
| 95 |
+
~75,000 steps on CC3M + CC12M (~6.9M image–text pairs), SigLIP-style contrastive
|
| 96 |
+
objective, 1× RTX 5090. The frozen encoder is then integrated into the full VLM via
|
| 97 |
+
`LevelAwareProjector` (384 → 2048 → 4096) → **Mistral-7B-Instruct-v0.3** (4-bit NF4
|
| 98 |
+
QLoRA), fine-tuned with SFT (LLaVA-Instruct-150K, 30K steps).
|
| 99 |
+
|
| 100 |
+
<div align="center">
|
| 101 |
+
<img src="assets/training_loss_curves.png" alt="Training loss curves" width="88%">
|
| 102 |
+
</div>
|
| 103 |
+
|
| 104 |
+
---
|
| 105 |
+
|
| 106 |
+
## 📊 Benchmark Results — SFT-30K
|
| 107 |
+
|
| 108 |
+
Downstream VLM benchmarks using this encoder (frozen) + projector + Mistral-7B at the
|
| 109 |
+
**SFT-30K checkpoint** (supervised fine-tuning, 30,000 steps; **before** DPO).
|
| 110 |
+
Evaluated on 1,000 samples each where applicable. The Keural encoder is **12.4×
|
| 111 |
+
smaller** than LLaVA's CLIP encoder (307M).
|
| 112 |
+
|
| 113 |
+
| Benchmark | Keural SFT-30K | LLaVA 1.5 <sub>(307M enc)</sub> | LLaVA 1.6 <sub>(307M enc)</sub> |
|
| 114 |
+
|---|:---:|:---:|:---:|
|
| 115 |
+
| VQAv2 Accuracy | **12.9%** | 78.5% | 81.8% |
|
| 116 |
+
| POPE F1 | **66.9%** | 85.9% | 86.5% |
|
| 117 |
+
| MME Total Score | **704.3** | 1510.7 | 1519.3 |
|
| 118 |
+
| TextVQA Accuracy | **0.8%** | 58.2% | 64.9% |
|
| 119 |
+
| ScienceQA Accuracy <sub>(img)</sub> | **39.7%** | 66.8% | 70.6% |
|
| 120 |
+
|
| 121 |
+
> These are the **SFT-30K** numbers (measured from `outputs/eval/sft_30k/`). Applying
|
| 122 |
+
> DPO alignment (RLHF-V, 3K steps) improves every metric — e.g. VQAv2 → 43.6%,
|
| 123 |
+
> ScienceQA → 53.7%, MME → 838.8. Full **SFT+DPO** results, the comparison chart, and
|
| 124 |
+
> the complete VLM live on **[mkd-hika/keural-vlm-poc](https://huggingface.co/mkd-hika/keural-vlm-poc)**.
|
| 125 |
+
|
| 126 |
+
---
|
| 127 |
+
|
| 128 |
+
## ⚡ Adaptive Token Budget — latency vs budget
|
| 129 |
+
|
| 130 |
+
Encode latency vs token budget — trade visual detail for speed at runtime, no retraining.
|
| 131 |
+
|
| 132 |
+
<div align="center">
|
| 133 |
+
<img src="assets/token_budget_curve.png" alt="ATB token budget vs latency" width="80%">
|
| 134 |
+
</div>
|
| 135 |
+
|
| 136 |
+
---
|
| 137 |
+
|
| 138 |
+
## 📐 Output Contract
|
| 139 |
+
|
| 140 |
+
Every forward pass returns a `KeuralEncoderOutput`:
|
| 141 |
+
|
| 142 |
+
| Field | Shape | Description |
|
| 143 |
+
|---|---|---|
|
| 144 |
+
| `tokens` | `(B, N, 384)` | Token vectors |
|
| 145 |
+
| `attention_mask` | `(B, N)` | 1 = real token, 0 = padding |
|
| 146 |
+
| `level_ids` | `(B, N)` | 0=global, 1=region, 2=detail |
|
| 147 |
+
| `spatial_metadata` | `(B, N, 4)` | Per-token (cx, cy, w, h) in [0,1] |
|
| 148 |
+
| `saliency_scores` | `(B, N)` | Allocator confidence per token |
|
| 149 |
+
| `pooled` | `(B, 384)` | Single-vector image summary |
|
| 150 |
+
| `token_budget_used` | `(B,)` | Actual tokens used per image |
|
| 151 |
+
|
| 152 |
+
---
|
| 153 |
+
|
| 154 |
+
## 🚀 Usage
|
| 155 |
+
|
| 156 |
+
```python
|
| 157 |
+
import torch
|
| 158 |
+
from architecture.encoder import KeuralVisionEncoder
|
| 159 |
+
from keural_config import KeuralConfig
|
| 160 |
+
|
| 161 |
+
cfg = KeuralConfig.from_yaml("configs/keural_tiny_poc.yaml")
|
| 162 |
+
model = KeuralVisionEncoder.from_pretrained("mkd-hika/keural-vision-encoder-poc")
|
| 163 |
+
model.eval()
|
| 164 |
+
|
| 165 |
+
from PIL import Image
|
| 166 |
+
from torchvision import transforms
|
| 167 |
+
|
| 168 |
+
transform = transforms.Compose([
|
| 169 |
+
transforms.Resize((256, 256)),
|
| 170 |
+
transforms.ToTensor(),
|
| 171 |
+
transforms.Normalize([0.5], [0.5]),
|
| 172 |
+
])
|
| 173 |
+
|
| 174 |
+
image = Image.open("image.jpg").convert("RGB")
|
| 175 |
+
x = transform(image).unsqueeze(0) # (1, 3, 256, 256)
|
| 176 |
+
|
| 177 |
+
with torch.no_grad():
|
| 178 |
+
out = model(x)
|
| 179 |
+
|
| 180 |
+
print(out.tokens.shape) # (1, 256, 384)
|
| 181 |
+
print(out.level_ids[0, :8]) # [0, 0, 1, 1, 1, 2, 2, 2]
|
| 182 |
+
print(out.pooled.shape) # (1, 384)
|
| 183 |
+
```
|
| 184 |
+
|
| 185 |
+
---
|
| 186 |
+
|
| 187 |
+
## 🗺️ Roadmap
|
| 188 |
+
|
| 189 |
+
| Phase | Params | Hardware | Status |
|
| 190 |
+
|---|:---:|:---:|:---:|
|
| 191 |
+
| **PoC** (this model) | 24.7M | 1× RTX 5090 | ✅ Phase 1 complete |
|
| 192 |
+
| Mid-level | ~230M | 8× H100 80 GB | 🔜 Planned |
|
| 193 |
+
| Commercial | ~1.1B | 64× H100 80 GB | 🔮 Future |
|
| 194 |
+
|
| 195 |
+
The mid-level model will use knowledge distillation from **SigLIP-400M** (natural images) and **InternViT-300M** (documents/OCR). Teachers are discarded after training — not part of the final model.
|
| 196 |
+
|
| 197 |
+
---
|
| 198 |
+
|
| 199 |
+
## 📝 Citation
|
| 200 |
+
|
| 201 |
+
```bibtex
|
| 202 |
+
@misc{keural_vision_encoder_2026,
|
| 203 |
+
title = {Keural Vision Encoder: Content-Adaptive Vision Encoding via Saliency-Guided Token Budgets},
|
| 204 |
+
author = {Barki, Hika and MKD Co., Ltd.},
|
| 205 |
+
year = {2026},
|
| 206 |
+
note = {V0.1 — Phase 1 complete},
|
| 207 |
+
}
|
| 208 |
+
```
|
| 209 |
+
|
| 210 |
+
---
|
| 211 |
+
|
| 212 |
+
## 📄 License
|
| 213 |
+
|
| 214 |
+
Code: MIT. Model weights trained on CC3M + CC12M — data licenses apply.
|
| 215 |
+
|
| 216 |
+
<div align="center">
|
| 217 |
+
|
| 218 |
+
**MKD Co., Ltd. — 2026**
|
| 219 |
+
|
| 220 |
+
</div>
|
assets/keural_vision_encoder.gif
ADDED
|
Git LFS Details
|
assets/token_budget_curve.png
ADDED
|
Git LFS Details
|
assets/training_loss_curves.png
ADDED
|
Git LFS Details
|
checkpoint-37000/config.json
ADDED
|
@@ -0,0 +1,133 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"architectures": [
|
| 3 |
+
"KeuralVisionModel"
|
| 4 |
+
],
|
| 5 |
+
"default_token_budget": 256,
|
| 6 |
+
"depth": 12,
|
| 7 |
+
"dtype": "float32",
|
| 8 |
+
"embed_dim": 384,
|
| 9 |
+
"keural_config": {
|
| 10 |
+
"data": {
|
| 11 |
+
"captions_path": "data/indexes/cc3m_wds_train.tsv",
|
| 12 |
+
"dataset_name": "cc3m+cc12m",
|
| 13 |
+
"extra_sources": [
|
| 14 |
+
{
|
| 15 |
+
"captions_path": "data/indexes/cc12m_wds_train.tsv",
|
| 16 |
+
"images_root": "data/cc12m"
|
| 17 |
+
}
|
| 18 |
+
],
|
| 19 |
+
"image_buckets": [
|
| 20 |
+
[
|
| 21 |
+
224,
|
| 22 |
+
224
|
| 23 |
+
],
|
| 24 |
+
[
|
| 25 |
+
256,
|
| 26 |
+
256
|
| 27 |
+
],
|
| 28 |
+
[
|
| 29 |
+
320,
|
| 30 |
+
320
|
| 31 |
+
]
|
| 32 |
+
],
|
| 33 |
+
"images_root": "data/cc3m",
|
| 34 |
+
"max_caption_length": 64,
|
| 35 |
+
"num_workers": 4,
|
| 36 |
+
"pixel_mean": [
|
| 37 |
+
0.485,
|
| 38 |
+
0.456,
|
| 39 |
+
0.406
|
| 40 |
+
],
|
| 41 |
+
"pixel_std": [
|
| 42 |
+
0.229,
|
| 43 |
+
0.224,
|
| 44 |
+
0.225
|
| 45 |
+
],
|
| 46 |
+
"text_encoder_name": "sentence-transformers/all-MiniLM-L6-v2"
|
| 47 |
+
},
|
| 48 |
+
"loss": {
|
| 49 |
+
"hct_alpha": 0.5,
|
| 50 |
+
"lambda_hct": 1.0,
|
| 51 |
+
"lambda_sal": 1.0,
|
| 52 |
+
"learnable_bias_init": -10.0,
|
| 53 |
+
"learnable_temperature_init": 2.3,
|
| 54 |
+
"primary": "sigmoid_siglip",
|
| 55 |
+
"saliency_beta": 0.01,
|
| 56 |
+
"saliency_lambda_smooth": 4.0
|
| 57 |
+
},
|
| 58 |
+
"name": "keural-tiny-poc",
|
| 59 |
+
"saliency": {
|
| 60 |
+
"hidden_channels": 96
|
| 61 |
+
},
|
| 62 |
+
"spatial": {
|
| 63 |
+
"depth": 12,
|
| 64 |
+
"embed_dim": 384,
|
| 65 |
+
"head_dim": 64,
|
| 66 |
+
"mlp_ratio": 4,
|
| 67 |
+
"norm_eps": 1e-06,
|
| 68 |
+
"num_heads": 6,
|
| 69 |
+
"rope_base": 10000.0,
|
| 70 |
+
"use_pool_token": true
|
| 71 |
+
},
|
| 72 |
+
"stem": {
|
| 73 |
+
"activation": "gelu",
|
| 74 |
+
"channels": [
|
| 75 |
+
64,
|
| 76 |
+
128,
|
| 77 |
+
256,
|
| 78 |
+
384
|
| 79 |
+
],
|
| 80 |
+
"kernel_size": 3
|
| 81 |
+
},
|
| 82 |
+
"tokenizer": {
|
| 83 |
+
"budget_floors": {
|
| 84 |
+
"detail": 0,
|
| 85 |
+
"global": 4,
|
| 86 |
+
"region": 8
|
| 87 |
+
},
|
| 88 |
+
"budget_split": {
|
| 89 |
+
"detail": 0.7,
|
| 90 |
+
"global": 0.05,
|
| 91 |
+
"region": 0.25
|
| 92 |
+
},
|
| 93 |
+
"default_token_budget": 256,
|
| 94 |
+
"gumbel_temperature_end": 0.1,
|
| 95 |
+
"gumbel_temperature_start": 1.0,
|
| 96 |
+
"max_token_budget": 1024,
|
| 97 |
+
"nms_radius": {
|
| 98 |
+
"detail": 1,
|
| 99 |
+
"region": 2
|
| 100 |
+
},
|
| 101 |
+
"roi_align_size": 2,
|
| 102 |
+
"saliency_smooth_kernel": {
|
| 103 |
+
"detail": 3,
|
| 104 |
+
"region": 5
|
| 105 |
+
},
|
| 106 |
+
"temperature_anneal_steps": 10000,
|
| 107 |
+
"warmup_steps_fixed_grid": 5000
|
| 108 |
+
},
|
| 109 |
+
"training": {
|
| 110 |
+
"batch_size_per_device": 256,
|
| 111 |
+
"betas": [
|
| 112 |
+
0.9,
|
| 113 |
+
0.95
|
| 114 |
+
],
|
| 115 |
+
"gradient_accumulation": 4,
|
| 116 |
+
"gradient_checkpointing": false,
|
| 117 |
+
"log_every": 50,
|
| 118 |
+
"lr": 0.001,
|
| 119 |
+
"lr_schedule": "cosine_with_warmup",
|
| 120 |
+
"max_steps": 75000,
|
| 121 |
+
"optimizer": "adamw",
|
| 122 |
+
"precision": "bfloat16",
|
| 123 |
+
"save_every": 1000,
|
| 124 |
+
"seed": 42,
|
| 125 |
+
"warmup_steps": 2000,
|
| 126 |
+
"weight_decay": 0.05
|
| 127 |
+
}
|
| 128 |
+
},
|
| 129 |
+
"max_token_budget": 1024,
|
| 130 |
+
"model_type": "keural_vision",
|
| 131 |
+
"num_heads": 6,
|
| 132 |
+
"transformers_version": "4.57.2"
|
| 133 |
+
}
|
checkpoint-37000/model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:0dfc3f8c3014f16d021c5c78cc3277e07aadc8fc9b31d241f90197f2b3a02b8d
|
| 3 |
+
size 98683012
|
checkpoint-37000/training_state.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:d7a551149922678f5c76e26c945d698469aa1cda8b0587d3a9e6798f8ef76ecf
|
| 3 |
+
size 198657138
|
checkpoint-44000/config.json
ADDED
|
@@ -0,0 +1,133 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"architectures": [
|
| 3 |
+
"KeuralVisionModel"
|
| 4 |
+
],
|
| 5 |
+
"default_token_budget": 256,
|
| 6 |
+
"depth": 12,
|
| 7 |
+
"dtype": "float32",
|
| 8 |
+
"embed_dim": 384,
|
| 9 |
+
"keural_config": {
|
| 10 |
+
"data": {
|
| 11 |
+
"captions_path": "data/indexes/cc3m_wds_train.tsv",
|
| 12 |
+
"dataset_name": "cc3m+cc12m",
|
| 13 |
+
"extra_sources": [
|
| 14 |
+
{
|
| 15 |
+
"captions_path": "data/indexes/cc12m_wds_train.tsv",
|
| 16 |
+
"images_root": "data/cc12m"
|
| 17 |
+
}
|
| 18 |
+
],
|
| 19 |
+
"image_buckets": [
|
| 20 |
+
[
|
| 21 |
+
224,
|
| 22 |
+
224
|
| 23 |
+
],
|
| 24 |
+
[
|
| 25 |
+
256,
|
| 26 |
+
256
|
| 27 |
+
],
|
| 28 |
+
[
|
| 29 |
+
320,
|
| 30 |
+
320
|
| 31 |
+
]
|
| 32 |
+
],
|
| 33 |
+
"images_root": "data/cc3m",
|
| 34 |
+
"max_caption_length": 64,
|
| 35 |
+
"num_workers": 4,
|
| 36 |
+
"pixel_mean": [
|
| 37 |
+
0.485,
|
| 38 |
+
0.456,
|
| 39 |
+
0.406
|
| 40 |
+
],
|
| 41 |
+
"pixel_std": [
|
| 42 |
+
0.229,
|
| 43 |
+
0.224,
|
| 44 |
+
0.225
|
| 45 |
+
],
|
| 46 |
+
"text_encoder_name": "sentence-transformers/all-MiniLM-L6-v2"
|
| 47 |
+
},
|
| 48 |
+
"loss": {
|
| 49 |
+
"hct_alpha": 0.5,
|
| 50 |
+
"lambda_hct": 1.0,
|
| 51 |
+
"lambda_sal": 1.0,
|
| 52 |
+
"learnable_bias_init": -10.0,
|
| 53 |
+
"learnable_temperature_init": 2.3,
|
| 54 |
+
"primary": "sigmoid_siglip",
|
| 55 |
+
"saliency_beta": 0.01,
|
| 56 |
+
"saliency_lambda_smooth": 4.0
|
| 57 |
+
},
|
| 58 |
+
"name": "keural-tiny-poc",
|
| 59 |
+
"saliency": {
|
| 60 |
+
"hidden_channels": 96
|
| 61 |
+
},
|
| 62 |
+
"spatial": {
|
| 63 |
+
"depth": 12,
|
| 64 |
+
"embed_dim": 384,
|
| 65 |
+
"head_dim": 64,
|
| 66 |
+
"mlp_ratio": 4,
|
| 67 |
+
"norm_eps": 1e-06,
|
| 68 |
+
"num_heads": 6,
|
| 69 |
+
"rope_base": 10000.0,
|
| 70 |
+
"use_pool_token": true
|
| 71 |
+
},
|
| 72 |
+
"stem": {
|
| 73 |
+
"activation": "gelu",
|
| 74 |
+
"channels": [
|
| 75 |
+
64,
|
| 76 |
+
128,
|
| 77 |
+
256,
|
| 78 |
+
384
|
| 79 |
+
],
|
| 80 |
+
"kernel_size": 3
|
| 81 |
+
},
|
| 82 |
+
"tokenizer": {
|
| 83 |
+
"budget_floors": {
|
| 84 |
+
"detail": 0,
|
| 85 |
+
"global": 4,
|
| 86 |
+
"region": 8
|
| 87 |
+
},
|
| 88 |
+
"budget_split": {
|
| 89 |
+
"detail": 0.7,
|
| 90 |
+
"global": 0.05,
|
| 91 |
+
"region": 0.25
|
| 92 |
+
},
|
| 93 |
+
"default_token_budget": 256,
|
| 94 |
+
"gumbel_temperature_end": 0.1,
|
| 95 |
+
"gumbel_temperature_start": 1.0,
|
| 96 |
+
"max_token_budget": 1024,
|
| 97 |
+
"nms_radius": {
|
| 98 |
+
"detail": 1,
|
| 99 |
+
"region": 2
|
| 100 |
+
},
|
| 101 |
+
"roi_align_size": 2,
|
| 102 |
+
"saliency_smooth_kernel": {
|
| 103 |
+
"detail": 3,
|
| 104 |
+
"region": 5
|
| 105 |
+
},
|
| 106 |
+
"temperature_anneal_steps": 10000,
|
| 107 |
+
"warmup_steps_fixed_grid": 5000
|
| 108 |
+
},
|
| 109 |
+
"training": {
|
| 110 |
+
"batch_size_per_device": 256,
|
| 111 |
+
"betas": [
|
| 112 |
+
0.9,
|
| 113 |
+
0.95
|
| 114 |
+
],
|
| 115 |
+
"gradient_accumulation": 4,
|
| 116 |
+
"gradient_checkpointing": false,
|
| 117 |
+
"log_every": 50,
|
| 118 |
+
"lr": 0.001,
|
| 119 |
+
"lr_schedule": "cosine_with_warmup",
|
| 120 |
+
"max_steps": 75000,
|
| 121 |
+
"optimizer": "adamw",
|
| 122 |
+
"precision": "bfloat16",
|
| 123 |
+
"save_every": 1000,
|
| 124 |
+
"seed": 42,
|
| 125 |
+
"warmup_steps": 2000,
|
| 126 |
+
"weight_decay": 0.05
|
| 127 |
+
}
|
| 128 |
+
},
|
| 129 |
+
"max_token_budget": 1024,
|
| 130 |
+
"model_type": "keural_vision",
|
| 131 |
+
"num_heads": 6,
|
| 132 |
+
"transformers_version": "4.57.2"
|
| 133 |
+
}
|
checkpoint-44000/model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:b2b01394c8201198f9de6905fcd9e70caf2f5e0fa450f97fb44a2cb1272e9610
|
| 3 |
+
size 98683012
|
checkpoint-44000/training_state.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:63cef5f73ee7defa9a88b65d31c308210fcfa824594a4af136aa98da8382f959
|
| 3 |
+
size 198657138
|
config.json
ADDED
|
@@ -0,0 +1,133 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"architectures": [
|
| 3 |
+
"KeuralVisionModel"
|
| 4 |
+
],
|
| 5 |
+
"default_token_budget": 256,
|
| 6 |
+
"depth": 12,
|
| 7 |
+
"dtype": "float32",
|
| 8 |
+
"embed_dim": 384,
|
| 9 |
+
"keural_config": {
|
| 10 |
+
"data": {
|
| 11 |
+
"captions_path": "data/indexes/cc3m_wds_train.tsv",
|
| 12 |
+
"dataset_name": "cc3m+cc12m",
|
| 13 |
+
"extra_sources": [
|
| 14 |
+
{
|
| 15 |
+
"captions_path": "data/indexes/cc12m_wds_train.tsv",
|
| 16 |
+
"images_root": "data/cc12m"
|
| 17 |
+
}
|
| 18 |
+
],
|
| 19 |
+
"image_buckets": [
|
| 20 |
+
[
|
| 21 |
+
224,
|
| 22 |
+
224
|
| 23 |
+
],
|
| 24 |
+
[
|
| 25 |
+
256,
|
| 26 |
+
256
|
| 27 |
+
],
|
| 28 |
+
[
|
| 29 |
+
320,
|
| 30 |
+
320
|
| 31 |
+
]
|
| 32 |
+
],
|
| 33 |
+
"images_root": "data/cc3m",
|
| 34 |
+
"max_caption_length": 64,
|
| 35 |
+
"num_workers": 4,
|
| 36 |
+
"pixel_mean": [
|
| 37 |
+
0.485,
|
| 38 |
+
0.456,
|
| 39 |
+
0.406
|
| 40 |
+
],
|
| 41 |
+
"pixel_std": [
|
| 42 |
+
0.229,
|
| 43 |
+
0.224,
|
| 44 |
+
0.225
|
| 45 |
+
],
|
| 46 |
+
"text_encoder_name": "sentence-transformers/all-MiniLM-L6-v2"
|
| 47 |
+
},
|
| 48 |
+
"loss": {
|
| 49 |
+
"hct_alpha": 0.5,
|
| 50 |
+
"lambda_hct": 1.0,
|
| 51 |
+
"lambda_sal": 1.0,
|
| 52 |
+
"learnable_bias_init": -10.0,
|
| 53 |
+
"learnable_temperature_init": 2.3,
|
| 54 |
+
"primary": "sigmoid_siglip",
|
| 55 |
+
"saliency_beta": 0.01,
|
| 56 |
+
"saliency_lambda_smooth": 4.0
|
| 57 |
+
},
|
| 58 |
+
"name": "keural-tiny-poc",
|
| 59 |
+
"saliency": {
|
| 60 |
+
"hidden_channels": 96
|
| 61 |
+
},
|
| 62 |
+
"spatial": {
|
| 63 |
+
"depth": 12,
|
| 64 |
+
"embed_dim": 384,
|
| 65 |
+
"head_dim": 64,
|
| 66 |
+
"mlp_ratio": 4,
|
| 67 |
+
"norm_eps": 1e-06,
|
| 68 |
+
"num_heads": 6,
|
| 69 |
+
"rope_base": 10000.0,
|
| 70 |
+
"use_pool_token": true
|
| 71 |
+
},
|
| 72 |
+
"stem": {
|
| 73 |
+
"activation": "gelu",
|
| 74 |
+
"channels": [
|
| 75 |
+
64,
|
| 76 |
+
128,
|
| 77 |
+
256,
|
| 78 |
+
384
|
| 79 |
+
],
|
| 80 |
+
"kernel_size": 3
|
| 81 |
+
},
|
| 82 |
+
"tokenizer": {
|
| 83 |
+
"budget_floors": {
|
| 84 |
+
"detail": 0,
|
| 85 |
+
"global": 4,
|
| 86 |
+
"region": 8
|
| 87 |
+
},
|
| 88 |
+
"budget_split": {
|
| 89 |
+
"detail": 0.7,
|
| 90 |
+
"global": 0.05,
|
| 91 |
+
"region": 0.25
|
| 92 |
+
},
|
| 93 |
+
"default_token_budget": 256,
|
| 94 |
+
"gumbel_temperature_end": 0.1,
|
| 95 |
+
"gumbel_temperature_start": 1.0,
|
| 96 |
+
"max_token_budget": 1024,
|
| 97 |
+
"nms_radius": {
|
| 98 |
+
"detail": 1,
|
| 99 |
+
"region": 2
|
| 100 |
+
},
|
| 101 |
+
"roi_align_size": 2,
|
| 102 |
+
"saliency_smooth_kernel": {
|
| 103 |
+
"detail": 3,
|
| 104 |
+
"region": 5
|
| 105 |
+
},
|
| 106 |
+
"temperature_anneal_steps": 10000,
|
| 107 |
+
"warmup_steps_fixed_grid": 5000
|
| 108 |
+
},
|
| 109 |
+
"training": {
|
| 110 |
+
"batch_size_per_device": 256,
|
| 111 |
+
"betas": [
|
| 112 |
+
0.9,
|
| 113 |
+
0.95
|
| 114 |
+
],
|
| 115 |
+
"gradient_accumulation": 4,
|
| 116 |
+
"gradient_checkpointing": false,
|
| 117 |
+
"log_every": 50,
|
| 118 |
+
"lr": 0.001,
|
| 119 |
+
"lr_schedule": "cosine_with_warmup",
|
| 120 |
+
"max_steps": 75000,
|
| 121 |
+
"optimizer": "adamw",
|
| 122 |
+
"precision": "bfloat16",
|
| 123 |
+
"save_every": 1000,
|
| 124 |
+
"seed": 42,
|
| 125 |
+
"warmup_steps": 2000,
|
| 126 |
+
"weight_decay": 0.05
|
| 127 |
+
}
|
| 128 |
+
},
|
| 129 |
+
"max_token_budget": 1024,
|
| 130 |
+
"model_type": "keural_vision",
|
| 131 |
+
"num_heads": 6,
|
| 132 |
+
"transformers_version": "4.57.2"
|
| 133 |
+
}
|
model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:8fe763e68959e79120156879fbbf6cc6f77744517f91123786d149b36c3ad0b3
|
| 3 |
+
size 98683012
|
preprocessor_config.json
ADDED
|
@@ -0,0 +1,15 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"do_resize": true,
|
| 3 |
+
"size": {"height": 256, "width": 256},
|
| 4 |
+
"do_normalize": true,
|
| 5 |
+
"image_mean": [0.485, 0.456, 0.406],
|
| 6 |
+
"image_std": [0.229, 0.224, 0.225],
|
| 7 |
+
"do_rescale": true,
|
| 8 |
+
"rescale_factor": 0.00392156862745098,
|
| 9 |
+
"image_processor_type": "KeuralImageProcessor",
|
| 10 |
+
"supported_sizes": [
|
| 11 |
+
{"height": 224, "width": 224},
|
| 12 |
+
{"height": 256, "width": 256},
|
| 13 |
+
{"height": 320, "width": 320}
|
| 14 |
+
]
|
| 15 |
+
}
|
saliency_viz/step_28k/cat_saliency.png
ADDED
|
Git LFS Details
|
saliency_viz/step_28k/dog_saliency.png
ADDED
|
Git LFS Details
|
saliency_viz/step_28k/horse_saliency.png
ADDED
|
Git LFS Details
|
saliency_viz/step_35k/cat_saliency.png
ADDED
|
Git LFS Details
|
saliency_viz/step_35k/dog_saliency.png
ADDED
|
Git LFS Details
|
saliency_viz/step_35k/horse_saliency.png
ADDED
|
Git LFS Details
|
saliency_viz/step_36k/cat_saliency.png
ADDED
|
Git LFS Details
|
saliency_viz/step_36k/dog_saliency.png
ADDED
|
Git LFS Details
|
saliency_viz/step_36k/horse_saliency.png
ADDED
|
Git LFS Details
|
saliency_viz/step_36k_cpu/cat_saliency.png
ADDED
|
Git LFS Details
|
saliency_viz/step_36k_cpu/dog_saliency.png
ADDED
|
Git LFS Details
|
saliency_viz/step_36k_cpu/horse_saliency.png
ADDED
|
Git LFS Details
|
training_state.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:1dfb625a59763051f3f84e92da3f087dbcb359847ae12a04190279c9ad4053ac
|
| 3 |
+
size 198657138
|