mkd-hika commited on
Commit
96faa83
·
0 Parent(s):

Duplicate from mkd-hika/keural-vision-encoder-poc

Browse files
.gitattributes ADDED
@@ -0,0 +1,51 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ *.7z filter=lfs diff=lfs merge=lfs -text
2
+ *.arrow filter=lfs diff=lfs merge=lfs -text
3
+ *.bin filter=lfs diff=lfs merge=lfs -text
4
+ *.bz2 filter=lfs diff=lfs merge=lfs -text
5
+ *.ckpt filter=lfs diff=lfs merge=lfs -text
6
+ *.ftz filter=lfs diff=lfs merge=lfs -text
7
+ *.gz filter=lfs diff=lfs merge=lfs -text
8
+ *.h5 filter=lfs diff=lfs merge=lfs -text
9
+ *.joblib filter=lfs diff=lfs merge=lfs -text
10
+ *.lfs.* filter=lfs diff=lfs merge=lfs -text
11
+ *.mlmodel filter=lfs diff=lfs merge=lfs -text
12
+ *.model filter=lfs diff=lfs merge=lfs -text
13
+ *.msgpack filter=lfs diff=lfs merge=lfs -text
14
+ *.npy filter=lfs diff=lfs merge=lfs -text
15
+ *.npz filter=lfs diff=lfs merge=lfs -text
16
+ *.onnx filter=lfs diff=lfs merge=lfs -text
17
+ *.ot filter=lfs diff=lfs merge=lfs -text
18
+ *.parquet filter=lfs diff=lfs merge=lfs -text
19
+ *.pb filter=lfs diff=lfs merge=lfs -text
20
+ *.pickle filter=lfs diff=lfs merge=lfs -text
21
+ *.pkl filter=lfs diff=lfs merge=lfs -text
22
+ *.pt filter=lfs diff=lfs merge=lfs -text
23
+ *.pth filter=lfs diff=lfs merge=lfs -text
24
+ *.rar filter=lfs diff=lfs merge=lfs -text
25
+ *.safetensors filter=lfs diff=lfs merge=lfs -text
26
+ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
27
+ *.tar.* filter=lfs diff=lfs merge=lfs -text
28
+ *.tar filter=lfs diff=lfs merge=lfs -text
29
+ *.tflite filter=lfs diff=lfs merge=lfs -text
30
+ *.tgz filter=lfs diff=lfs merge=lfs -text
31
+ *.wasm filter=lfs diff=lfs merge=lfs -text
32
+ *.xz filter=lfs diff=lfs merge=lfs -text
33
+ *.zip filter=lfs diff=lfs merge=lfs -text
34
+ *.zst filter=lfs diff=lfs merge=lfs -text
35
+ *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ saliency_viz/step_28k/cat_saliency.png filter=lfs diff=lfs merge=lfs -text
37
+ saliency_viz/step_28k/dog_saliency.png filter=lfs diff=lfs merge=lfs -text
38
+ saliency_viz/step_28k/horse_saliency.png filter=lfs diff=lfs merge=lfs -text
39
+ saliency_viz/step_35k/cat_saliency.png filter=lfs diff=lfs merge=lfs -text
40
+ saliency_viz/step_35k/dog_saliency.png filter=lfs diff=lfs merge=lfs -text
41
+ saliency_viz/step_35k/horse_saliency.png filter=lfs diff=lfs merge=lfs -text
42
+ saliency_viz/step_36k/cat_saliency.png filter=lfs diff=lfs merge=lfs -text
43
+ saliency_viz/step_36k/dog_saliency.png filter=lfs diff=lfs merge=lfs -text
44
+ saliency_viz/step_36k/horse_saliency.png filter=lfs diff=lfs merge=lfs -text
45
+ saliency_viz/step_36k_cpu/cat_saliency.png filter=lfs diff=lfs merge=lfs -text
46
+ saliency_viz/step_36k_cpu/dog_saliency.png filter=lfs diff=lfs merge=lfs -text
47
+ saliency_viz/step_36k_cpu/horse_saliency.png filter=lfs diff=lfs merge=lfs -text
48
+ assets/keural_vision_encoder.gif filter=lfs diff=lfs merge=lfs -text
49
+ assets/training_loss_curves.png filter=lfs diff=lfs merge=lfs -text
50
+ assets/benchmark_results.png filter=lfs diff=lfs merge=lfs -text
51
+ assets/token_budget_curve.png filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,220 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language: en
3
+ license: mit
4
+ library_name: transformers
5
+ tags:
6
+ - vision
7
+ - vision-encoder
8
+ - image-feature-extraction
9
+ - image-text
10
+ - contrastive-learning
11
+ - custom-architecture
12
+ - vlm
13
+ pipeline_tag: image-feature-extraction
14
+ ---
15
+
16
+ <div align="center">
17
+
18
+ # 👁️ Keural Vision Encoder <sub>(PoC · V0.1)</sub>
19
+
20
+ **A 24.7M-parameter vision encoder trained _from scratch_ — no pretrained backbone, no CLIP weights. Structured tokens that plug into any LLM.**
21
+
22
+ <p>
23
+ <img src="https://img.shields.io/badge/MKD_Co.,_Ltd.-Keural-6d28d9?style=for-the-badge" alt="MKD">
24
+ <img src="https://img.shields.io/badge/status-V0.1_PoC-16a34a?style=for-the-badge" alt="Status">
25
+ </p>
26
+
27
+ <p>
28
+ <img src="https://img.shields.io/badge/params-24.7M-8b5cf6" alt="Params">
29
+ <img src="https://img.shields.io/badge/from-scratch-ec4899" alt="From scratch">
30
+ <img src="https://img.shields.io/badge/embed_dim-384-f59e0b" alt="Dim">
31
+ <img src="https://img.shields.io/badge/token_budget-adaptive-22d3ee" alt="ATB">
32
+ <img src="https://img.shields.io/badge/PyTorch-2.x-ee4c2c?logo=pytorch&logoColor=white" alt="PyTorch">
33
+ <img src="https://img.shields.io/badge/trained_on-1×_RTX_5090-38bdf8" alt="Hardware">
34
+ </p>
35
+
36
+ <br>
37
+
38
+ <img src="assets/keural_vision_encoder.gif" alt="Where the Keural encoder sits in the VLM pipeline" width="98%">
39
+
40
+ <br>
41
+
42
+ *The encoder (CNN Stem → ATB → Spatial Transformer) feeds the LevelAware Projector → Mistral-7B in the full [Keural VLM](https://huggingface.co/mkd-hika/keural-vlm-poc).*
43
+
44
+ </div>
45
+
46
+ ---
47
+
48
+ ## ✨ Key Innovations
49
+
50
+ **🎯 Adaptive Token Budget (ATB) Tokenization** — token count is a runtime parameter. Tokens are allocated to information-dense regions: a blank wall gets fewer, a dense document gets more.
51
+
52
+ ```python
53
+ out = encoder(image, token_budget=64) # cheap
54
+ out = encoder(image, token_budget=256) # default
55
+ out = encoder(image, token_budget=1024) # full fidelity
56
+ ```
57
+
58
+ **🪜 Hierarchical Concept Tokenization (HCT)** — every token carries a semantic level tag: global (whole-image), region (object-scale), or detail (fine-grained).
59
+
60
+ ```python
61
+ out = encoder(image)
62
+ print(out.level_ids) # tensor of {0=global, 1=region, 2=detail}
63
+ ```
64
+
65
+ ---
66
+
67
+ ## ⚙️ Model Specifications
68
+
69
+ | Property | Value |
70
+ |---|---|
71
+ | Parameters | **24.7M** (trained from scratch) |
72
+ | Architecture | CNN Stem + ATB Tokenizer + 12-layer Spatial Transformer |
73
+ | Embedding dim | 384 |
74
+ | Token budget | 256 (default) · runtime-adjustable |
75
+ | Token split | 5% global / 25% region / 70% detail |
76
+ | Precision | bfloat16 |
77
+ | Training data | CC3M + CC12M (~6.9M image-text pairs) |
78
+ | Hardware | 1× RTX 5090 (32 GB VRAM) |
79
+
80
+ ---
81
+
82
+ ## 🎨 Saliency & Token Placement
83
+
84
+ The ATB tokenizer concentrates tokens on salient regions. Left → right: original · saliency heatmap · token placement (global / region / detail).
85
+
86
+ <div align="center">
87
+ <img src="saliency_viz/step_36k/cat_saliency.png" alt="Saliency and token placement" width="88%">
88
+ </div>
89
+
90
+ ---
91
+
92
+ ## 🧪 Training
93
+
94
+ **Phase 1 — Vision Encoder Pretraining ✅ COMPLETE.** Trained from scratch for
95
+ ~75,000 steps on CC3M + CC12M (~6.9M image–text pairs), SigLIP-style contrastive
96
+ objective, 1× RTX 5090. The frozen encoder is then integrated into the full VLM via
97
+ `LevelAwareProjector` (384 → 2048 → 4096) → **Mistral-7B-Instruct-v0.3** (4-bit NF4
98
+ QLoRA), fine-tuned with SFT (LLaVA-Instruct-150K, 30K steps).
99
+
100
+ <div align="center">
101
+ <img src="assets/training_loss_curves.png" alt="Training loss curves" width="88%">
102
+ </div>
103
+
104
+ ---
105
+
106
+ ## 📊 Benchmark Results — SFT-30K
107
+
108
+ Downstream VLM benchmarks using this encoder (frozen) + projector + Mistral-7B at the
109
+ **SFT-30K checkpoint** (supervised fine-tuning, 30,000 steps; **before** DPO).
110
+ Evaluated on 1,000 samples each where applicable. The Keural encoder is **12.4×
111
+ smaller** than LLaVA's CLIP encoder (307M).
112
+
113
+ | Benchmark | Keural SFT-30K | LLaVA 1.5 <sub>(307M enc)</sub> | LLaVA 1.6 <sub>(307M enc)</sub> |
114
+ |---|:---:|:---:|:---:|
115
+ | VQAv2 Accuracy | **12.9%** | 78.5% | 81.8% |
116
+ | POPE F1 | **66.9%** | 85.9% | 86.5% |
117
+ | MME Total Score | **704.3** | 1510.7 | 1519.3 |
118
+ | TextVQA Accuracy | **0.8%** | 58.2% | 64.9% |
119
+ | ScienceQA Accuracy <sub>(img)</sub> | **39.7%** | 66.8% | 70.6% |
120
+
121
+ > These are the **SFT-30K** numbers (measured from `outputs/eval/sft_30k/`). Applying
122
+ > DPO alignment (RLHF-V, 3K steps) improves every metric — e.g. VQAv2 → 43.6%,
123
+ > ScienceQA → 53.7%, MME → 838.8. Full **SFT+DPO** results, the comparison chart, and
124
+ > the complete VLM live on **[mkd-hika/keural-vlm-poc](https://huggingface.co/mkd-hika/keural-vlm-poc)**.
125
+
126
+ ---
127
+
128
+ ## ⚡ Adaptive Token Budget — latency vs budget
129
+
130
+ Encode latency vs token budget — trade visual detail for speed at runtime, no retraining.
131
+
132
+ <div align="center">
133
+ <img src="assets/token_budget_curve.png" alt="ATB token budget vs latency" width="80%">
134
+ </div>
135
+
136
+ ---
137
+
138
+ ## 📐 Output Contract
139
+
140
+ Every forward pass returns a `KeuralEncoderOutput`:
141
+
142
+ | Field | Shape | Description |
143
+ |---|---|---|
144
+ | `tokens` | `(B, N, 384)` | Token vectors |
145
+ | `attention_mask` | `(B, N)` | 1 = real token, 0 = padding |
146
+ | `level_ids` | `(B, N)` | 0=global, 1=region, 2=detail |
147
+ | `spatial_metadata` | `(B, N, 4)` | Per-token (cx, cy, w, h) in [0,1] |
148
+ | `saliency_scores` | `(B, N)` | Allocator confidence per token |
149
+ | `pooled` | `(B, 384)` | Single-vector image summary |
150
+ | `token_budget_used` | `(B,)` | Actual tokens used per image |
151
+
152
+ ---
153
+
154
+ ## 🚀 Usage
155
+
156
+ ```python
157
+ import torch
158
+ from architecture.encoder import KeuralVisionEncoder
159
+ from keural_config import KeuralConfig
160
+
161
+ cfg = KeuralConfig.from_yaml("configs/keural_tiny_poc.yaml")
162
+ model = KeuralVisionEncoder.from_pretrained("mkd-hika/keural-vision-encoder-poc")
163
+ model.eval()
164
+
165
+ from PIL import Image
166
+ from torchvision import transforms
167
+
168
+ transform = transforms.Compose([
169
+ transforms.Resize((256, 256)),
170
+ transforms.ToTensor(),
171
+ transforms.Normalize([0.5], [0.5]),
172
+ ])
173
+
174
+ image = Image.open("image.jpg").convert("RGB")
175
+ x = transform(image).unsqueeze(0) # (1, 3, 256, 256)
176
+
177
+ with torch.no_grad():
178
+ out = model(x)
179
+
180
+ print(out.tokens.shape) # (1, 256, 384)
181
+ print(out.level_ids[0, :8]) # [0, 0, 1, 1, 1, 2, 2, 2]
182
+ print(out.pooled.shape) # (1, 384)
183
+ ```
184
+
185
+ ---
186
+
187
+ ## 🗺️ Roadmap
188
+
189
+ | Phase | Params | Hardware | Status |
190
+ |---|:---:|:---:|:---:|
191
+ | **PoC** (this model) | 24.7M | 1× RTX 5090 | ✅ Phase 1 complete |
192
+ | Mid-level | ~230M | 8× H100 80 GB | 🔜 Planned |
193
+ | Commercial | ~1.1B | 64× H100 80 GB | 🔮 Future |
194
+
195
+ The mid-level model will use knowledge distillation from **SigLIP-400M** (natural images) and **InternViT-300M** (documents/OCR). Teachers are discarded after training — not part of the final model.
196
+
197
+ ---
198
+
199
+ ## 📝 Citation
200
+
201
+ ```bibtex
202
+ @misc{keural_vision_encoder_2026,
203
+ title = {Keural Vision Encoder: Content-Adaptive Vision Encoding via Saliency-Guided Token Budgets},
204
+ author = {Barki, Hika and MKD Co., Ltd.},
205
+ year = {2026},
206
+ note = {V0.1 — Phase 1 complete},
207
+ }
208
+ ```
209
+
210
+ ---
211
+
212
+ ## 📄 License
213
+
214
+ Code: MIT. Model weights trained on CC3M + CC12M — data licenses apply.
215
+
216
+ <div align="center">
217
+
218
+ **MKD Co., Ltd. — 2026**
219
+
220
+ </div>
assets/keural_vision_encoder.gif ADDED

Git LFS Details

  • SHA256: 452a169e2821b95035d990e2599714f34fc2194fe7858d9247f69fc631441ef1
  • Pointer size: 132 Bytes
  • Size of remote file: 2.79 MB
assets/token_budget_curve.png ADDED

Git LFS Details

  • SHA256: 09fdff2988132b300bbdf92e8b888bacb77dc8c112f06c1575c3bfbb232687c1
  • Pointer size: 131 Bytes
  • Size of remote file: 108 kB
assets/training_loss_curves.png ADDED

Git LFS Details

  • SHA256: a8016bc7fbe9107ac9e0290adb5ba0acaaf55d8928731ed509a57ead9e4273d2
  • Pointer size: 131 Bytes
  • Size of remote file: 171 kB
checkpoint-37000/config.json ADDED
@@ -0,0 +1,133 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "KeuralVisionModel"
4
+ ],
5
+ "default_token_budget": 256,
6
+ "depth": 12,
7
+ "dtype": "float32",
8
+ "embed_dim": 384,
9
+ "keural_config": {
10
+ "data": {
11
+ "captions_path": "data/indexes/cc3m_wds_train.tsv",
12
+ "dataset_name": "cc3m+cc12m",
13
+ "extra_sources": [
14
+ {
15
+ "captions_path": "data/indexes/cc12m_wds_train.tsv",
16
+ "images_root": "data/cc12m"
17
+ }
18
+ ],
19
+ "image_buckets": [
20
+ [
21
+ 224,
22
+ 224
23
+ ],
24
+ [
25
+ 256,
26
+ 256
27
+ ],
28
+ [
29
+ 320,
30
+ 320
31
+ ]
32
+ ],
33
+ "images_root": "data/cc3m",
34
+ "max_caption_length": 64,
35
+ "num_workers": 4,
36
+ "pixel_mean": [
37
+ 0.485,
38
+ 0.456,
39
+ 0.406
40
+ ],
41
+ "pixel_std": [
42
+ 0.229,
43
+ 0.224,
44
+ 0.225
45
+ ],
46
+ "text_encoder_name": "sentence-transformers/all-MiniLM-L6-v2"
47
+ },
48
+ "loss": {
49
+ "hct_alpha": 0.5,
50
+ "lambda_hct": 1.0,
51
+ "lambda_sal": 1.0,
52
+ "learnable_bias_init": -10.0,
53
+ "learnable_temperature_init": 2.3,
54
+ "primary": "sigmoid_siglip",
55
+ "saliency_beta": 0.01,
56
+ "saliency_lambda_smooth": 4.0
57
+ },
58
+ "name": "keural-tiny-poc",
59
+ "saliency": {
60
+ "hidden_channels": 96
61
+ },
62
+ "spatial": {
63
+ "depth": 12,
64
+ "embed_dim": 384,
65
+ "head_dim": 64,
66
+ "mlp_ratio": 4,
67
+ "norm_eps": 1e-06,
68
+ "num_heads": 6,
69
+ "rope_base": 10000.0,
70
+ "use_pool_token": true
71
+ },
72
+ "stem": {
73
+ "activation": "gelu",
74
+ "channels": [
75
+ 64,
76
+ 128,
77
+ 256,
78
+ 384
79
+ ],
80
+ "kernel_size": 3
81
+ },
82
+ "tokenizer": {
83
+ "budget_floors": {
84
+ "detail": 0,
85
+ "global": 4,
86
+ "region": 8
87
+ },
88
+ "budget_split": {
89
+ "detail": 0.7,
90
+ "global": 0.05,
91
+ "region": 0.25
92
+ },
93
+ "default_token_budget": 256,
94
+ "gumbel_temperature_end": 0.1,
95
+ "gumbel_temperature_start": 1.0,
96
+ "max_token_budget": 1024,
97
+ "nms_radius": {
98
+ "detail": 1,
99
+ "region": 2
100
+ },
101
+ "roi_align_size": 2,
102
+ "saliency_smooth_kernel": {
103
+ "detail": 3,
104
+ "region": 5
105
+ },
106
+ "temperature_anneal_steps": 10000,
107
+ "warmup_steps_fixed_grid": 5000
108
+ },
109
+ "training": {
110
+ "batch_size_per_device": 256,
111
+ "betas": [
112
+ 0.9,
113
+ 0.95
114
+ ],
115
+ "gradient_accumulation": 4,
116
+ "gradient_checkpointing": false,
117
+ "log_every": 50,
118
+ "lr": 0.001,
119
+ "lr_schedule": "cosine_with_warmup",
120
+ "max_steps": 75000,
121
+ "optimizer": "adamw",
122
+ "precision": "bfloat16",
123
+ "save_every": 1000,
124
+ "seed": 42,
125
+ "warmup_steps": 2000,
126
+ "weight_decay": 0.05
127
+ }
128
+ },
129
+ "max_token_budget": 1024,
130
+ "model_type": "keural_vision",
131
+ "num_heads": 6,
132
+ "transformers_version": "4.57.2"
133
+ }
checkpoint-37000/model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0dfc3f8c3014f16d021c5c78cc3277e07aadc8fc9b31d241f90197f2b3a02b8d
3
+ size 98683012
checkpoint-37000/training_state.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d7a551149922678f5c76e26c945d698469aa1cda8b0587d3a9e6798f8ef76ecf
3
+ size 198657138
checkpoint-44000/config.json ADDED
@@ -0,0 +1,133 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "KeuralVisionModel"
4
+ ],
5
+ "default_token_budget": 256,
6
+ "depth": 12,
7
+ "dtype": "float32",
8
+ "embed_dim": 384,
9
+ "keural_config": {
10
+ "data": {
11
+ "captions_path": "data/indexes/cc3m_wds_train.tsv",
12
+ "dataset_name": "cc3m+cc12m",
13
+ "extra_sources": [
14
+ {
15
+ "captions_path": "data/indexes/cc12m_wds_train.tsv",
16
+ "images_root": "data/cc12m"
17
+ }
18
+ ],
19
+ "image_buckets": [
20
+ [
21
+ 224,
22
+ 224
23
+ ],
24
+ [
25
+ 256,
26
+ 256
27
+ ],
28
+ [
29
+ 320,
30
+ 320
31
+ ]
32
+ ],
33
+ "images_root": "data/cc3m",
34
+ "max_caption_length": 64,
35
+ "num_workers": 4,
36
+ "pixel_mean": [
37
+ 0.485,
38
+ 0.456,
39
+ 0.406
40
+ ],
41
+ "pixel_std": [
42
+ 0.229,
43
+ 0.224,
44
+ 0.225
45
+ ],
46
+ "text_encoder_name": "sentence-transformers/all-MiniLM-L6-v2"
47
+ },
48
+ "loss": {
49
+ "hct_alpha": 0.5,
50
+ "lambda_hct": 1.0,
51
+ "lambda_sal": 1.0,
52
+ "learnable_bias_init": -10.0,
53
+ "learnable_temperature_init": 2.3,
54
+ "primary": "sigmoid_siglip",
55
+ "saliency_beta": 0.01,
56
+ "saliency_lambda_smooth": 4.0
57
+ },
58
+ "name": "keural-tiny-poc",
59
+ "saliency": {
60
+ "hidden_channels": 96
61
+ },
62
+ "spatial": {
63
+ "depth": 12,
64
+ "embed_dim": 384,
65
+ "head_dim": 64,
66
+ "mlp_ratio": 4,
67
+ "norm_eps": 1e-06,
68
+ "num_heads": 6,
69
+ "rope_base": 10000.0,
70
+ "use_pool_token": true
71
+ },
72
+ "stem": {
73
+ "activation": "gelu",
74
+ "channels": [
75
+ 64,
76
+ 128,
77
+ 256,
78
+ 384
79
+ ],
80
+ "kernel_size": 3
81
+ },
82
+ "tokenizer": {
83
+ "budget_floors": {
84
+ "detail": 0,
85
+ "global": 4,
86
+ "region": 8
87
+ },
88
+ "budget_split": {
89
+ "detail": 0.7,
90
+ "global": 0.05,
91
+ "region": 0.25
92
+ },
93
+ "default_token_budget": 256,
94
+ "gumbel_temperature_end": 0.1,
95
+ "gumbel_temperature_start": 1.0,
96
+ "max_token_budget": 1024,
97
+ "nms_radius": {
98
+ "detail": 1,
99
+ "region": 2
100
+ },
101
+ "roi_align_size": 2,
102
+ "saliency_smooth_kernel": {
103
+ "detail": 3,
104
+ "region": 5
105
+ },
106
+ "temperature_anneal_steps": 10000,
107
+ "warmup_steps_fixed_grid": 5000
108
+ },
109
+ "training": {
110
+ "batch_size_per_device": 256,
111
+ "betas": [
112
+ 0.9,
113
+ 0.95
114
+ ],
115
+ "gradient_accumulation": 4,
116
+ "gradient_checkpointing": false,
117
+ "log_every": 50,
118
+ "lr": 0.001,
119
+ "lr_schedule": "cosine_with_warmup",
120
+ "max_steps": 75000,
121
+ "optimizer": "adamw",
122
+ "precision": "bfloat16",
123
+ "save_every": 1000,
124
+ "seed": 42,
125
+ "warmup_steps": 2000,
126
+ "weight_decay": 0.05
127
+ }
128
+ },
129
+ "max_token_budget": 1024,
130
+ "model_type": "keural_vision",
131
+ "num_heads": 6,
132
+ "transformers_version": "4.57.2"
133
+ }
checkpoint-44000/model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b2b01394c8201198f9de6905fcd9e70caf2f5e0fa450f97fb44a2cb1272e9610
3
+ size 98683012
checkpoint-44000/training_state.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:63cef5f73ee7defa9a88b65d31c308210fcfa824594a4af136aa98da8382f959
3
+ size 198657138
config.json ADDED
@@ -0,0 +1,133 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "KeuralVisionModel"
4
+ ],
5
+ "default_token_budget": 256,
6
+ "depth": 12,
7
+ "dtype": "float32",
8
+ "embed_dim": 384,
9
+ "keural_config": {
10
+ "data": {
11
+ "captions_path": "data/indexes/cc3m_wds_train.tsv",
12
+ "dataset_name": "cc3m+cc12m",
13
+ "extra_sources": [
14
+ {
15
+ "captions_path": "data/indexes/cc12m_wds_train.tsv",
16
+ "images_root": "data/cc12m"
17
+ }
18
+ ],
19
+ "image_buckets": [
20
+ [
21
+ 224,
22
+ 224
23
+ ],
24
+ [
25
+ 256,
26
+ 256
27
+ ],
28
+ [
29
+ 320,
30
+ 320
31
+ ]
32
+ ],
33
+ "images_root": "data/cc3m",
34
+ "max_caption_length": 64,
35
+ "num_workers": 4,
36
+ "pixel_mean": [
37
+ 0.485,
38
+ 0.456,
39
+ 0.406
40
+ ],
41
+ "pixel_std": [
42
+ 0.229,
43
+ 0.224,
44
+ 0.225
45
+ ],
46
+ "text_encoder_name": "sentence-transformers/all-MiniLM-L6-v2"
47
+ },
48
+ "loss": {
49
+ "hct_alpha": 0.5,
50
+ "lambda_hct": 1.0,
51
+ "lambda_sal": 1.0,
52
+ "learnable_bias_init": -10.0,
53
+ "learnable_temperature_init": 2.3,
54
+ "primary": "sigmoid_siglip",
55
+ "saliency_beta": 0.01,
56
+ "saliency_lambda_smooth": 4.0
57
+ },
58
+ "name": "keural-tiny-poc",
59
+ "saliency": {
60
+ "hidden_channels": 96
61
+ },
62
+ "spatial": {
63
+ "depth": 12,
64
+ "embed_dim": 384,
65
+ "head_dim": 64,
66
+ "mlp_ratio": 4,
67
+ "norm_eps": 1e-06,
68
+ "num_heads": 6,
69
+ "rope_base": 10000.0,
70
+ "use_pool_token": true
71
+ },
72
+ "stem": {
73
+ "activation": "gelu",
74
+ "channels": [
75
+ 64,
76
+ 128,
77
+ 256,
78
+ 384
79
+ ],
80
+ "kernel_size": 3
81
+ },
82
+ "tokenizer": {
83
+ "budget_floors": {
84
+ "detail": 0,
85
+ "global": 4,
86
+ "region": 8
87
+ },
88
+ "budget_split": {
89
+ "detail": 0.7,
90
+ "global": 0.05,
91
+ "region": 0.25
92
+ },
93
+ "default_token_budget": 256,
94
+ "gumbel_temperature_end": 0.1,
95
+ "gumbel_temperature_start": 1.0,
96
+ "max_token_budget": 1024,
97
+ "nms_radius": {
98
+ "detail": 1,
99
+ "region": 2
100
+ },
101
+ "roi_align_size": 2,
102
+ "saliency_smooth_kernel": {
103
+ "detail": 3,
104
+ "region": 5
105
+ },
106
+ "temperature_anneal_steps": 10000,
107
+ "warmup_steps_fixed_grid": 5000
108
+ },
109
+ "training": {
110
+ "batch_size_per_device": 256,
111
+ "betas": [
112
+ 0.9,
113
+ 0.95
114
+ ],
115
+ "gradient_accumulation": 4,
116
+ "gradient_checkpointing": false,
117
+ "log_every": 50,
118
+ "lr": 0.001,
119
+ "lr_schedule": "cosine_with_warmup",
120
+ "max_steps": 75000,
121
+ "optimizer": "adamw",
122
+ "precision": "bfloat16",
123
+ "save_every": 1000,
124
+ "seed": 42,
125
+ "warmup_steps": 2000,
126
+ "weight_decay": 0.05
127
+ }
128
+ },
129
+ "max_token_budget": 1024,
130
+ "model_type": "keural_vision",
131
+ "num_heads": 6,
132
+ "transformers_version": "4.57.2"
133
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8fe763e68959e79120156879fbbf6cc6f77744517f91123786d149b36c3ad0b3
3
+ size 98683012
preprocessor_config.json ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "do_resize": true,
3
+ "size": {"height": 256, "width": 256},
4
+ "do_normalize": true,
5
+ "image_mean": [0.485, 0.456, 0.406],
6
+ "image_std": [0.229, 0.224, 0.225],
7
+ "do_rescale": true,
8
+ "rescale_factor": 0.00392156862745098,
9
+ "image_processor_type": "KeuralImageProcessor",
10
+ "supported_sizes": [
11
+ {"height": 224, "width": 224},
12
+ {"height": 256, "width": 256},
13
+ {"height": 320, "width": 320}
14
+ ]
15
+ }
saliency_viz/step_28k/cat_saliency.png ADDED

Git LFS Details

  • SHA256: 30071017c41a92f026e4dc536a6fa628240c03036dfc80fe3d7b70b959b32dfd
  • Pointer size: 131 Bytes
  • Size of remote file: 862 kB
saliency_viz/step_28k/dog_saliency.png ADDED

Git LFS Details

  • SHA256: 84636d533f4aede90c9a383cbbb81dd29802e58befbbd3936c26ae29c5215764
  • Pointer size: 131 Bytes
  • Size of remote file: 775 kB
saliency_viz/step_28k/horse_saliency.png ADDED

Git LFS Details

  • SHA256: fb017a93d3bf2333d33a18b6f7e670e73d007e693900ac7afa3b7905c37715f3
  • Pointer size: 131 Bytes
  • Size of remote file: 407 kB
saliency_viz/step_35k/cat_saliency.png ADDED

Git LFS Details

  • SHA256: a2bc69ba85a308462f6db228a81194f16aa3477ba4942cead92853a896736715
  • Pointer size: 131 Bytes
  • Size of remote file: 840 kB
saliency_viz/step_35k/dog_saliency.png ADDED

Git LFS Details

  • SHA256: 6c5a623689bff62d0276d35fda95e18943b31b7e94161496ed711a172f99e059
  • Pointer size: 131 Bytes
  • Size of remote file: 728 kB
saliency_viz/step_35k/horse_saliency.png ADDED

Git LFS Details

  • SHA256: cc5182eb4ea41c77ff02c4be64c7f4db32d0faa64552bdc523b12d8182212828
  • Pointer size: 131 Bytes
  • Size of remote file: 363 kB
saliency_viz/step_36k/cat_saliency.png ADDED

Git LFS Details

  • SHA256: 4fe5422998c63860d8a178f9f71f3941fe09d121ff20759f2485c07276336af2
  • Pointer size: 131 Bytes
  • Size of remote file: 837 kB
saliency_viz/step_36k/dog_saliency.png ADDED

Git LFS Details

  • SHA256: b984b167a3d8217d5e65ceb11166fb0b8ccb58c6a5ce3cb8824640bd2fdf6bdb
  • Pointer size: 131 Bytes
  • Size of remote file: 729 kB
saliency_viz/step_36k/horse_saliency.png ADDED

Git LFS Details

  • SHA256: c3f18d2cabf128ac8571baee9227965620d73cf87a8e37a515652d213178fa9d
  • Pointer size: 131 Bytes
  • Size of remote file: 368 kB
saliency_viz/step_36k_cpu/cat_saliency.png ADDED

Git LFS Details

  • SHA256: ec203cdfa2d45c1fd2b93047a363abf6b5c70faa1ac80497c96365f01d2ef478
  • Pointer size: 131 Bytes
  • Size of remote file: 837 kB
saliency_viz/step_36k_cpu/dog_saliency.png ADDED

Git LFS Details

  • SHA256: b9daac334fb848b2d55f129c2324978be08c3a79859b757c77b837698988c739
  • Pointer size: 131 Bytes
  • Size of remote file: 729 kB
saliency_viz/step_36k_cpu/horse_saliency.png ADDED

Git LFS Details

  • SHA256: 1cd891a0f0b2e646ac7a7912e9eab3b878d062568fd01fdb8169d599102b1ff5
  • Pointer size: 131 Bytes
  • Size of remote file: 368 kB
training_state.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1dfb625a59763051f3f84e92da3f087dbcb359847ae12a04190279c9ad4053ac
3
+ size 198657138