akeshkumar commited on
Commit
65caa84
·
verified ·
1 Parent(s): ebf13d5

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +61 -53
README.md CHANGED
@@ -7,8 +7,8 @@ tags:
7
  - pope
8
  - system-one-vision
9
  - non-autoregressive
10
- - experimentation
11
  - research
 
12
  pipeline_tag: visual-question-answering
13
  model-index:
14
  - name: SnapVL
@@ -72,28 +72,37 @@ model-index:
72
  value: 67.00
73
  ---
74
 
75
- # SnapVL: An Experimental Study on System-1 Vision
76
 
77
- > **Status:** Research Experimentation & Exploration
78
- > **Goal:** Investigating the capabilities and limitations of non-autoregressive visual decision-making without an LLM text decoder.
79
- > **Checkpoint Note:** All benchmark evaluations reported in this card were conducted on **early, pre-convergence experimental checkpoints** (training was paused early for analytical understanding and has not been trained to full loss convergence).
80
 
81
  ---
82
 
83
- ## 1. Project Overview & Motivation
 
 
 
 
 
 
 
 
 
 
84
 
85
- Most modern vision-language models (VLMs) rely on large autoregressive language models (LLMs) to generate answers token-by-token. While this approach enables conversational abilities, it introduces substantial latency (1–3 seconds per query), high compute requirements, and text-prior hallucinations.
86
 
87
- **SnapVL** is an experimental project created to study an alternative paradigm: **System-1 Vision**.
88
- In cognitive science, System-1 represents fast, instinctual, reflex-driven perception. We wanted to answer a practical engineering question:
89
 
90
- > *Can a compact (~450M parameter) non-autoregressive model make accurate, instant visual decisions (~1.71s per batch of 256 / ~6.7 ms per sample on 2x T4 GPUs) when constrained to structured schemas (Boolean verification and multi-choice options)?*
91
 
92
  ---
93
 
94
  ## 2. Model Architecture
95
 
96
- The architecture is deliberately kept minimal to focus on single forward-pass decision making:
97
 
98
  ```
99
  [Image: 224x224] ──► CLIP ViT-L/14 (Top-4 Layers Unfrozen) ──► 256 Patch Tokens (1024-d)
@@ -104,70 +113,71 @@ The architecture is deliberately kept minimal to focus on single forward-pass de
104
  Cross-Attention Fusion (2 Layers, 512-d)
105
  │
106
  ▼
107
- Schema Head (Task-Specific Output)
108
- ├─► Boolean: Single logit -> Sigmoid
109
  └─► Choice: Cosine similarity with candidate choices
110
  ```
111
 
112
- * **Backbone:** OpenAI CLIP ViT-L/14 (Vision) + CLIP Text (Frozen).
113
- * **Trainable Parameters:** ~57.2M (Top 4 Transformer blocks of ViT + Fusion + Schema Head). Total model parameters: ~450M.
114
- * **Inference Mechanism:** Single forward pass (no autoregressive decoding loop).
115
- * **Empirical Throughput & Latency:** **1.71s (±0.02s)** per batch of 256 (`128 * 2` on 2x NVIDIA T4 GPUs), translating to **~6.68 ms per sample** (~150 samples/second throughput).
 
 
 
 
 
116
 
117
  ---
118
 
119
- ## 3. Experimental Results
120
 
121
- The model was evaluated across three benchmarks to test spatial reasoning, factual recognition, and object hallucination under structured decision schemas.
122
 
123
- > [!NOTE]
124
- > **Pre-Convergence Evaluations:**
125
- > All benchmarks reported below (GQA, VQA-v2, and POPE) were evaluated on **intermediate experimental checkpoints before reaching full loss convergence**. Training was stopped early to analyze behavior and architectural trade-offs; these metrics do not represent fully converged weights.
126
 
127
- ### A. Benchmark Metrics Summary
128
-
129
- | Benchmark | Task / Scope | Evaluation Protocol | Loss | Accuracy | Checkpoint Status |
130
  |---|---|---|:---:|:---:|---|
131
  | **GQA** | Spatial & Relational Reasoning | Closed-Schema (Boolean & 4-Choice) | **0.3481** | **82.81%** | Pre-convergence intermediate checkpoint |
132
  | **VQA-v2** | Factual Recognition | Closed-Schema (Boolean & 4-Choice) | **0.4901** | **78.77%** | Pre-convergence intermediate checkpoint |
133
- | **POPE** | Object Hallucination Evaluation | 9,000 Yes/No Queries (Standard Binary) | — | **78.73%** | Pre-convergence intermediate checkpoint |
134
- | **SNLI-VE** | Visual Entailment | Boolean (Entailment vs Contradiction) | **0.6321** | **67.00%** | **Pure Zero-Shot:** Completely unseen during training (+17% over random baseline) |
 
 
 
 
 
135
 
136
  ### B. POPE Benchmark Breakdown (9,000 Samples)
137
 
138
- The **POPE (Polling-based Object Probing Evaluation)** benchmark was used to test whether the model hallucinates objects that are not present in the scene:
139
 
140
- | POPE Split | Samples | Accuracy | Precision | Recall | F1-Score | Yes-Ratio |
141
  |---|:---:|:---:|:---:|:---:|:---:|:---:|
142
  | **Random** | 3,000 | 81.90% | 85.74% | 76.53% | 80.87% | 44.63% |
143
  | **Popular** | 3,000 | 79.70% | 81.71% | 76.53% | 79.04% | 46.83% |
144
  | **Adversarial** | 3,000 | 74.60% | 73.90% | 76.07% | 74.97% | 51.47% |
145
  | **Overall Average** | **9,000** | **78.73%** | **80.15%** | **76.38%** | **78.22%** | **47.64%** |
146
 
147
- *Observation on POPE:* Generative models often have a high Yes-ratio (60–70%) due to language co-occurrence priors. SnapVL achieved a balanced **47.64% Yes-ratio**, confirming that its decisions are driven directly by visual verification rather than text frequency biases.
148
 
149
  ---
150
 
151
- ## 4. What We Learned: Strengths vs. Limitations
152
-
153
- This experimentation highlighted clear boundaries for non-autoregressive decision models:
154
 
155
- ### What Worked Well:
156
- 1. **Inference Speed:** At **1.71s (±0.02s) per batch of 256** (~6.68 ms per sample across 2x T4 GPUs), the model is suitable for real-time robotic perception, drone navigation, or video frame filtering where multi-second LLM latency is unusable.
157
- 2. **Deterministic Output:** Because output is constrained to a schema (`boolean` or `choice`), there is zero risk of JSON parsing errors, prompt injection, or conversational drift.
158
- 3. **Spatial Perception:** Direct attention over the 16x16 visual patch grid proved effective for spatial questions (*"is X to the left of Y?"*).
159
- 4. **Zero-Shot Visual Entailment Transfer:** The model was never trained on any SNLI-VE data. Despite this dataset being completely unseen, the model scored **67.00% accuracy (Loss: 0.6321)** on SNLI-VE, demonstrating a clear +17.0% margin above the 50.0% random baseline purely from representations learned on VQA and GQA.
160
 
161
- ### What Did Not Work / Architectural Trade-offs:
162
- 1. **Fine-Grained Linguistic Logic vs Grounding:** While the model transfers cleanly to zero-shot entailment (67.00%), reaching higher precision (80%+) on nuanced multi-clause sentence negation likely requires a scaled textual representation or language decoder rather than a frozen CLIP text encoder.
163
- 2. **Evaluation Protocol Difference:**
164
- Unlike generative models that generate open-ended text from a 32,000+ vocabulary, SnapVL operates as a **closed-schema classifier** (choosing among candidates or evaluating boolean truth). The high accuracy on GQA/VQA reflects this structured decision format.
165
- 3. **No Conversational Ability:**
166
- SnapVL cannot generate paragraphs, explain reasoning steps, or engage in open-ended chat. It is strictly a decision model.
167
 
168
  ---
169
 
170
- ## 5. Usage & Inference Example
171
 
172
  ```python
173
  import torch
@@ -179,7 +189,7 @@ from models.schema_head import SchemaHead, ArchitectureBModel
179
 
180
  device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
181
 
182
- # 1. Initialize architecture
183
  vision_enc = CLIPViTEncoder(unfreeze_last_n=4).to(device)
184
  text_enc = CLIPTextEncoder().to(device)
185
  fusion = CrossAttentionFusion(vision_dim=1024, text_dim=768, hidden_dim=512).to(device)
@@ -192,21 +202,19 @@ checkpoint = torch.load(ckpt_path, map_location="cpu", weights_only=False)
192
  model.load_state_dict(checkpoint.get("model_state_dict", checkpoint), strict=False)
193
  model.eval()
194
 
195
- # 3. Model supports structured queries:
196
- # - Boolean schema: question + image -> True / False
197
- # - Choice schema: question + image + candidate list -> Argmax Choice
198
  ```
199
 
200
  ---
201
 
202
- ## 6. Citation & Context
203
-
204
- This repository is shared as an experimental artifact for researchers and engineers exploring non-autoregressive vision-language interfaces and real-time multimodal decision making.
205
 
206
  ```bibtex
207
  @misc{snapvl_experiment_2026,
208
  author = {Kumar, Akesh},
209
- title = {SnapVL: Experimental Exploration of Non-Autoregressive System-1 Vision Decision Models},
210
  year = {2026},
211
  publisher = {Hugging Face},
212
  howpublished = {\url{https://huggingface.co/akeshkumar/SnapVL}}
 
7
  - pope
8
  - system-one-vision
9
  - non-autoregressive
 
10
  - research
11
+ - experimentation
12
  pipeline_tag: visual-question-answering
13
  model-index:
14
  - name: SnapVL
 
72
  value: 67.00
73
  ---
74
 
75
+ # SnapVL: An Experimental Non-Autoregressive Vision-Language Decision Model
76
 
77
+ > **Status:** Exploratory Research Checkpoint
78
+ > **Evaluation Stage:** Pre-Convergence (Exploratory Training Run)
79
+ > **Paradigm:** System-1-Inspired Visual Decision Modeling (Non-Autoregressive)
80
 
81
  ---
82
 
83
+ ## Important Disclaimers & Evaluation Scope
84
+
85
+ 1. **Pre-Convergence Checkpoint:** The metrics reported in this document were evaluated on an intermediate experimental checkpoint where training was paused early to analyze representation dynamics. These numbers do not represent fully converged final weights.
86
+ 2. **Closed-Schema Evaluation Protocol:** Unlike generative vision-language models that decode answers token-by-token from an open vocabulary, SnapVL operates as a **closed-schema decision classifier**. Questions are framed either as Boolean verification (`True`/`False`) or multiple-choice candidate selection (`N-choice`). Direct numerical comparison with generative LLM decoders should take this protocol difference into account.
87
+ 3. **Scope:** SnapVL is designed strictly for discrete visual decision-making, not for open-ended conversation or free-form text generation.
88
+
89
+ ---
90
+
91
+ ## 1. Research Motivation
92
+
93
+ Standard Vision-Language Models (VLMs) typically couple a vision encoder with an autoregressive Large Language Model (LLM). While this architecture provides conversational flexibility, it introduces non-trivial inference latency due to autoregressive decoding loops, high memory footprints, and potential susceptibility to language-prior affirmation biases.
94
 
95
+ **SnapVL** investigates an alternative question:
96
 
97
+ > *How effectively can a compact (~450M parameter) model perform structured visual decision-making using a single forward pass without an autoregressive text generator?*
 
98
 
99
+ This project explores System-1-inspired visual decision modeling—focusing on fast, single-pass perceptual decisions over structured outputs (Boolean truth values and candidate choice selections).
100
 
101
  ---
102
 
103
  ## 2. Model Architecture
104
 
105
+ The architecture is streamlined for single forward-pass execution without token decoding loops:
106
 
107
  ```
108
  [Image: 224x224] ──► CLIP ViT-L/14 (Top-4 Layers Unfrozen) ──► 256 Patch Tokens (1024-d)
 
113
  Cross-Attention Fusion (2 Layers, 512-d)
114
  │
115
  ▼
116
+ Schema Head (Task Output)
117
+ ├─► Boolean: Logit -> Sigmoid
118
  └─► Choice: Cosine similarity with candidate choices
119
  ```
120
 
121
+ ### Architectural Breakdown:
122
+ * **Vision Backbone:** OpenAI CLIP ViT-L/14 (24 Transformer layers; top 4 layers unfrozen during stage 3 fine-tuning).
123
+ * **Text Backbone:** OpenAI CLIP ViT-L/14 Text Encoder (Frozen).
124
+ * **Fusion Module:** 2-layer Cross-Attention (512 hidden dimension).
125
+ * **Schema Head:** Shared projection mapping fused multimodal states into task-specific decision outputs (`boolean` and `choice`).
126
+ * **Parameter Counts:**
127
+ * Total Parameters: ~450 Million
128
+ * Trainable Parameters: ~57.2 Million (Top 4 ViT blocks + Cross-Attention Fusion + Schema Head)
129
+ * **Measured Batched Throughput:** **1.71s (±0.02s)** per batch of 256 samples (`128 * 2` across 2x NVIDIA T4 GPUs), corresponding to an amortized **~6.68 ms per sample** in batched execution (~150 samples/second). *Note: This represents amortized multi-GPU batched throughput, not single-sample serial latency.*
130
 
131
  ---
132
 
133
+ ## 3. Experimental Results (Pre-Convergence)
134
 
135
+ The checkpoint was evaluated across multiple benchmark datasets under its structured decision protocol:
136
 
137
+ ### A. Benchmark Summary
 
 
138
 
139
+ | Dataset | Task Type | Evaluation Protocol | Loss | Accuracy | Checkpoint Status |
 
 
140
  |---|---|---|:---:|:---:|---|
141
  | **GQA** | Spatial & Relational Reasoning | Closed-Schema (Boolean & 4-Choice) | **0.3481** | **82.81%** | Pre-convergence intermediate checkpoint |
142
  | **VQA-v2** | Factual Recognition | Closed-Schema (Boolean & 4-Choice) | **0.4901** | **78.77%** | Pre-convergence intermediate checkpoint |
143
+ | **POPE** | Object Probing (Hallucination) | Standard Binary Verification (9,000 queries) | — | **78.73%** | Pre-convergence intermediate checkpoint |
144
+ | **SNLI-VE** | Visual Entailment | Boolean (Entailment vs Contradiction) | **0.6321** | **67.00%** | **Zero-Shot:** Model was never trained on SNLI-VE (17 percentage points above 50% random baseline) |
145
+
146
+ > [!IMPORTANT]
147
+ > **Protocol Distinction:** These results are not directly comparable to standard open-ended VQA/GQA leaderboard scores because SnapVL evaluates a transformed closed-schema task rather than unrestricted answer generation.
148
+
149
+ ---
150
 
151
  ### B. POPE Benchmark Breakdown (9,000 Samples)
152
 
153
+ The **POPE (Polling-based Object Probing Evaluation)** benchmark evaluates visual faithfulness by asking whether specific objects exist in an image across three sampling splits:
154
 
155
+ | Split | Query Count | Accuracy | Precision | Recall | F1-Score | Yes-Ratio |
156
  |---|:---:|:---:|:---:|:---:|:---:|:---:|
157
  | **Random** | 3,000 | 81.90% | 85.74% | 76.53% | 80.87% | 44.63% |
158
  | **Popular** | 3,000 | 79.70% | 81.71% | 76.53% | 79.04% | 46.83% |
159
  | **Adversarial** | 3,000 | 74.60% | 73.90% | 76.07% | 74.97% | 51.47% |
160
  | **Overall Average** | **9,000** | **78.73%** | **80.15%** | **76.38%** | **78.22%** | **47.64%** |
161
 
162
+ *Observation:* While generative models frequently display an unbalanced Yes-ratio (often 60–70%) due to language co-occurrence priors, SnapVL exhibited a **47.64% Yes-ratio** output distribution across the 9,000 queries, showing a relatively balanced positive/negative distribution across splits without exhibiting extreme affirmation skew.
163
 
164
  ---
165
 
166
+ ## 4. Empirical Observations & Limitations
 
 
167
 
168
+ ### Positive Findings:
169
+ 1. **Batched Throughput:** At ~6.68 ms per sample amortized across a 256-sample batch on 2x T4 GPUs (1.71s per batch), non-autoregressive decision architectures provide efficient throughput for parallel processing pipelines (e.g., frame filtering, offline verification).
170
+ 2. **Schema-Constrained Outputs:** Formatting outputs as structured primitives (`boolean` or `choice`) prevents textual parsing failures and unbounded response lengths.
171
+ 3. **Zero-Shot Transfer on Entailment:** The model was not trained on any SNLI-VE data. On unseen SNLI-VE validation data, it achieved **67.00% accuracy (Loss: 0.6321)**—17 percentage points above the 50% random baseline—demonstrating that cross-attention representations trained on VQA/GQA transferred partially to binary visual entailment without task-specific fine-tuning.
 
172
 
173
+ ### Technical Limitations & Trade-offs:
174
+ 1. **Linguistic Complexity & Text Representation:** Because the CLIP text encoder remains frozen, fine-grained compositional and syntactic structures (such as nested sentence negations) face representation limitations inherent to a frozen contrastive encoder.
175
+ 2. **Protocol Specificity:** Performance on GQA (82.81%) and VQA-v2 (78.77%) relies on pre-defined choice candidates and Boolean formats. The model cannot answer unconstrained, open-ended questions outside these schemas.
176
+ 3. **Lack of Generative Capability:** The model cannot explain its reasoning, produce multi-sentence descriptions, or hold conversational dialog.
 
 
177
 
178
  ---
179
 
180
+ ## 5. Quick Usage & Verification
181
 
182
  ```python
183
  import torch
 
189
 
190
  device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
191
 
192
+ # 1. Instantiate architecture
193
  vision_enc = CLIPViTEncoder(unfreeze_last_n=4).to(device)
194
  text_enc = CLIPTextEncoder().to(device)
195
  fusion = CrossAttentionFusion(vision_dim=1024, text_dim=768, hidden_dim=512).to(device)
 
202
  model.load_state_dict(checkpoint.get("model_state_dict", checkpoint), strict=False)
203
  model.eval()
204
 
205
+ # 3. Model supports structured evaluation:
206
+ # - Boolean schema: question + image -> True / False logit
207
+ # - Choice schema: question + image + candidate list -> Argmax over candidate embeddings
208
  ```
209
 
210
  ---
211
 
212
+ ## 6. Citation
 
 
213
 
214
  ```bibtex
215
  @misc{snapvl_experiment_2026,
216
  author = {Kumar, Akesh},
217
+ title = {SnapVL: An Experimental Non-Autoregressive Vision-Language Decision Model},
218
  year = {2026},
219
  publisher = {Hugging Face},
220
  howpublished = {\url{https://huggingface.co/akeshkumar/SnapVL}}