fix(vision): unpack processor output by keyword in native fusion

Transformers image processors return BatchFeature (a UserDict, not a dict
subclass), so the isinstance(..., dict) check failed and the processor output
was passed positionally into the encoder's input_ids parameter. Against a real
encoder this raised ValueError: You have to specify input_ids.

Test against collections.abc.Mapping instead, and add a regression test using a
UserDict-based processor that fails on the previous code.

Also add AGENTS.md recording verified repo facts, sizing constraints, licensing
corrections and the small-model testing approach.

Co-authored-by: openhands <openhands@all-hands.dev>

AGENTS.md

@@ -0,0 +1,95 @@
+# CORTEX AI — verified facts and constraints
+
+This file records what was *measured* in this workspace, so later sessions do not re-derive it
+or repeat claims that turn out to be wrong.
+
+## Repository state
+
+- `Frankenstein-Labs/Cortex-ai` HEAD = `a82e5d802d597d9429a4ecd1d4d0160e4f4eb4ad`
+  ("feat(cortex-ai): add native vision fusion prototype"), branch `main`, no open PRs, no tags.
+- The repo is a **redistribution** of `deepseek-ai/DeepSeek-V4-Pro-0813`.
+- All 66 `model-*.safetensors` shards plus `model.safetensors.index.json` have SHA-256
+  **identical to upstream** (67/67 LFS objects match). Verified via the HF tree API.
+- There is no `git-lfs` binary in this image. Fetch weights as pointer files only:
+  set `GIT_LFS_SKIP_SMUDGE=1` and neutralise the smudge filter, otherwise a clone pulls ~893 GB
+  into a 25 GB disk.
+
+## Model facts (from `config.json`)
+
+| Field | Value |
+|---|---|
+| architecture | `DeepseekV4ForCausalLM` (`model_type: deepseek_v4`) |
+| hidden_size | 7168 |
+| layers | 61 (+3 MTP) |
+| vocab | 129280 |
+| max positions | 1048576 |
+| routed experts | 384, 6 per token |
+| expert storage | `fp4`; weights FP8 `e4m3` |
+| declared dtype | bfloat16 |
+| tie_word_embeddings | false |
+
+`tie_word_embeddings: false` means there is no `lm_head`-sharing trick for grafting new
+modalities; any new vocabulary or modality embedding needs explicit, trained parameters.
+
+## Sizing reality
+
+- Stored payload ≈ 892.76 GB; ~1.65 T params. Minimum inference memory at 1 byte/param ≈ 1.65 TB.
+- This development sandbox: **4 CPU cores, ~15 GiB RAM, 25 GB disk, no GPU** (`nvidia-smi`
+  absent, no `/dev/nvidia*`).
+- Consequence: the 1.65 T checkpoint **cannot be loaded, run, or fine-tuned here**. Any plan
+  requiring an end-to-end forward pass of the real checkpoint is not executable in this
+  environment. Validate fusion paths against small stand-ins instead.
+
+## Cost warning for "keep the weights untouched" plans
+
+A vision projector must map into a 7168-dim space, but pretrained open projectors map to their
+own models' hidden sizes (e.g. 4096). A randomly-initialised projector produces **noise** into the
+core, so "never train anything" and "native vision that works" are mutually exclusive. Expected
+behaviour must be stated honestly as an untrained prototype, or the projector must be trained.
+Training a projector alone (frozen encoder + frozen core) is the cheap option; full multimodal
+pretraining at 1.65 T is not affordable outside a cluster.
+
+## Licensing corrections (checked against the Hub, not from memory)
+
+Claims that appear in circulating roadmap documents are wrong in several places:
+
+| Model | Claimed | Actual (verified) |
+|---|---|---|
+| Qwen2.5-VL | "non-commercial" | Apache-2.0 (7B and 72B both Apache) |
+| LLaVA | "MIT" | `llama2` license (7B hub repo) |
+| SDXL | "non-commercial RAIL" | OpenRAIL++-M — permits commercial use with use-restrictions |
+| Coqui TTS | "MIT" | Coqui Public Model License — non-commercial |
+| Kimi K2.5 | "Modified MIT, 5B + MoonViT" | Modified MIT, but **1.03 T params**, `image-text-to-text` |
+| HunyuanVideo | "community, non-commercial" | Tencent Hunyuan Community; **explicitly does not apply in EU/UK/South Korea** |
+| Wan2.2 | Apache-2.0 | confirmed Apache-2.0, claims no rights over outputs |
+
+Note the Qwen 2.5 vs 3 split: `Qwen2.5-VL-*` is Apache, but Qwen3-VL and some Qwen3 text models
+carry the `qwen` license. Check each repo individually; never generalise from a family name.
+
+## Known bug found and fixed
+
+`cortex_ai/vision/native.py::encode_image` branched on `isinstance(model_input, dict)`.
+Transformers image processors return `BatchFeature`, which derives from `UserDict`, **not** from
+`dict`. The check therefore failed and the processor output was passed positionally into
+`SiglipModel.forward(input_ids=...)`, raising
+`ValueError: You have to specify input_ids`.
+
+Fixed by testing `isinstance(model_input, Mapping)`. Regression test:
+`tests/test_native_fusion.py::test_processor_output_is_unpacked_by_keyword_not_positionally`
+(fails on the old code, passes on the new).
+
+## Verified fusion path
+
+`NativeVisionFusion` works against a **real** encoder (`google/siglip-base-patch16-224` vision
+tower): 196 image tokens + 8 text tokens -> 204-length fused sequence, logits shaped
+`(1, 204, 1000)`, and embeddings change with image content (red-vs-blue delta 0.46). It does
+**not** yet work with the real 1.65 T core.
+
+## Test commands
+
+```bash
+PYTHONPATH=/workspace/project python3 -m pytest tests/ -q           # 6 tests
+cd encoding && PYTHONPATH=$PWD python3 -m pytest test_encoding_dsv4.py -q   # 4 tests
+```
+
+Requires `torch` (CPU wheel), `torchvision`, `pillow`, `transformers`.

cortex_ai/vision/native.py

@@ -6,6 +6,7 @@ existing 1.65T DeepSeek checkpoint is not modified, and a core is only
 accepted when its runtime exposes the required inputs_embeds contract.
 """
 
+from collections.abc import Mapping
 from dataclasses import dataclass
 from typing import Any, Callable
 
@@ -47,7 +48,13 @@ class NativeVisionFusion(nn.Module):
             if self.image_processor is not None
             else image
         )
-        encoded = self.vision_encoder(**model_input) if isinstance(model_input, dict) else self.vision_encoder(model_input)
+        # Transformers processors return BatchFeature, a UserDict rather than a
+        # dict subclass, so a plain isinstance(..., dict) check would pass the
+        # processor output positionally into the encoder's input_ids parameter.
+        if isinstance(model_input, Mapping):
+            encoded = self.vision_encoder(**model_input)
+        else:
+            encoded = self.vision_encoder(model_input)
         if hasattr(encoded, "last_hidden_state"):
             encoded = encoded.last_hidden_state
         if not isinstance(encoded, Tensor) or encoded.ndim != 3:

tests/test_native_fusion.py

@@ -1,3 +1,5 @@
+from collections import UserDict
+
 import torch
 from torch import nn
 
@@ -31,3 +33,29 @@ def test_native_image_embedding_fusion_executes_real_tensor_path():
     assert result.text_token_count == 2
     assert core.last_inputs_embeds.shape == (1, 5, 8)
     assert result.outputs["attention_mask"].shape == (1, 5)
+
+
+class BatchFeatureLikeEncoder(nn.Module):
+    """Encoder whose processor output is a UserDict, as Transformers processors return."""
+
+    def forward(self, pixel_values):
+        return torch.ones(pixel_values.shape[0], 2, 4)
+
+
+class BatchFeatureLikeProcessor:
+    def __call__(self, image, return_tensors=None):
+        return UserDict({"pixel_values": torch.zeros(1, 3, 16, 16)})
+
+
+def test_processor_output_is_unpacked_by_keyword_not_positionally():
+    core = TinyCore()
+    fusion = build_native_fusion(
+        BatchFeatureLikeEncoder(),
+        core,
+        vision_hidden_size=4,
+        cortex_hidden_size=8,
+        image_processor=BatchFeatureLikeProcessor(),
+    )
+    result = fusion(input_ids=torch.tensor([[1, 2]]), image="unused")
+    assert result.image_token_count == 2
+    assert result.text_token_count == 2