Expand usage, use cases, performance details, and runnable examples

9524435 verified 23 days ago

11.1 kB

---
language:
- en
base_model:
- answerdotai/ModernBERT-base
- google/siglip-base-patch16-224
tags:
- kodama
- typed-decisions
- custom-code
- safetensors
- multimodal
inference: false
---
# Kodama Vision

Experimental image-and-text decision model combining a ModernBERT-base decision model, frozen SigLIP, and a trained image adapter. Formerly `jevlike-mm-proto`. The frozen encoder and its tokenizer are included under `vision/`.

All models return structured `Choice`, `Score`, and `Noul` decisions. They score answer options directly; they do not generate conversational responses. These are custom `jevlike` checkpoints, loaded with the bundled runtime below. Generic Transformers `pipeline()` / `AutoModel` loading is not the supported entry point.

## Which Kodama model should I use?

| Model | Inputs | Typical reason to choose it |
|---|---|---|
| [Kodama Core](https://huggingface.co/Cem13/kodama-core) | Text; JSON serialized as text | Fast routing, classification, and small decisions inside an application |
| [Kodama Vision](https://huggingface.co/Cem13/kodama-vision) | One image plus optional text | Experimental object, vehicle, or document-type classification |
| [Kodama Sense](https://huggingface.co/Cem13/kodama-sense) | Text, images, PDFs, charts, audio, video, and mixed media | Decisions that require visual or audio evidence |

These models choose from the answers you supply. They are not conversational chatbots,
open-ended report writers, transcription services, or general document extraction systems.
The use cases below are applications to evaluate on your own examples; the benchmark
results do not establish reliable performance on every such application.

## Practical use cases

| Application | Example question | Important scope |
|---|---|---|
| Image-library categorization | “Which object category is shown?” | Classify among the labels you provide |
| Vehicle-photo experiments | “Which vehicle category best fits this image?” | Strong local car results do not establish accuracy on every vehicle collection |
| Document-type triage | “Is this an invoice, letter, form, or report?” | Coarse page appearance; small printed text may be unreadable |
| Visual checks in a workflow | “Does the picture show a red object?” | One image and a structured question |

Vision is an experimental specialist. Use Sense for native PDFs, chart interpretation,
audio, video, or combined modalities. Vision resizes each image to 224×224 pixels;
it is not an OCR engine or a document-field extraction model. In the small controlled
chart/PDF comparison, Vision scored 0/8 in each group, so chart arithmetic is not a
supported performance claim for this checkpoint.

## Install and download

Tested environment: Python 3.12, PyTorch 2.10, Transformers 5.2.0, Linux, and an NVIDIA
RTX 4050 Laptop GPU with 6 GB VRAM. Python 3.10+ is supported by the package. Install a
CUDA-enabled PyTorch build appropriate for your system when using the GPU examples.
Approximate model download size: **1.48 GB**, excluding Python/CUDA dependencies and caches.

```bash
python -m pip install huggingface_hub
# For a private repository, authenticate once with: hf auth login
hf download Cem13/kodama-vision --local-dir kodama-vision
python -m pip install "./kodama-vision/runtime[model,ingest]" "transformers==5.2.0"
```

Run the examples from the directory containing `kodama-vision`. After the download and
dependency installation, prediction runs locally; no API key or paid inference service
is needed for prediction. Private downloads require access to the repository. An existing
`HF_TOKEN` can authenticate the Hub client without adding a token to source code.

## Python: classify an image

```python
from PIL import Image
from jevlike import Choice, Noul
from jevlike.mm import MultimodalSystemOne
model = MultimodalSystemOne.load(
    "kodama-vision", device="cuda", vision="kodama-vision/vision"
)
questions = {
    "color": Choice("Which color dominates the picture?", ["red", "green", "blue"]),
    "red": Noul("The picture is predominantly red."),
}
image = Image.new("RGB", (224, 224), "red")  # Self-contained demonstration.
result = model.predict(None, questions, image=image)
print(result["answers"])
# For your own input: model.predict("Optional context", questions, image="photo.jpg")
```

Pass the bundled `vision/` path to use the included frozen SigLIP weights and tokenizer.
The `image` argument accepts a local image path, PIL image, or an RGB NumPy array.
`None` is a valid state when an image provides the evidence. A runnable example is included:

```bash
python kodama-vision/examples/quickstart.py
```

### Multiple images and text-only calls

```python
results = model.predict_batch([
    (None, questions, "first.jpg"),
    (None, questions, "second.jpg"),
])
```

Each tuple supplies one state, its questions, and one image. A call with no image uses
the underlying text model. For image calls, long text is truncated to the available
512-token sequence budget; image slots, instructions, and options also consume that budget.
Use concise context. CPU execution is available with `device="cpu"`, but has not been
speed-benchmarked. This release does not expose a dedicated Vision image-upload HTTP server.

## Understanding the answers

Every response contains `model`, `latency_ms`, and `answers`, keyed by your question names.

| Question type | How to define it | Main output |
|---|---|---|
| `Choice` | 2–255 distinct labels, optionally with descriptions | `choice` and a label-to-probability dictionary |
| `Score` | 2–10 ordered descriptions, lowest first | Zero-based `level`, its `label`, a probability list, and expected `score` scaled to 0–10 |
| `Noul` | A statement to test; no answer options | `noul`, the probability that the statement is true |

`confidence` is the largest answer probability. For `Noul`, a confident **false** answer
can have `noul=0.02` and `confidence=0.98`; use `noul` when deciding whether the statement
is true. `escalate` and `escalate_prob` help flag uncertain answers for another step in
your application. Neither confidence nor escalation is a guarantee of correctness.

Use specific instructions and descriptive, distinct options. Probabilities are conditional
on the supplied options; adding an `other` option can be useful when your label set does
not cover every input, but the model's use of that option still needs validation.

Core and Vision apply saved temperature calibration by default. Their `escalate_prob`
comes from a separately trained escalation head with saved calibration; default escalation
is above 0.5. Set `calibrated=False` on prediction calls to inspect uncalibrated outputs.
Their answer dictionaries do not include Sense's per-answer `calibrated` flag.

## Local evaluation

Measured on an NVIDIA RTX 4050 Laptop GPU (6 GB). Latency includes tokenization and excludes model loading and HTTP.

| Evaluation split | Decisions | Accuracy |
|---|---:|---:|
| Caltech101 objects | 1,371 | 92.85% |
| Cars | 1,153 | 97.66% |
| RVL, eight unseen document types | 1,675 | 58.99% |
| In-domain validation | 3,407 | 88.02% |

See `evaluation/vision_full.json` for exact metrics and split details. In-domain
validation is not an unseen test. The smaller shared comparison used 48 object,
48 car, and 64 document examples; those sample results should not be confused
with the full evaluation above.

## Timing details and interpretation

Vision does not have a dedicated repeated-call latency benchmark comparable to the
Core/Sense timing suites. In the earlier small shared evaluation, per-decision medians
were **87.5 ms** for objects (48 decisions), **72.4 ms** for cars (48), and **46.5 ms** for
document types (64). These are evaluation-run timings, not controlled throughput or p95
measurements. No batch-throughput claim is made for Vision.

A separate packaged-model smoke check used **1.28 GiB** peak PyTorch GPU allocation while
loading and scoring a simple image. That is one smoke workload, not a worst-case memory
bound. The frozen SigLIP branch uses BF16 on CUDA; the decision model uses float32 weights
with BF16 autocast. Image decoding, size, option counts, and batch lengths affect latency.

The full accuracy table mixes held-out object/car/document splits with an explicitly
labelled in-domain validation split. It is not directly comparable to Sense's much smaller
shared image subset. On that shared car subset Vision scored 93.75% and Sense 83.33%, each
on 48 examples; on document types Sense scored 78.13% and Vision 57.81%, each on 64.
See [evaluation/vision_full.json](evaluation/vision_full.json) and
[evaluation/shared_image_comparison.json](evaluation/shared_image_comparison.json).
In the shared comparison, `existing` means Vision for image groups and Core for text groups.
The memory smoke measurement is saved in [evaluation/package_smoke.json](evaluation/package_smoke.json).

## Training and limitations

The base encoders were pretrained by their upstream authors. Kodama adds local
training/adaptation; these foundation models were not pretrained from scratch.
Included training metadata describes the local run. Evaluation data are not
redistributed in this repository. Public pretraining overlap is unknown, and
small benchmark samples should not be treated as broad capability guarantees.
Accuracy depends on task and option wording. Confidence is not a guarantee of
correctness; use suitable validation before relying on decisions.

## Attribution and licensing

The pretrained base components identify their license as Apache-2.0. Upstream cards, source revisions, and the Apache license text are preserved in `third_party/` and `NOTICE`. No separate public license has been selected for the original Kodama additions and runtime in this release; the upstream license statements should not be read as a new license grant for those additions.

Base sources:

- [answerdotai/ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base/tree/8949b909ec900327062f0ebf497f51aef5e6f0c8)
- [google/siglip-base-patch16-224](https://huggingface.co/google/siglip-base-patch16-224/tree/7fd15f0689c79d79e38b1c2e2e2370a7bf2761ed)

## Troubleshooting and reproducibility

- **401/403 while downloading:** authenticate with an account allowed to read the repository.
- **Model configuration / AutoModel error:** use the bundled `jevlike` runtime and the custom loader shown above.
- **Import or tokenizer mismatch:** use the documented Transformers version and install the runtime extras for this model.
- **Different answers near a tie:** numerical precision and batch shapes can affect probabilities; measure your own task before changing inference settings.

`release_manifest.json` records file hashes and pinned base revisions. The runtime is
bundled under `runtime/`; `examples/quickstart.py` provides a complete local example.
The pretrained foundations were not trained from scratch by Kodama. Earlier evaluation
artifacts use `jevlike` for Core, `omni` for Sense, and `prototype` for Vision.

	---
	language:
	- en
	base_model:
	- answerdotai/ModernBERT-base
	- google/siglip-base-patch16-224
	tags:
	- kodama
	- typed-decisions
	- custom-code
	- safetensors
	- multimodal
	inference: false
	---

	# Kodama Vision

	Experimental image-and-text decision model combining a ModernBERT-base decision model, frozen SigLIP, and a trained image adapter. Formerly `jevlike-mm-proto`. The frozen encoder and its tokenizer are included under `vision/`.

	All models return structured `Choice`, `Score`, and `Noul` decisions. They score answer options directly; they do not generate conversational responses. These are custom `jevlike` checkpoints, loaded with the bundled runtime below. Generic Transformers `pipeline()` / `AutoModel` loading is not the supported entry point.

	## Which Kodama model should I use?

	\| Model \| Inputs \| Typical reason to choose it \|
	\|---\|---\|---\|
	\| [Kodama Core](https://huggingface.co/Cem13/kodama-core) \| Text; JSON serialized as text \| Fast routing, classification, and small decisions inside an application \|
	\| [Kodama Vision](https://huggingface.co/Cem13/kodama-vision) \| One image plus optional text \| Experimental object, vehicle, or document-type classification \|
	\| [Kodama Sense](https://huggingface.co/Cem13/kodama-sense) \| Text, images, PDFs, charts, audio, video, and mixed media \| Decisions that require visual or audio evidence \|

	These models choose from the answers you supply. They are not conversational chatbots,
	open-ended report writers, transcription services, or general document extraction systems.
	The use cases below are applications to evaluate on your own examples; the benchmark
	results do not establish reliable performance on every such application.

	## Practical use cases

	\| Application \| Example question \| Important scope \|
	\|---\|---\|---\|
	\| Image-library categorization \| “Which object category is shown?” \| Classify among the labels you provide \|
	\| Vehicle-photo experiments \| “Which vehicle category best fits this image?” \| Strong local car results do not establish accuracy on every vehicle collection \|
	\| Document-type triage \| “Is this an invoice, letter, form, or report?” \| Coarse page appearance; small printed text may be unreadable \|
	\| Visual checks in a workflow \| “Does the picture show a red object?” \| One image and a structured question \|

	Vision is an experimental specialist. Use Sense for native PDFs, chart interpretation,
	audio, video, or combined modalities. Vision resizes each image to 224×224 pixels;
	it is not an OCR engine or a document-field extraction model. In the small controlled
	chart/PDF comparison, Vision scored 0/8 in each group, so chart arithmetic is not a
	supported performance claim for this checkpoint.

	## Install and download

	Tested environment: Python 3.12, PyTorch 2.10, Transformers 5.2.0, Linux, and an NVIDIA
	RTX 4050 Laptop GPU with 6 GB VRAM. Python 3.10+ is supported by the package. Install a
	CUDA-enabled PyTorch build appropriate for your system when using the GPU examples.
	Approximate model download size: 1.48 GB, excluding Python/CUDA dependencies and caches.

	```bash
	python -m pip install huggingface_hub
	# For a private repository, authenticate once with: hf auth login
	hf download Cem13/kodama-vision --local-dir kodama-vision
	python -m pip install "./kodama-vision/runtime[model,ingest]" "transformers==5.2.0"
	```

	Run the examples from the directory containing `kodama-vision`. After the download and
	dependency installation, prediction runs locally; no API key or paid inference service
	is needed for prediction. Private downloads require access to the repository. An existing
	`HF_TOKEN` can authenticate the Hub client without adding a token to source code.

	## Python: classify an image

	```python
	from PIL import Image
	from jevlike import Choice, Noul
	from jevlike.mm import MultimodalSystemOne

	model = MultimodalSystemOne.load(
	"kodama-vision", device="cuda", vision="kodama-vision/vision"
	)
	questions = {
	"color": Choice("Which color dominates the picture?", ["red", "green", "blue"]),
	"red": Noul("The picture is predominantly red."),
	}
	image = Image.new("RGB", (224, 224), "red") # Self-contained demonstration.
	result = model.predict(None, questions, image=image)
	print(result["answers"])
	# For your own input: model.predict("Optional context", questions, image="photo.jpg")
	```

	Pass the bundled `vision/` path to use the included frozen SigLIP weights and tokenizer.
	The `image` argument accepts a local image path, PIL image, or an RGB NumPy array.
	`None` is a valid state when an image provides the evidence. A runnable example is included:

	```bash
	python kodama-vision/examples/quickstart.py
	```

	### Multiple images and text-only calls

	```python
	results = model.predict_batch([
	(None, questions, "first.jpg"),
	(None, questions, "second.jpg"),
	])
	```

	Each tuple supplies one state, its questions, and one image. A call with no image uses
	the underlying text model. For image calls, long text is truncated to the available
	512-token sequence budget; image slots, instructions, and options also consume that budget.
	Use concise context. CPU execution is available with `device="cpu"`, but has not been
	speed-benchmarked. This release does not expose a dedicated Vision image-upload HTTP server.

	## Understanding the answers

	Every response contains `model`, `latency_ms`, and `answers`, keyed by your question names.

	\| Question type \| How to define it \| Main output \|
	\|---\|---\|---\|
	\| `Choice` \| 2–255 distinct labels, optionally with descriptions \| `choice` and a label-to-probability dictionary \|
	\| `Score` \| 2–10 ordered descriptions, lowest first \| Zero-based `level`, its `label`, a probability list, and expected `score` scaled to 0–10 \|
	\| `Noul` \| A statement to test; no answer options \| `noul`, the probability that the statement is true \|

	`confidence` is the largest answer probability. For `Noul`, a confident false answer
	can have `noul=0.02` and `confidence=0.98`; use `noul` when deciding whether the statement
	is true. `escalate` and `escalate_prob` help flag uncertain answers for another step in
	your application. Neither confidence nor escalation is a guarantee of correctness.

	Use specific instructions and descriptive, distinct options. Probabilities are conditional
	on the supplied options; adding an `other` option can be useful when your label set does
	not cover every input, but the model's use of that option still needs validation.

	Core and Vision apply saved temperature calibration by default. Their `escalate_prob`
	comes from a separately trained escalation head with saved calibration; default escalation
	is above 0.5. Set `calibrated=False` on prediction calls to inspect uncalibrated outputs.
	Their answer dictionaries do not include Sense's per-answer `calibrated` flag.

	## Local evaluation

	Measured on an NVIDIA RTX 4050 Laptop GPU (6 GB). Latency includes tokenization and excludes model loading and HTTP.

	\| Evaluation split \| Decisions \| Accuracy \|
	\|---\|---:\|---:\|
	\| Caltech101 objects \| 1,371 \| 92.85% \|
	\| Cars \| 1,153 \| 97.66% \|
	\| RVL, eight unseen document types \| 1,675 \| 58.99% \|
	\| In-domain validation \| 3,407 \| 88.02% \|

	See `evaluation/vision_full.json` for exact metrics and split details. In-domain
	validation is not an unseen test. The smaller shared comparison used 48 object,
	48 car, and 64 document examples; those sample results should not be confused
	with the full evaluation above.

	## Timing details and interpretation

	Vision does not have a dedicated repeated-call latency benchmark comparable to the
	Core/Sense timing suites. In the earlier small shared evaluation, per-decision medians
	were 87.5 ms for objects (48 decisions), 72.4 ms for cars (48), and 46.5 ms for
	document types (64). These are evaluation-run timings, not controlled throughput or p95
	measurements. No batch-throughput claim is made for Vision.

	A separate packaged-model smoke check used 1.28 GiB peak PyTorch GPU allocation while
	loading and scoring a simple image. That is one smoke workload, not a worst-case memory
	bound. The frozen SigLIP branch uses BF16 on CUDA; the decision model uses float32 weights
	with BF16 autocast. Image decoding, size, option counts, and batch lengths affect latency.

	The full accuracy table mixes held-out object/car/document splits with an explicitly
	labelled in-domain validation split. It is not directly comparable to Sense's much smaller
	shared image subset. On that shared car subset Vision scored 93.75% and Sense 83.33%, each
	on 48 examples; on document types Sense scored 78.13% and Vision 57.81%, each on 64.
	See [evaluation/vision_full.json](evaluation/vision_full.json) and
	[evaluation/shared_image_comparison.json](evaluation/shared_image_comparison.json).
	In the shared comparison, `existing` means Vision for image groups and Core for text groups.
	The memory smoke measurement is saved in [evaluation/package_smoke.json](evaluation/package_smoke.json).

	## Training and limitations

	The base encoders were pretrained by their upstream authors. Kodama adds local
	training/adaptation; these foundation models were not pretrained from scratch.
	Included training metadata describes the local run. Evaluation data are not
	redistributed in this repository. Public pretraining overlap is unknown, and
	small benchmark samples should not be treated as broad capability guarantees.
	Accuracy depends on task and option wording. Confidence is not a guarantee of
	correctness; use suitable validation before relying on decisions.

	## Attribution and licensing

	The pretrained base components identify their license as Apache-2.0. Upstream cards, source revisions, and the Apache license text are preserved in `third_party/` and `NOTICE`. No separate public license has been selected for the original Kodama additions and runtime in this release; the upstream license statements should not be read as a new license grant for those additions.

	Base sources:

	- [answerdotai/ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base/tree/8949b909ec900327062f0ebf497f51aef5e6f0c8)
	- [google/siglip-base-patch16-224](https://huggingface.co/google/siglip-base-patch16-224/tree/7fd15f0689c79d79e38b1c2e2e2370a7bf2761ed)

	## Troubleshooting and reproducibility

	- 401/403 while downloading: authenticate with an account allowed to read the repository.
	- Model configuration / AutoModel error: use the bundled `jevlike` runtime and the custom loader shown above.
	- Import or tokenizer mismatch: use the documented Transformers version and install the runtime extras for this model.
	- Different answers near a tie: numerical precision and batch shapes can affect probabilities; measure your own task before changing inference settings.

	`release_manifest.json` records file hashes and pinned base revisions. The runtime is
	bundled under `runtime/`; `examples/quickstart.py` provides a complete local example.
	The pretrained foundations were not trained from scratch by Kodama. Earlier evaluation
	artifacts use `jevlike` for Core, `omni` for Sense, and `prototype` for Vision.