Instructions to use mis2/ViSS-8B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use mis2/ViSS-8B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="mis2/ViSS-8B")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("mis2/ViSS-8B")
model = AutoModelForMultimodalLM.from_pretrained("mis2/ViSS-8B", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use mis2/ViSS-8B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "mis2/ViSS-8B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mis2/ViSS-8B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/mis2/ViSS-8B

SGLang

How to use mis2/ViSS-8B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "mis2/ViSS-8B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mis2/ViSS-8B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "mis2/ViSS-8B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mis2/ViSS-8B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use mis2/ViSS-8B with Docker Model Runner:
```
docker model run hf.co/mis2/ViSS-8B
```

ViSS-8B / code /docs /EVALUATION.md

mis2

Release ViSS-8B weights, processors and source code

78ac017 verified 21 days ago

preview code

Raw History Blame Contribute Delete

4.26 kB

# Inference, TVS, and scoring

Run from the repository root after [installation](INSTALL.md) and
[data preparation](DATA.md). The code separates model inference, view-policy
composition, and official scoring.

## 1. Generate recent-view proposals

Create a local configuration from the public template and your own assets:

```bash
python -m scripts.configure_evaluation_protocol \
  --streaming-samples data/streamingbench/queries.jsonl \
  --ovo-samples data/ovo/queries.jsonl \
  --processor-config checkpoints/viss/locked_processor/preprocessor_config.json \
  --output configs/evaluation/recent4_full_v1.json
```

The generated file contains local paths and file identities. It remains
private and is excluded from the source archive. Then validate coverage:

```bash
python -m scripts.validate_evaluation_protocol \
  --protocol configs/evaluation/recent4_full_v1.json
```

This check requires the external benchmark exports. It is not a data-free
installation test.

Use the command in the main README with the prepared checkpoint and locked
processor. The runner accepts `--benchmark streamingbench` or
`--benchmark ovo`; annotations, query exports, and media must match.

Use `--record-choice-confidence` when the proposals will be passed to TVS.
For a reproducibility run, supply `--model-manifest` with the matching model
identity artifact. The same identity must be retained across all views.
The runner also supports `--start-index` and `--end-index` for sharding.

The optional multi-GPU launcher is:

```text
bash scripts/run_recent4_single_benchmark.sh \
  REPO MODEL MODEL_MANIFEST OUTPUT PYTHON BENCHMARK GPU_CSV MODEL_ID
```

It additionally requires the benchmark paths documented by its environment
variable checks. Start with the Python entry point when preparing new assets.

## 2. Apply the TVS acquisition and commitment policy

The event policy is `causalbudget20`; the compact-history policy is
`causalhybrid32`. Use the same checkpoint and the view encodings prescribed
by your protocol. Selecting a policy name alone does not fix image resolution
or the processor configuration.

Given recent-view predictions and a frozen calibration artifact:

```bash
python -m experiments.select_uncertainty_escalation_uids \
  --recent outputs/recent.json \
  --threshold data/calibration/thresholds.json \
  --samples data/benchmark/queries.jsonl \
  --output-uids outputs/event_uids.json \
  --output-samples outputs/event_queries.jsonl \
  --output-audit outputs/selection.json
```

Run event-view inference on the selected queries. Then compose the first
two stages:

```bash
python -m experiments.materialize_uncertainty_temporal_agent \
  --recent outputs/recent.json \
  --temporal outputs/event.json \
  --selection-audit outputs/selection.json \
  --threshold-audit data/calibration/thresholds.json \
  --output outputs/two_view.json
```

Once the required compact-history proposals are available:

```bash
python -m experiments.materialize_confidence_gated_third_view_agent \
  --recent outputs/recent.json \
  --event outputs/event.json \
  --base-agent outputs/two_view.json \
  --third-view outputs/history.json \
  --threshold-audit data/calibration/thresholds.json \
  --output outputs/tvs.json
```

These commands name user-generated files; no prediction or score artifacts
are shipped. Materialization combines existing proposals and does not call
the VLM. `materialize_tvs_view_oracle.py` uses reference correctness for
analysis and is not the deployed selection policy.

## 3. Score saved predictions

```bash
python -m scripts.score_official_benchmark_predictions \
  --benchmark streamingbench \
  --annotation data/streamingbench/annotations.json \
  --predictions outputs/tvs.json \
  --official-root external/StreamingBench \
  --output-json outputs/streamingbench_score.json
```

Use `--benchmark ovo` and the corresponding official assets for OVO-Bench.
The adapter checks query coverage and invokes the original scorer. For OVO,
the primary metric is the task-then-group macro average. The official scorer
may have additional dependencies; install those from its own repository.

Generated predictions and scores are your run outputs. They are excluded
from version control and from the source-release archive.

	# Inference, TVS, and scoring

	Run from the repository root after [installation](INSTALL.md) and
	[data preparation](DATA.md). The code separates model inference, view-policy
	composition, and official scoring.

	## 1. Generate recent-view proposals

	Create a local configuration from the public template and your own assets:

	```bash
	python -m scripts.configure_evaluation_protocol \
	--streaming-samples data/streamingbench/queries.jsonl \
	--ovo-samples data/ovo/queries.jsonl \
	--processor-config checkpoints/viss/locked_processor/preprocessor_config.json \
	--output configs/evaluation/recent4_full_v1.json
	```

	The generated file contains local paths and file identities. It remains
	private and is excluded from the source archive. Then validate coverage:

	```bash
	python -m scripts.validate_evaluation_protocol \
	--protocol configs/evaluation/recent4_full_v1.json
	```

	This check requires the external benchmark exports. It is not a data-free
	installation test.

	Use the command in the main README with the prepared checkpoint and locked
	processor. The runner accepts `--benchmark streamingbench` or
	`--benchmark ovo`; annotations, query exports, and media must match.

	Use `--record-choice-confidence` when the proposals will be passed to TVS.
	For a reproducibility run, supply `--model-manifest` with the matching model
	identity artifact. The same identity must be retained across all views.
	The runner also supports `--start-index` and `--end-index` for sharding.

	The optional multi-GPU launcher is:

	```text
	bash scripts/run_recent4_single_benchmark.sh \
	REPO MODEL MODEL_MANIFEST OUTPUT PYTHON BENCHMARK GPU_CSV MODEL_ID
	```

	It additionally requires the benchmark paths documented by its environment
	variable checks. Start with the Python entry point when preparing new assets.

	## 2. Apply the TVS acquisition and commitment policy

	The event policy is `causalbudget20`; the compact-history policy is
	`causalhybrid32`. Use the same checkpoint and the view encodings prescribed
	by your protocol. Selecting a policy name alone does not fix image resolution
	or the processor configuration.

	Given recent-view predictions and a frozen calibration artifact:

	```bash
	python -m experiments.select_uncertainty_escalation_uids \
	--recent outputs/recent.json \
	--threshold data/calibration/thresholds.json \
	--samples data/benchmark/queries.jsonl \
	--output-uids outputs/event_uids.json \
	--output-samples outputs/event_queries.jsonl \
	--output-audit outputs/selection.json
	```

	Run event-view inference on the selected queries. Then compose the first
	two stages:

	```bash
	python -m experiments.materialize_uncertainty_temporal_agent \
	--recent outputs/recent.json \
	--temporal outputs/event.json \
	--selection-audit outputs/selection.json \
	--threshold-audit data/calibration/thresholds.json \
	--output outputs/two_view.json
	```

	Once the required compact-history proposals are available:

	```bash
	python -m experiments.materialize_confidence_gated_third_view_agent \
	--recent outputs/recent.json \
	--event outputs/event.json \
	--base-agent outputs/two_view.json \
	--third-view outputs/history.json \
	--threshold-audit data/calibration/thresholds.json \
	--output outputs/tvs.json
	```

	These commands name user-generated files; no prediction or score artifacts
	are shipped. Materialization combines existing proposals and does not call
	the VLM. `materialize_tvs_view_oracle.py` uses reference correctness for
	analysis and is not the deployed selection policy.

	## 3. Score saved predictions

	```bash
	python -m scripts.score_official_benchmark_predictions \
	--benchmark streamingbench \
	--annotation data/streamingbench/annotations.json \
	--predictions outputs/tvs.json \
	--official-root external/StreamingBench \
	--output-json outputs/streamingbench_score.json
	```

	Use `--benchmark ovo` and the corresponding official assets for OVO-Bench.
	The adapter checks query coverage and invokes the original scorer. For OVO,
	the primary metric is the task-then-group macro average. The official scorer
	may have additional dependencies; install those from its own repository.

	Generated predictions and scores are your run outputs. They are excluded
	from version control and from the source-release archive.