Instructions to use hsmin92/internvl35-fp8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use hsmin92/internvl35-fp8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="hsmin92/internvl35-fp8")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("hsmin92/internvl35-fp8")
model = AutoModelForMultimodalLM.from_pretrained("hsmin92/internvl35-fp8", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use hsmin92/internvl35-fp8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "hsmin92/internvl35-fp8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "hsmin92/internvl35-fp8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/hsmin92/internvl35-fp8

SGLang

How to use hsmin92/internvl35-fp8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "hsmin92/internvl35-fp8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "hsmin92/internvl35-fp8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "hsmin92/internvl35-fp8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "hsmin92/internvl35-fp8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use hsmin92/internvl35-fp8 with Docker Model Runner:
```
docker model run hf.co/hsmin92/internvl35-fp8
```

InternVL3.5-4B-HF FP8 Dynamic

This repository contains a compressed-tensors FP8 Dynamic quantization of OpenGVLab/InternVL3_5-4B-HF, prepared for memory-conscious vLLM serving.

Important runtime note

The checkpoint stores the language decoder Linear weights in FP8 E4M3 format. On an NVIDIA Ampere GPU such as the RTX 3070, vLLM 0.26.0 serves these weights through its W8A16 FP8 path (Humming kernel): weights are compressed to 8-bit, while activations run in FP16. This is primarily a VRAM-saving configuration; a speedup is not guaranteed on Ampere.

Quantization scope

Quantized:

252 language decoder Linear modules
Scheme: FP8_DYNAMIC
Weight format: FP8 E4M3
Activation scaling: dynamic per token at runtime
Calibration dataset: not required

Kept in BF16:

Vision tower
Multimodal projector
Input embeddings
lm_head
Normalization layers and other protected parameters

The checkpoint was generated from the base-model revision:

6bd4487402110ef9889ba50eb7aefeb302526fed

See quantization/recipe.py for the compression recipe.

Verified environment

The following setup was used for the initial serving validation:

Component	Version / value
GPU	NVIDIA GeForce RTX 3070 8GB
Host	Windows + WSL2
WSL distribution	Ubuntu 24.04
NVIDIA driver	591.86
vLLM	0.26.0
PyTorch	2.11.0+cu130
Transformers	5.14.1
Quantization backend	compressed-tensors
vLLM runner	V1
Attention	FlashAttention 2
Max context used in validation	2,048 tokens
Maximum images per request	4

Observed during startup with the verified preset:

Model weight memory: approximately 5.51 GiB
Available KV-cache memory: approximately 0.59 GiB
GPU KV-cache capacity: 4,288 tokens
Validated RTX 3070 profile: --gpu-memory-utilization 0.80 with --kv-cache-memory-bytes 384M, using approximately 6.3-6.4 GiB for the vLLM process

Windows graphics applications consume additional VRAM outside the vLLM process. The total value shown by Windows nvidia-smi can therefore be higher.

Quick start: WSL2 + RTX 3070

RTX 3070 runtime profile: The included startup script disables the V2 model runner, FlashInfer sampling JIT, and DeepGEMM. It uses the Ampere-compatible Humming W8A16 kernel for the FP8 checkpoint.

1. Install system build requirements

Humming compiles a small runtime extension on first use.

sudo apt update
sudo apt install -y build-essential python3.12-dev

Do not install a Linux NVIDIA display driver inside WSL2. The Windows NVIDIA driver exposes libcuda.so under /usr/lib/wsl/lib.

2. Create the Python environment

Install uv first when it is not already available.

uv venv --python 3.12 .venv-vllm
source .venv-vllm/bin/activate
uv pip install "vllm==0.26.0" hf_xet

3. Validate the environment

./scripts/check_wsl_runtime.sh

4. Start the server directly from Hugging Face

source .venv-vllm/bin/activate
./scripts/start_vllm_wsl_rtx3070.sh

The default model ID is:

hsmin92/internvl35-fp8

Successful startup ends with:

Application startup complete.

The OpenAI-compatible endpoint is then available at:

http://127.0.0.1:8000/v1

Optional: lower KV-cache preset

The default script uses the startup configuration that was validated first. To reduce the fixed KV-cache allocation, set both values together:

GPU_MEMORY_UTILIZATION=0.80 \
KV_CACHE_MEMORY_BYTES=384M \
./scripts/start_vllm_wsl_rtx3070.sh

The fixed KV-cache option does not replace GPU_MEMORY_UTILIZATION; both are needed by vLLM 0.26.0. Validate this preset on the target machine because the Windows desktop and browser processes also consume VRAM.

Runtime options

The startup script accepts environment variables:

Variable	Default	Description
`MODEL_ID`	`hsmin92/internvl35-fp8`	Hub model ID or local model path
`SERVED_MODEL_NAME`	`internvl35-fp8`	Name exposed by the API
`HOST`	`127.0.0.1`	Listen address
`PORT`	`8000`	Listen port
`MAX_MODEL_LEN`	`2048`	Total context budget, including image and output tokens
`MAX_NUM_SEQS`	`1`	Maximum concurrent sequences
`MAX_IMAGES`	`4`	Maximum images in one request
`GPU_MEMORY_UTILIZATION`	`0.80`	RTX 3070 GPU-memory safety target
`KV_CACHE_MEMORY_BYTES`	`384M`	Fixed KV-cache size for the RTX 3070 8GB profile

Example:

PORT=8100 MAX_IMAGES=1 MAX_MODEL_LEN=1024 \
./scripts/start_vllm_wsl_rtx3070.sh

API tests

Health and model list

curl -s http://127.0.0.1:8000/health
curl -s http://127.0.0.1:8000/v1/models | python3 -m json.tool

Text request

./examples/chat_text.sh

Local image request

python examples/chat_image.py /path/to/image.jpg \
  "Describe the scene and list any safety-relevant events."

The image client sends the local image as a base64 data URL and uses only the Python standard library.

Native Linux and other GPUs

The WSL2 script deliberately applies compatibility settings required by the validated RTX 3070 environment:

VLLM_USE_V2_MODEL_RUNNER=0 because the V2 runner required UVA in this WSL setup.
VLLM_USE_FLASHINFER_SAMPLER=0 because FlashInfer sampling JIT required nvcc.
/usr/lib/wsl/lib is added to the compile and runtime linker paths.
pip-installed CUDA NVRTC libraries are added to LD_LIBRARY_PATH.
--enforce-eager disables CUDA graphs and torch.compile for compatibility.

Native Linux systems with a full CUDA Toolkit or newer GPUs may not need these workarounds. Start from the documented script, then remove compatibility flags one at a time and validate output quality, memory, and stability.

Intended use

This model is suitable for experimentation with:

Image understanding
Multi-image comparison
CCTV frame summarization
Visual question answering
OpenAI-compatible multimodal API integration

For video analysis on an 8GB GPU, sample a small number of frames externally, resize them appropriately, and send the frames as multiple images rather than passing every frame of a video.

Limitations

This is a quantized derivative, not an independently trained model.
The vision tower and output head remain BF16 and account for a meaningful portion of the loaded weights.
FP8 on Ampere is served through a W8A16 compatibility kernel rather than native FP8 Tensor Core execution.
A comprehensive quality benchmark against the BF16 base model has not yet been published in this repository.
The first server start may compile and cache Humming runtime components.
VRAM figures depend on driver, desktop applications, context length, multimodal limits, and vLLM version.

Attribution and license

This repository is a quantized derivative of OpenGVLab/InternVL3_5-4B-HF. The original project and this derivative are distributed under the Apache-2.0 license. Review the upstream model card for the original training details, limitations, and citation information.

Citation

@article{wang2025internvl3_5,
  title={InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency},
  author={Wang, Weiyun and Gao, Zhangwei and Gu, Lixin and Pu, Hengjun and Cui, Long and Wei, Xingguang and Liu, Zhaoyang and Jing, Linglin and Ye, Shenglong and Shao, Jie and others},
  journal={arXiv preprint arXiv:2508.18265},
  year={2025}
}