Instructions to use DevelopingDad/Gemma4-12B-it-Fast with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use DevelopingDad/Gemma4-12B-it-Fast with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="DevelopingDad/Gemma4-12B-it-Fast")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("DevelopingDad/Gemma4-12B-it-Fast")
model = AutoModelForMultimodalLM.from_pretrained("DevelopingDad/Gemma4-12B-it-Fast", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use DevelopingDad/Gemma4-12B-it-Fast with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "DevelopingDad/Gemma4-12B-it-Fast"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "DevelopingDad/Gemma4-12B-it-Fast",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/DevelopingDad/Gemma4-12B-it-Fast

SGLang

How to use DevelopingDad/Gemma4-12B-it-Fast with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "DevelopingDad/Gemma4-12B-it-Fast" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "DevelopingDad/Gemma4-12B-it-Fast",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "DevelopingDad/Gemma4-12B-it-Fast" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "DevelopingDad/Gemma4-12B-it-Fast",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use DevelopingDad/Gemma4-12B-it-Fast with Docker Model Runner:
```
docker model run hf.co/DevelopingDad/Gemma4-12B-it-Fast
```

Gemma4-12B-it-Fast / serving /README.md

DevelopingDad

Update weight-only checkpoint and generic DGX Spark serving package

231d9a6 verified 2 days ago

preview code

Raw History Blame Contribute Delete

4.21 kB

Optional pinned vLLM recipe

This package provides a mixed NVFP4/FP8 weight-only target with higher-precision activations, plus the existing experimental MTP8 / FP8 draft-body/private-head runtime overlays for one ARM64 DGX Spark GB10.

Download

Use an ARM64 Linux host with the NVIDIA container runtime configured:

hf download DevelopingDad/Gemma4-12B-it-Fast --local-dir Gemma4-12B-it-Fast
cd Gemma4-12B-it-Fast
sha256sum --check SHA256SUMS
hf download google/gemma-4-12B-it-assistant --revision 46d4c6f13f0ac0ad827b915669b8df9b81c64c51 --local-dir assistant

For a frozen deployment pass a Hub commit with --revision to the target download too. The separately downloaded Google assistant is not bundled. artifact-manifest.json fences target and assistant bytes.

Inspect and run

python3 serving/launch.py
python3 serving/launch.py --run

The first command prints a Docker plan. The second verifies target/assistant hashes, overlay hashes and the pinned image's original source checksums before creating a new container. Existing containers are not replaced or stopped. The default name is gemma4-fast-server, bound to loopback port 8000. Use --port, --name, --model-dir, --assistant-dir and --cache-dir to choose local paths and naming. The target, assistant and overlays are read-only; cache and per-boot integrity receipts use the chosen writable cache directory.

Default resource/inference settings:

Setting	Value
Image	`vllm/vllm-openai@sha256:3f7dd5b777d34d1724456ce71f87385dca288c3bb23029ab27dee358f5d2b971`
Total context	65,536 tokens, prompt plus output
Maximum sequences / batched tokens	8 / 4,096
KV	FP8, 60 GiB
Target	BF16 unquantized components; W4A16 MLP / W8A16 attention
Speculation	MTP8, pinned Google assistant, FP8 body/private head
Prefill	Chunked, long-prefill threshold 1,024 with adaptive control
Prefix caching	Enabled
Template / default reasoning	Native-format compatibility / off
Container memory ceiling	96 GiB
Restart policy	`no`

The 60 GiB KV allocation requires considerable unified memory. The container ceiling is not a guarantee of host headroom; reserve memory for the OS and other workloads. Eight slots do not establish eight simultaneous full-window capacity. For cold-performance comparisons pass --disable-prefix-caching. This changes only the prefix-cache flag, leaving the checkpoint and other settings fixed. Automatic restart and reboot recovery are not qualified by this package.

Request controls

Enable reasoning with chat_template_kwargs.enable_thinking=true. Optional thinking_token_budget=2048 is a top-level API request field in the pinned runtime. Set a larger total max_tokens to leave room for the final answer. The launcher does not enforce a global reasoning or total completion budget. Early reasoning cutoffs can cause incomplete answers or visible reasoning.

The optional native-compat-chat-template.jinja is generated from the preserved root native template. It adds only system-block whitespace (a space, or a newline before a supplied system message) to match the measured rendering. It adds no task-specific contract, response suffix, automatic repair, or retry workflow. Use explicit JSON schemas for strict JSON output. Tool and media parsers retain the model's native protocol. Their availability does not establish comprehensive semantic or multimodal quality.

Integrity and attribution

The optional overlay quantizes a separate draft output head and checks that the target's tied BF16 embedding/output head remains protected. Per-boot receipts are archived before restart. overlay-manifest.json binds the modified source to the pinned vLLM source; upstream notices remain intact.

The launcher is checked against the existing native serving configuration and receives CPU/source/checksum validation. Publication does not imply a fresh GPU restart of this exact portable wrapper or qualification on other systems. See generic diagnostics, provenance, LICENSE, NOTICE and MODIFICATIONS.md.