Instructions to use DevelopingDad/Gemma4-12B-it-Fast with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use DevelopingDad/Gemma4-12B-it-Fast with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="DevelopingDad/Gemma4-12B-it-Fast")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("DevelopingDad/Gemma4-12B-it-Fast")
model = AutoModelForMultimodalLM.from_pretrained("DevelopingDad/Gemma4-12B-it-Fast", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use DevelopingDad/Gemma4-12B-it-Fast with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "DevelopingDad/Gemma4-12B-it-Fast"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "DevelopingDad/Gemma4-12B-it-Fast",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/DevelopingDad/Gemma4-12B-it-Fast

SGLang

How to use DevelopingDad/Gemma4-12B-it-Fast with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "DevelopingDad/Gemma4-12B-it-Fast" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "DevelopingDad/Gemma4-12B-it-Fast",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "DevelopingDad/Gemma4-12B-it-Fast" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "DevelopingDad/Gemma4-12B-it-Fast",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use DevelopingDad/Gemma4-12B-it-Fast with Docker Model Runner:
```
docker model run hf.co/DevelopingDad/Gemma4-12B-it-Fast
```

Gemma4-12B-it-Fast / serving /README.md

DevelopingDad

Update weight-only checkpoint and generic DGX Spark serving package

231d9a6 verified 2 days ago

preview code

Raw History Blame Contribute Delete

4.21 kB

# Optional pinned vLLM recipe

This package provides a mixed NVFP4/FP8 weight-only target with higher-precision
activations, plus the existing experimental MTP8 / FP8 draft-body/private-head
runtime overlays for one ARM64 DGX Spark GB10.

## Download

Use an ARM64 Linux host with the NVIDIA container runtime configured:

```sh
hf download DevelopingDad/Gemma4-12B-it-Fast --local-dir Gemma4-12B-it-Fast
cd Gemma4-12B-it-Fast
sha256sum --check SHA256SUMS
hf download google/gemma-4-12B-it-assistant --revision 46d4c6f13f0ac0ad827b915669b8df9b81c64c51 --local-dir assistant
```

For a frozen deployment pass a Hub commit with `--revision` to the target
download too. The separately downloaded Google assistant is not bundled.
`artifact-manifest.json` fences target and assistant bytes.

## Inspect and run

```sh
python3 serving/launch.py
python3 serving/launch.py --run
```

The first command prints a Docker plan. The second verifies target/assistant
hashes, overlay hashes and the pinned image's original source checksums before
creating a new container. Existing containers are not replaced or stopped.
The default name is `gemma4-fast-server`, bound to loopback port 8000.
Use `--port`, `--name`, `--model-dir`, `--assistant-dir` and `--cache-dir` to
choose local paths and naming. The target, assistant and overlays are read-only;
cache and per-boot integrity receipts use the chosen writable cache directory.

Default resource/inference settings:

| Setting | Value |
| --- | --- |
| Image | `vllm/vllm-openai@sha256:3f7dd5b777d34d1724456ce71f87385dca288c3bb23029ab27dee358f5d2b971` |
| Total context | 65,536 tokens, prompt plus output |
| Maximum sequences / batched tokens | 8 / 4,096 |
| KV | FP8, 60 GiB |
| Target | BF16 unquantized components; W4A16 MLP / W8A16 attention |
| Speculation | MTP8, pinned Google assistant, FP8 body/private head |
| Prefill | Chunked, long-prefill threshold 1,024 with adaptive control |
| Prefix caching | Enabled |
| Template / default reasoning | Native-format compatibility / off |
| Container memory ceiling | 96 GiB |
| Restart policy | `no` |

The 60 GiB KV allocation requires considerable unified memory. The container
ceiling is not a guarantee of host headroom; reserve memory for the OS and other
workloads. Eight slots do not establish eight simultaneous full-window capacity.
For cold-performance comparisons pass `--disable-prefix-caching`. This changes
only the prefix-cache flag, leaving the checkpoint and other settings fixed.
Automatic restart and reboot recovery are not qualified by this package.

## Request controls

Enable reasoning with `chat_template_kwargs.enable_thinking=true`. Optional
`thinking_token_budget=2048` is a top-level API request field in the pinned
runtime. Set a larger total `max_tokens` to leave room for the final answer.
The launcher does not enforce a global reasoning or total completion budget.
Early reasoning cutoffs can cause incomplete answers or visible reasoning.

The optional `native-compat-chat-template.jinja` is generated from the preserved
root native template. It adds only system-block whitespace (a space, or a newline
before a supplied system message) to match the measured rendering. It adds no
task-specific contract, response suffix, automatic
repair, or retry workflow. Use explicit JSON schemas for strict JSON output.
Tool and media parsers retain the model's native protocol. Their availability
does not establish comprehensive semantic or multimodal quality.

## Integrity and attribution

The optional overlay quantizes a separate draft output head and checks that the
target's tied BF16 embedding/output head remains protected. Per-boot receipts
are archived before restart. `overlay-manifest.json` binds the modified source
to the pinned vLLM source; upstream notices remain intact.

The launcher is checked against the existing native serving configuration and
receives CPU/source/checksum validation. Publication does not imply a fresh
GPU restart of this exact portable wrapper or qualification on other systems.
See [generic diagnostics](../benchmarks/README.md), [provenance](../provenance.json),
[LICENSE](../LICENSE), [NOTICE](../NOTICE) and [MODIFICATIONS.md](../MODIFICATIONS.md).

	# Optional pinned vLLM recipe

	This package provides a mixed NVFP4/FP8 weight-only target with higher-precision
	activations, plus the existing experimental MTP8 / FP8 draft-body/private-head
	runtime overlays for one ARM64 DGX Spark GB10.

	## Download

	Use an ARM64 Linux host with the NVIDIA container runtime configured:

	```sh
	hf download DevelopingDad/Gemma4-12B-it-Fast --local-dir Gemma4-12B-it-Fast
	cd Gemma4-12B-it-Fast
	sha256sum --check SHA256SUMS
	hf download google/gemma-4-12B-it-assistant --revision 46d4c6f13f0ac0ad827b915669b8df9b81c64c51 --local-dir assistant
	```

	For a frozen deployment pass a Hub commit with `--revision` to the target
	download too. The separately downloaded Google assistant is not bundled.
	`artifact-manifest.json` fences target and assistant bytes.

	## Inspect and run

	```sh
	python3 serving/launch.py
	python3 serving/launch.py --run
	```

	The first command prints a Docker plan. The second verifies target/assistant
	hashes, overlay hashes and the pinned image's original source checksums before
	creating a new container. Existing containers are not replaced or stopped.
	The default name is `gemma4-fast-server`, bound to loopback port 8000.
	Use `--port`, `--name`, `--model-dir`, `--assistant-dir` and `--cache-dir` to
	choose local paths and naming. The target, assistant and overlays are read-only;
	cache and per-boot integrity receipts use the chosen writable cache directory.

	Default resource/inference settings:

	\| Setting \| Value \|
	\| --- \| --- \|
	\| Image \| `vllm/vllm-openai@sha256:3f7dd5b777d34d1724456ce71f87385dca288c3bb23029ab27dee358f5d2b971` \|
	\| Total context \| 65,536 tokens, prompt plus output \|
	\| Maximum sequences / batched tokens \| 8 / 4,096 \|
	\| KV \| FP8, 60 GiB \|
	\| Target \| BF16 unquantized components; W4A16 MLP / W8A16 attention \|
	\| Speculation \| MTP8, pinned Google assistant, FP8 body/private head \|
	\| Prefill \| Chunked, long-prefill threshold 1,024 with adaptive control \|
	\| Prefix caching \| Enabled \|
	\| Template / default reasoning \| Native-format compatibility / off \|
	\| Container memory ceiling \| 96 GiB \|
	\| Restart policy \| `no` \|

	The 60 GiB KV allocation requires considerable unified memory. The container
	ceiling is not a guarantee of host headroom; reserve memory for the OS and other
	workloads. Eight slots do not establish eight simultaneous full-window capacity.
	For cold-performance comparisons pass `--disable-prefix-caching`. This changes
	only the prefix-cache flag, leaving the checkpoint and other settings fixed.
	Automatic restart and reboot recovery are not qualified by this package.

	## Request controls

	Enable reasoning with `chat_template_kwargs.enable_thinking=true`. Optional
	`thinking_token_budget=2048` is a top-level API request field in the pinned
	runtime. Set a larger total `max_tokens` to leave room for the final answer.
	The launcher does not enforce a global reasoning or total completion budget.
	Early reasoning cutoffs can cause incomplete answers or visible reasoning.

	The optional `native-compat-chat-template.jinja` is generated from the preserved
	root native template. It adds only system-block whitespace (a space, or a newline
	before a supplied system message) to match the measured rendering. It adds no
	task-specific contract, response suffix, automatic
	repair, or retry workflow. Use explicit JSON schemas for strict JSON output.
	Tool and media parsers retain the model's native protocol. Their availability
	does not establish comprehensive semantic or multimodal quality.

	## Integrity and attribution

	The optional overlay quantizes a separate draft output head and checks that the
	target's tied BF16 embedding/output head remains protected. Per-boot receipts
	are archived before restart. `overlay-manifest.json` binds the modified source
	to the pinned vLLM source; upstream notices remain intact.

	The launcher is checked against the existing native serving configuration and
	receives CPU/source/checksum validation. Publication does not imply a fresh
	GPU restart of this exact portable wrapper or qualification on other systems.
	See [generic diagnostics](../benchmarks/README.md), [provenance](../provenance.json),
	[LICENSE](../LICENSE), [NOTICE](../NOTICE) and [MODIFICATIONS.md](../MODIFICATIONS.md).