Instructions to use google/gemma-4-31B-it with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use google/gemma-4-31B-it with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="google/gemma-4-31B-it") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("google/gemma-4-31B-it") model = AutoModelForMultimodalLM.from_pretrained("google/gemma-4-31B-it", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Inference
- HuggingChat
- Notebooks
- Google Colab
- Kaggle
- AMD Developer Cloud
- Local Apps Settings
- vLLM
How to use google/gemma-4-31B-it with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "google/gemma-4-31B-it" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "google/gemma-4-31B-it", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/google/gemma-4-31B-it
- SGLang
How to use google/gemma-4-31B-it with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "google/gemma-4-31B-it" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "google/gemma-4-31B-it", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "google/gemma-4-31B-it" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "google/gemma-4-31B-it", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use google/gemma-4-31B-it with Docker Model Runner:
docker model run hf.co/google/gemma-4-31B-it
Failure to initiate the reasoning process. Happens at 29 000 - 31 000 tokens deep in chat, quite consistently.
The moment it reaches 29K to 31K chat depth - it starts to fail at reasoning, outputting the final answer immediately instead. This does not concern the tool calling, most likely (I'm not using it for agentic tasks).
No idea whether it's .jinja issue or something is wrong with llama.cpp or the model itself, but it's been like that for a long time already, for more than 1 month.
I haven't tested .safetensors but I can surely say that every GGUF file out there is affected, including some community-made finetunes. It doesn't matter whether .mmproj is involved or not. Personally, I encountered it with Q4KM / Q6 / Q8 variants, and with the official QAT version too.
How to reproduce the issue: Give it a complex, large enough system prompt in the ballpark of 10K to 20K tokens, and fill the chat with many short, conversational messasges until the context reaches 30K tokens. Overall context window doesn't matter for as long as it's got the room to fit these 30K to reveal the issue.
How often the issue manifests: Once the sufficient chat depth is reached, I'd say there's about 80% chance it'll fail to initiate reasoning in every following generation. Re-generating sometimes helps, but it does seem like it'll get even more inclined not to reason at all if the chat continues further into 30K+ context.
Sampler settings: using the officially recommended temperature, top_k, etc.
I've also found this Reddit thread where other users complained about a similar issue about 4 months ago: https://www.reddit.com/r/LocalLLaMA/comments/1sfr5jh/gemma_4_31b_silently_stops_reasoning_on_complex/
After having a chat with some people, I've decided to close this discussion for now.
Will reopen it if/when a more definite proof of an issue appears.
No point keeping it open, considering there are "everything's fine" claims.
After a month of fumbling around, redownloading other quants of the model and even attempting to use a different hardware - I haven't found anything definite, due to my general lack of specialised knowledge.
However, the model's behavior (at least in llama.cpp) remains bizarrely inconsistent, still failing to initiate the reasoning process after a certain breaking point I mentioned before (= a big set of system instructions, ~30K context depth, many short exchanges (20 to 40 input messages and the model's output conterparts) in chat).
I am still not sure whether this is somehow contained to Windows-based systems but it does seem to be unlikely. Whether this issue is unique to llama.cpp, or does it appear in VLLM or Exllama v3, for instance, I also do not know. Nonetheless, I think it is worth investigating. Is it the model's flaw? Is there something wrong with llama.cpp?
One thing I am sure of: it doesn't matter if the end user picks the "raw" llama.cpp or Unsloth Studio or LM Studio or anything else, up to date and properly configured, with the latest .jinja template: the result is always the same, reasoning breaks abruptly no matter what.
Something is not right!
It's llama.cpp. With all due respect, that thing is too bloated and barely anyone is keeping eye on what's going on with no-so-hot models (coding community deemed Gemma 4 as "meh"..... there were claims of certain PRs boosting the inference speed and it might have backfired in weird ways). Using EXL3 quants and @turboderp 's exllama v3 - no such problems whatsoever. I'm glad I switched and I'll never look back.
Running it via ExllamaV3 + TabbyAPI resulted in the same thing (mind you, that would be entirely different model files!). Around 30K context, with lots and lots of short, conversational messages and a sufficiently long system prompt (which it handles exceptionally well!) - it CEASES to reason. No more <|channel>thought output at the start of the reply. It reasons fine for 30 or 40 replies straight - not even once skipping the reasoning and then it just doesn't do it anymore. Abruptly. No, it's not a frontend issue either. I don't think I'm making a mistake anywhere. I've exhausted all the options.
Anyway, sad to see this falling on deaf ears. I was sceptical of my own findings for a long time, but now I'm inclined to believe this is something Google should be concerned with, not the community / llama.cpp / exllama / whatever else developers and contributors.