Instructions to use Nanbeige/Nanbeige4.2-3B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Nanbeige/Nanbeige4.2-3B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Nanbeige/Nanbeige4.2-3B", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("Nanbeige/Nanbeige4.2-3B", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Nanbeige/Nanbeige4.2-3B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Nanbeige/Nanbeige4.2-3B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Nanbeige/Nanbeige4.2-3B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Nanbeige/Nanbeige4.2-3B

SGLang

How to use Nanbeige/Nanbeige4.2-3B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Nanbeige/Nanbeige4.2-3B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Nanbeige/Nanbeige4.2-3B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Nanbeige/Nanbeige4.2-3B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Nanbeige/Nanbeige4.2-3B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Nanbeige/Nanbeige4.2-3B with Docker Model Runner:
```
docker model run hf.co/Nanbeige/Nanbeige4.2-3B
```

Issue: Grammar-constrained decoding (`response_format: json_schema`) fails at sampler init — but `--no-jinja` fixes it

by dmhagar - opened 4 days ago

Discussion

dmhagar

4 days ago

Grammar-constrained decoding (`response_format: json_schema`) fails at sampler init — but `--no-jinja` fixes it

First, thanks for shipping this — the looped-transformer 3B is genuinely nice to run locally, and once I got past the issue below it produces clean, sensible structured output.

The problem. With the default jinja chat path, any request that uses a grammar / JSON-schema response_format fails at sampler initialization, before a single token is generated:

HTTP 400
{"error":{"code":400,"type":"invalid_request_error",
 "message":"Failed to initialize samplers: Unexpected empty grammar stack after accepting piece:  assistant (13886)"}}

Note the accepted piece is the assistant-turn marker token " assistant" (id 13886) — the grammar sampler is being fed a trigger derived from the chat template's assistant marker, and the grammar stack empties on it.

Reproduction (built from the nanbeige42 branch, git log -1 = "support nanbeige4.2 model"):

llama-server -m Nanbeige4.2-3B-Q4_K_M.gguf -c 4096 --port 8096   # jinja is default

curl -s http://127.0.0.1:8096/v1/chat/completions -H "content-type: application/json" -d '{
  "model":"nanbeige-4.2-3b",
  "messages":[{"role":"system","content":"Extract entities as strict JSON."},
              {"role":"user","content":"Kael the smith greets you at the Iron Forge in Dunhold."}],
  "response_format":{"type":"json_schema","json_schema":{"name":"extraction","strict":false,
    "schema":{"type":"object","properties":{"entities":{"type":"array","items":{"type":"object",
      "properties":{"name":{"type":"string"},"type":{"type":"string"}},"required":["name","type"]}}},
      "required":["entities"]}}},
  "max_tokens":300,"temperature":0
}'
# -> HTTP 400, "empty grammar stack after accepting piece:  assistant (13886)"

Plain (non-grammar) generation is unaffected — the model answers normally, and enable_thinking:false gives clean non-reasoning output.

The fix / workaround. Adding --no-jinja bypasses it completely. The identical grammar request then returns HTTP 200 with valid, correct JSON:

{"entities":[
  {"name":"Kael","type":"person"},
  {"name":"smith","type":"occupation"},
  {"name":"Iron Forge","type":"location"},
  {"name":"Dunhold","type":"location"}
]}

Because --no-jinja fixes it, the bug is isolated to the jinja chat path — specifically the auto-parser / grammar-trigger construction (common/chat-auto-parser-generator.cpp and the surrounding common/chat.cpp trigger handling). Something on that path installs a grammar trigger that resolves to the assistant marker token even for a plain json_schema request (no tools, tool_choice unset), and common_sampler_init then can't build the grammar. Overriding the surface chat template (--chat-template chatml) does not help — only disabling jinja does — which points at the trigger-generation logic rather than the template text itself.

Why it matters. This blocks the model as a backend for anything that relies on constrained decoding — JSON extraction, tool/function calling, or any grammar-guided output — under the default configuration. --no-jinja is a viable stopgap for pure schema-constrained JSON, but it also disables the reasoning/tool-call template handling, so it isn't a fix for the tool-calling case.

Happy to share the full server logs or test more configurations if useful. Thanks again for the model.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment

Issue: Grammar-constrained decoding (`response_format: json_schema`) fails at sampler init — but `--no-jinja` fixes it

Grammar-constrained decoding (response_format: json_schema) fails at sampler init — but --no-jinja fixes it

Grammar-constrained decoding (`response_format: json_schema`) fails at sampler init — but `--no-jinja` fixes it