Instructions to use nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16")
model = AutoModelForCausalLM.from_pretrained("nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
HuggingChat
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16

SGLang

How to use nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 with Docker Model Runner:
```
docker model run hf.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16
```

Add community evaluation results

by SaylorTwift HF Staff - opened Aug 12

base: refs/heads/main

←

from: refs/pr/4

Discussion Files changed

Aug 12

PR Description: Add Evaluation Results for nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16

Summary

This PR adds evaluation results extracted from the model card's "Benchmarks" table for nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 to the .eval_results/ directory, following the Hugging Face Hub evaluation-results specification.

Benchmarks Added

Benchmarks Skipped (Not Registered on Hub)

The following benchmarks are reported on the model card but could not be added because they do not have a registered eval.yaml on the Hugging Face Hub:

AA-Omniscience: 17.50
SciCode: 32.60
Terminal-Bench 2.1: 24.58 — Hub only registers Terminal-Bench 2.0 (harborframework/terminal-bench-2.0); skipped due to version mismatch rather than mapped to a different version.
PinchBench: 85.37
BrowseComp: 36.97
τ³-bench (Banking): 9.28
GDPval-AA-V2: 832
IFBench (loose): 71.88
AA-LCR: 52.00

These can be added once the benchmark authors register their eval.yaml on the Hub.

Source

Model card: https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16

Files Added

.eval_results/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.yaml

Verification

These results were extracted from the model card's own published benchmark table. NVIDIA states they were measured under a consistent internal harness (NeMo Gym / Nemo Evaluator SDK), which may differ from vendors' self-reported numbers for the comparison models shown on the same card. No verified token is provided as these were not run via HF Jobs with inspect-ai.

Add community evaluation resultsd7c4f9ba

Nerdsking

Aug 15

In other words: Horrible. Almost any garage Chinese company can beat you, specially in coding. And it amazes me how much incompetence can be found in a giant company.
What are the causes? D&I hirings? Lack of vision? Low salaries? Lack of resources? A mistery.
Anyway, just close the doors and do not waste the time of the comunity with those subpar models.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

Ready to merge

This branch is ready to get merged automatically.

· Sign up or log in to comment