Instructions to use Nanbeige/Nanbeige4.2-3B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Nanbeige/Nanbeige4.2-3B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Nanbeige/Nanbeige4.2-3B", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("Nanbeige/Nanbeige4.2-3B", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Nanbeige/Nanbeige4.2-3B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Nanbeige/Nanbeige4.2-3B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Nanbeige/Nanbeige4.2-3B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Nanbeige/Nanbeige4.2-3B

SGLang

How to use Nanbeige/Nanbeige4.2-3B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Nanbeige/Nanbeige4.2-3B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Nanbeige/Nanbeige4.2-3B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Nanbeige/Nanbeige4.2-3B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Nanbeige/Nanbeige4.2-3B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Nanbeige/Nanbeige4.2-3B with Docker Model Runner:
```
docker model run hf.co/Nanbeige/Nanbeige4.2-3B
```

Some questions about training

#14

by mobinx - opened 1 day ago

Discussion

mobinx

1 day ago

I am working on loop transformer and I have some questions about your architecture. You provided two loop pass as you included on report.
1.Does pass 1 perform coarse reasoning and pass 2 refine it?
².Do you inject a learned loop embedding?
3.Are there different LayerNorm parameters per loop?
4.Is there any regularization encouraging different behavior between passes?
5. Are you using any new training way for updating loop gradient?
6. Does there any loop specific gradient loss update or global gradient update?
Again the work is marvelous. It's give me hope and energy to continue work on my loop transformer optimization algorithm. It will really really help me you answer those questions. Please I humblely request you for the answers.

leran1995

Nanbeige LLM Lab org about 21 hours ago

Thanks a lot for the thoughtful questions and for your interest in our work.

For Nanbeige4.2, the two passes reuse the same Transformer stack. We do not explicitly constrain the first pass to perform coarse reasoning or the second pass to refine it. Based on the training signals we monitor after the first pass, our current observation is that it behaves more like a stage of high-level representation formation than coarse reasoning.

For Nanbeige4.5, which is currently being trained and will be released later, we have further improved this design with LoopSplit, where only part of the layer stack is looped while the remaining layers are executed once. The implementation is already available in our modeling code (https://huggingface.co/Nanbeige/Nanbeige4.2-3B/blob/main/modeling_nanbeige.py).

Your other questions are closely related to directions we are still actively investigating. We do not yet have firm conclusions on all of them, but we plan to share more findings in future model releases.

We look forward to further discussion and exchange in this direction.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment