Instructions to use optimum/gpt2-neuronx with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use optimum/gpt2-neuronx with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="optimum/gpt2-neuronx")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("optimum/gpt2-neuronx") model = AutoModelForCausalLM.from_pretrained("optimum/gpt2-neuronx", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use optimum/gpt2-neuronx with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "optimum/gpt2-neuronx" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "optimum/gpt2-neuronx", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/optimum/gpt2-neuronx
- SGLang
How to use optimum/gpt2-neuronx with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "optimum/gpt2-neuronx" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "optimum/gpt2-neuronx", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "optimum/gpt2-neuronx" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "optimum/gpt2-neuronx", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use optimum/gpt2-neuronx with Docker Model Runner:
docker model run hf.co/optimum/gpt2-neuronx
Commit History
-
Upload checkpoint/generation_config.json with huggingface_hub
eef8462 -
Upload checkpoint/config.json with huggingface_hub
b3de078
-
Upload checkpoint/pytorch_model.bin/p141.transformer.h.11.ln_2.bias with huggingface_hub
b243856 -
Upload checkpoint/pytorch_model.bin/p126.transformer.h.10.attn.c_proj.weight with huggingface_hub
078a913 -
Upload checkpoint/pytorch_model.bin/p86.transformer.h.7.ln_1.weight with huggingface_hub
a184c38 -
Upload checkpoint/pytorch_model.bin/p34.transformer.h.2.mlp.c_fc.weight with huggingface_hub
03286a1 -
Upload checkpoint/pytorch_model.bin/p1.transformer.wpe.weight with huggingface_hub
c168124 -
Upload checkpoint/pytorch_model.bin/p95.transformer.h.7.mlp.c_fc.bias with huggingface_hub
767fd22 -
Upload checkpoint/pytorch_model.bin/p7.transformer.h.0.attn.c_proj.bias with huggingface_hub
91450a3 -
Upload checkpoint/pytorch_model.bin/p99.transformer.h.8.ln_1.bias with huggingface_hub
90e997f -
Upload checkpoint/pytorch_model.bin/p23.transformer.h.1.mlp.c_fc.bias with huggingface_hub
bc5ae42 -
Upload checkpoint/pytorch_model.bin/p30.transformer.h.2.attn.c_proj.weight with huggingface_hub
9a1546a -
Upload checkpoint/pytorch_model.bin/p2.transformer.h.0.ln_1.weight with huggingface_hub
3f08e2b -
Upload checkpoint/pytorch_model.bin/p119.transformer.h.9.mlp.c_fc.bias with huggingface_hub
f810c26 -
Upload checkpoint/pytorch_model.bin/p21.transformer.h.1.ln_2.bias with huggingface_hub
f897ff4 -
Upload checkpoint/pytorch_model.bin/p53.transformer.h.4.attn.c_attn.bias with huggingface_hub
447004e -
Upload checkpoint/pytorch_model.bin/p138.transformer.h.11.attn.c_proj.weight with huggingface_hub
a38e2e4 -
Upload checkpoint/pytorch_model.bin/p24.transformer.h.1.mlp.c_proj.weight with huggingface_hub
60abf74 -
Upload checkpoint/pytorch_model.bin/p117.transformer.h.9.ln_2.bias with huggingface_hub
731f281 -
Upload checkpoint/pytorch_model.bin/p146.transformer.ln_f.weight with huggingface_hub
bec7fef -
Upload checkpoint/pytorch_model.bin/p145.transformer.h.11.mlp.c_proj.bias with huggingface_hub
b7ac993 -
Upload checkpoint/pytorch_model.bin/p66.transformer.h.5.attn.c_proj.weight with huggingface_hub
3491019 -
Upload checkpoint/pytorch_model.bin/p41.transformer.h.3.attn.c_attn.bias with huggingface_hub
5744086 -
Upload checkpoint/pytorch_model.bin/p44.transformer.h.3.ln_2.weight with huggingface_hub
6e77776 -
Upload checkpoint/pytorch_model.bin/p69.transformer.h.5.ln_2.bias with huggingface_hub
e7f99ec -
Upload checkpoint/pytorch_model.bin/p140.transformer.h.11.ln_2.weight with huggingface_hub
84d6948 -
Upload checkpoint/pytorch_model.bin/p104.transformer.h.8.ln_2.weight with huggingface_hub
c0e0369 -
Upload checkpoint/pytorch_model.bin/p35.transformer.h.2.mlp.c_fc.bias with huggingface_hub
4632083 -
Upload checkpoint/pytorch_model.bin/p97.transformer.h.7.mlp.c_proj.bias with huggingface_hub
1cc05ee -
Upload checkpoint/pytorch_model.bin/p148.lm_head.weight with huggingface_hub
33fafa8 -
Upload checkpoint/pytorch_model.bin/p64.transformer.h.5.attn.c_attn.weight with huggingface_hub
4493c2b -
Upload checkpoint/pytorch_model.bin/p25.transformer.h.1.mlp.c_proj.bias with huggingface_hub
f9dac97 -
Upload checkpoint/pytorch_model.bin/p51.transformer.h.4.ln_1.bias with huggingface_hub
6ee3135 -
Upload checkpoint/pytorch_model.bin/p110.transformer.h.9.ln_1.weight with huggingface_hub
0bb1ec0 -
Upload checkpoint/pytorch_model.bin/p105.transformer.h.8.ln_2.bias with huggingface_hub
3f6567b -
Upload checkpoint/pytorch_model.bin/p127.transformer.h.10.attn.c_proj.bias with huggingface_hub
536bc9f -
Upload checkpoint/pytorch_model.bin/p17.transformer.h.1.attn.c_attn.bias with huggingface_hub
8bc47e3 -
Upload checkpoint/pytorch_model.bin/p76.transformer.h.6.attn.c_attn.weight with huggingface_hub
a5b19d3 -
Upload checkpoint/pytorch_model.bin/p137.transformer.h.11.attn.c_attn.bias with huggingface_hub
b31e485 -
Upload checkpoint/pytorch_model.bin/p132.transformer.h.10.mlp.c_proj.weight with huggingface_hub
c7b08a6 -
Upload checkpoint/pytorch_model.bin/p73.transformer.h.5.mlp.c_proj.bias with huggingface_hub
e412970 -
Upload checkpoint/pytorch_model.bin/p32.transformer.h.2.ln_2.weight with huggingface_hub
045677c -
Upload checkpoint/pytorch_model.bin/p70.transformer.h.5.mlp.c_fc.weight with huggingface_hub
14932a0 -
Upload checkpoint/pytorch_model.bin/p52.transformer.h.4.attn.c_attn.weight with huggingface_hub
02d5d38 -
Upload checkpoint/pytorch_model.bin/p63.transformer.h.5.ln_1.bias with huggingface_hub
89f8691 -
Upload checkpoint/pytorch_model.bin/p109.transformer.h.8.mlp.c_proj.bias with huggingface_hub
c5bc2a2 -
Upload checkpoint/pytorch_model.bin/p131.transformer.h.10.mlp.c_fc.bias with huggingface_hub
0e02f45 -
Upload checkpoint/pytorch_model.bin/p3.transformer.h.0.ln_1.bias with huggingface_hub
ba64f6d -
Upload checkpoint/pytorch_model.bin/p115.transformer.h.9.attn.c_proj.bias with huggingface_hub
c43ee6e -
Upload checkpoint/pytorch_model.bin/p50.transformer.h.4.ln_1.weight with huggingface_hub
3beb721