Instructions to use facebook/opt-13b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use facebook/opt-13b with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="facebook/opt-13b")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("facebook/opt-13b") model = AutoModelForCausalLM.from_pretrained("facebook/opt-13b", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use facebook/opt-13b with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "facebook/opt-13b" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "facebook/opt-13b", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/facebook/opt-13b
- SGLang
How to use facebook/opt-13b with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "facebook/opt-13b" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "facebook/opt-13b", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "facebook/opt-13b" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "facebook/opt-13b", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use facebook/opt-13b with Docker Model Runner:
docker model run hf.co/facebook/opt-13b
Commit History
-
Add evaluation results on the sen_en config and test split of futin/feed
Beep boop, I am a bot from Hugging Face's automatic model evaluator 👋!\ Your model has been evaluated on the sen_en config and test split of the [futin/feed](https://huggingface.co/datasets/futin/feed) dataset by @futin, using the predictions stored [here](https://huggingface.co/datasets/autoevaluate/autoeval-eval-futin__feed-sen_en-2f01d7-2175769987).\ Accept this pull request to see the results displayed on the [Hub leaderboard](https://huggingface.co/spaces/autoevaluate/leaderboards?dataset=futin/feed).\ Evaluate your model on more datasets [here](https://huggingface.co/spaces/autoevaluate/model-evaluator?dataset=futin/feed).fd8b157
-
Fix scripts (#6)
- Fix scripts (3ef0f9aabff6718375e8b98c560bfe469d1362cc) Co-authored-by: Manuel Romero <mrm8488@users.noreply.huggingface.co>d4fe85a
-
Update README.md
Update generations after major fix: https://github.com/huggingface/transformers/commit/abc400b06a8ab26cd438b6e9add3aad082ffc48fe3ba57e
-
Update config.json
2445748 -
correct checkpoints see: https://github.com/facebookresearch/metaseq/pull/164
2e08484
-
Add flax_weights
3e1c328 -
Add tf_weights
9c10d01
-
Change BOS token from 0 to 2 as BOS token is equal to EOS for OPT. See: https://github.com/huggingface/transformers/issues/17431 (#1)
- Change BOS token from 0 to 2 as BOS token is equal to EOS for OPT. See: https://github.com/huggingface/transformers/issues/17431 (c665816be91a53444e29b317487d74732168c71d)e4e74e8
-
Add license
e55d5e7 -
Set commercial attribute to False
90715ce -
Update README
501b4dd
-
Temporarily deactivate the inference API
795807d -
Update config.json
a59887a
-
Update README.md
b445847 -
Update README.md
added - text-generation tagb002a96 -
Update README.md
5dcfc51 -
Update README.md
65cf736 -
Create README.md
3c69d24 -
finish
d9874ea -
Update config.json
9b6c00a
-
upload model
aa3a158 -
initial commit
edd0750