Instructions to use seeweb/SeewebLLM-it with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use seeweb/SeewebLLM-it with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="seeweb/SeewebLLM-it")

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("seeweb/SeewebLLM-it")
model = AutoModelForCausalLM.from_pretrained("seeweb/SeewebLLM-it", device_map="auto")

Inference
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use seeweb/SeewebLLM-it with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "seeweb/SeewebLLM-it"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "seeweb/SeewebLLM-it",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/seeweb/SeewebLLM-it

SGLang

How to use seeweb/SeewebLLM-it with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "seeweb/SeewebLLM-it" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "seeweb/SeewebLLM-it",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "seeweb/SeewebLLM-it" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "seeweb/SeewebLLM-it",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use seeweb/SeewebLLM-it with Docker Model Runner:
```
docker model run hf.co/seeweb/SeewebLLM-it
```

itsrocchi commited on Aug 21, 2023

Commit

01fc5b7

1 Parent(s): a6d1714

Update README.md

Browse files

Files changed (1) hide show

README.md +6 -2

README.md CHANGED Viewed

@@ -97,10 +97,14 @@ It's important to note that AI is not a single entity, but rather a rapidly evol
 #### Conclusion
-LLama 2 can understand the question and gives the user a very specific and overall a better answer compared to the one given by the fine-tuned model. However the fine-tuned model answers with a sentence written in a perfect italian.
 ### Training Data and Details
 The dataset used is [seeweb/Seeweb-it-292-forLLM](https://huggingface.co/datasets/seeweb/Seeweb-it-292-forLLM), a dataset containing approx. 300 italian prompt-answer conversations.
-The training has been made on RTX A6000, inside [Seeweb's Cloud Server GPU](https://www.seeweb.it/prodotti/cloud-server-gpu)

 #### Conclusion
+LLama 2 can understand the question and gives the user a very specific and overall a better answer compared to the one given by the fine-tuned model. However the fine-tuned model answers with a sentence written in a perfect italian, which is what we were trying to achieve with this fine-tuning process.
 ### Training Data and Details
 The dataset used is [seeweb/Seeweb-it-292-forLLM](https://huggingface.co/datasets/seeweb/Seeweb-it-292-forLLM), a dataset containing approx. 300 italian prompt-answer conversations.
+The training has been made on RTX A6000, inside [Seeweb's Cloud Server GPU](https://www.seeweb.it/prodotti/cloud-server-gpu)
+### What next?
+The model must be improved: a much bigger dataset needs to be created so that the model can learn many more ways to answer.