Instructions to use witfoo/witq-1.0 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use witfoo/witq-1.0 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="witfoo/witq-1.0")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("witfoo/witq-1.0")
model = AutoModelForCausalLM.from_pretrained("witfoo/witq-1.0", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use witfoo/witq-1.0 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "witfoo/witq-1.0"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "witfoo/witq-1.0",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/witfoo/witq-1.0

SGLang

How to use witfoo/witq-1.0 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "witfoo/witq-1.0" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "witfoo/witq-1.0",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "witfoo/witq-1.0" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "witfoo/witq-1.0",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use witfoo/witq-1.0 with Docker Model Runner:
```
docker model run hf.co/witfoo/witq-1.0
```

f15hb0wn commited on Sep 14, 2024

Commit

5ae6a1a

verified ·

1 Parent(s): 473bd18

Update README.md

Browse files

Files changed (1) hide show

README.md +17 -11

README.md CHANGED Viewed

@@ -1,5 +1,5 @@
 ---
-license: apache-2.0
 datasets:
 - witfoo/syslog-to-artifact
 language:
@@ -9,15 +9,21 @@ base_model:
 - unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit
 pipeline_tag: text2text-generation
 widget:
-- text: "2 123456789010 eni-1235b8ca123456789 172.31.16.139 172.31.16.21 20641 22 6 20 4249 1418530010 1418530070 ACCEPT OK"
-  instruction: "Identify this syslog message"
-  example_title: "Syslog Identification"
-- text: "2 123456789010 eni-1235b8ca123456789 172.31.16.139 172.31.16.21 20641 22 6 20 4249 1418530010 1418530070 ACCEPT OK"
-  instruction: "Create a JSON artifact from the message"
-  example_title: "Transform Syslog"
-- text: "2 123456789010 eni-1235b8ca123456789 172.31.16.139 172.31.16.21 20641 22 6 20 4249 1418530010 1418530070 ACCEPT OK"
-  instruction: "Explain this syslog message"
-  example_title: "Explain Syslog in English"
 ---
 # WitQ 1.0
 A model built to assist cybersecurity researchers and investigators.
@@ -1053,4 +1059,4 @@ Finally, we put in place a set of resources including an [output reporting mecha
 The core values of Llama 3.1 are openness, inclusivity and helpfulness. It is meant to serve everyone, and to work for a wide range of use cases. It is thus designed to be accessible to people across many different backgrounds, experiences and perspectives. Llama 3.1 addresses users and their needs as they are, without insertion unnecessary judgment or normativity, while reflecting the understanding that even content that may appear problematic in some cases can serve valuable purposes in others. It respects the dignity and autonomy of all users, especially in terms of the values of free thought and expression that power innovation and progress.
-But Llama 3.1 is a new technology, and like any new technology, there are risks associated with its use. Testing conducted to date has not covered, nor could it cover, all scenarios. For these reasons, as with all LLMs, Llama 3.1’s potential outputs cannot be predicted in advance, and the model may in some instances produce inaccurate, biased or other objectionable responses to user prompts. Therefore, before deploying any applications of Llama 3.1 models, developers should perform safety testing and tuning tailored to their specific applications of the model. Please refer to available resources including our [Responsible Use Guide](https://llama.meta.com/responsible-use-guide), [Trust and Safety](https://llama.meta.com/trust-and-safety/) solutions, and other [resources](https://llama.meta.com/docs/get-started/) to learn more about responsible development.

 ---
+license: llama3.1
 datasets:
 - witfoo/syslog-to-artifact
 language:
 - unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit
 pipeline_tag: text2text-generation
 widget:
+- text: >-
+    2 123456789010 eni-1235b8ca123456789 172.31.16.139 172.31.16.21 20641 22 6
+    20 4249 1418530010 1418530070 ACCEPT OK
+  instruction: Identify this syslog message
+  example_title: Syslog Identification
+- text: >-
+    2 123456789010 eni-1235b8ca123456789 172.31.16.139 172.31.16.21 20641 22 6
+    20 4249 1418530010 1418530070 ACCEPT OK
+  instruction: Create a JSON artifact from the message
+  example_title: Transform Syslog
+- text: >-
+    2 123456789010 eni-1235b8ca123456789 172.31.16.139 172.31.16.21 20641 22 6
+    20 4249 1418530010 1418530070 ACCEPT OK
+  instruction: Explain this syslog message
+  example_title: Explain Syslog in English
 ---
 # WitQ 1.0
 A model built to assist cybersecurity researchers and investigators.
 The core values of Llama 3.1 are openness, inclusivity and helpfulness. It is meant to serve everyone, and to work for a wide range of use cases. It is thus designed to be accessible to people across many different backgrounds, experiences and perspectives. Llama 3.1 addresses users and their needs as they are, without insertion unnecessary judgment or normativity, while reflecting the understanding that even content that may appear problematic in some cases can serve valuable purposes in others. It respects the dignity and autonomy of all users, especially in terms of the values of free thought and expression that power innovation and progress.
+But Llama 3.1 is a new technology, and like any new technology, there are risks associated with its use. Testing conducted to date has not covered, nor could it cover, all scenarios. For these reasons, as with all LLMs, Llama 3.1’s potential outputs cannot be predicted in advance, and the model may in some instances produce inaccurate, biased or other objectionable responses to user prompts. Therefore, before deploying any applications of Llama 3.1 models, developers should perform safety testing and tuning tailored to their specific applications of the model. Please refer to available resources including our [Responsible Use Guide](https://llama.meta.com/responsible-use-guide), [Trust and Safety](https://llama.meta.com/trust-and-safety/) solutions, and other [resources](https://llama.meta.com/docs/get-started/) to learn more about responsible development.