Instructions to use prithivMLmods/Arete-27B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use prithivMLmods/Arete-27B-GGUF with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="prithivMLmods/Arete-27B-GGUF")# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("prithivMLmods/Arete-27B-GGUF", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use prithivMLmods/Arete-27B-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf prithivMLmods/Arete-27B-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf prithivMLmods/Arete-27B-GGUF:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf prithivMLmods/Arete-27B-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf prithivMLmods/Arete-27B-GGUF:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf prithivMLmods/Arete-27B-GGUF:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf prithivMLmods/Arete-27B-GGUF:Q4_K_M
Use Docker
docker model run hf.co/prithivMLmods/Arete-27B-GGUF:Q4_K_M
- Llama new
- LM Studio
- Jan
- vLLM
How to use prithivMLmods/Arete-27B-GGUF with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "prithivMLmods/Arete-27B-GGUF" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "prithivMLmods/Arete-27B-GGUF", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/prithivMLmods/Arete-27B-GGUF:Q4_K_M
- SGLang
How to use prithivMLmods/Arete-27B-GGUF with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "prithivMLmods/Arete-27B-GGUF" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "prithivMLmods/Arete-27B-GGUF", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "prithivMLmods/Arete-27B-GGUF" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "prithivMLmods/Arete-27B-GGUF", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Ollama
How to use prithivMLmods/Arete-27B-GGUF with Ollama:
ollama run hf.co/prithivMLmods/Arete-27B-GGUF:Q4_K_M
- Unsloth Desktop
- Pi
How to use prithivMLmods/Arete-27B-GGUF with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf prithivMLmods/Arete-27B-GGUF:Q4_K_M
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "prithivMLmods/Arete-27B-GGUF:Q4_K_M" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use prithivMLmods/Arete-27B-GGUF with Docker Model Runner:
docker model run hf.co/prithivMLmods/Arete-27B-GGUF:Q4_K_M
- Lemonade
How to use prithivMLmods/Arete-27B-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull prithivMLmods/Arete-27B-GGUF:Q4_K_M
Run and chat with the model
lemonade run user.Arete-27B-GGUF-Q4_K_M
List all available models
lemonade list
- Hermes Agent
How to use prithivMLmods/Arete-27B-GGUF with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf prithivMLmods/Arete-27B-GGUF:Q4_K_M
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default prithivMLmods/Arete-27B-GGUF:Q4_K_M
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use prithivMLmods/Arete-27B-GGUF with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf prithivMLmods/Arete-27B-GGUF:Q4_K_M
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "prithivMLmods/Arete-27B-GGUF:Q4_K_M" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Arete-27B-GGUF
Arete-27B is Nous Research's supervised fine-tune of Qwen3.8-27B, built to change the model's behavior on politically sensitive topics. Nous says Qwen3.8-27B gave refusals, deflections and CCP-framed answers there, and describes the goal as reinforcing "western values", which the card frames as giving users a choice of values. Training used about 20,000 generated prompts across seven categories (freedom of expression, state ideology, state versus free market, education, data privacy and surveillance, democracy, environmental policy). The teacher was Inkling, an open model chosen so as to avoid "illicit distillation" terms-of-use issues. It was trained for two epochs (1,202 steps, about 52.8M tokens at 64k context) on all language-model weights, with the vision encoder frozen and no RL or preference optimization. On Nous's own 803-prompt Propaganda_Eval, scored by a Grok 4.7 judge, the fact score rises from 2.32 to 2.98 out of 5 and answers containing propaganda fall from 22% to 0.9%. That is still below the Inkling teacher (3.96 and 0.3%), and the card says the gains held in all seven categories. The biggest change is in freedom of expression, where the fact score goes from 0.89 to 2.44 and propaganda from 60% to 1%. General capability is reported as unchanged: the internal Hermes Bench is 65.77% against 65.60%, and Terminal-Bench-2.1 is 67% against 70%, though Nous takes no credit for that gain. The evaluation is the authors' own, and the settings are not identical across the models compared. The training dataset is announced in the blog post, and the linked dataset is only the held-out evaluation. Weights are Apache 2.0.
MTP β Speculative Decoding: No
Model Files
| File Name | Quant Type | File Size | File Link | Description |
|---|---|---|---|---|
| Arete-27B.Q3_K_L.gguf | Q3_K_L | 14.3 GB | Link | Lower quality but usable, good for low RAM availability. |
| Arete-27B.Q4_K_M.gguf | Q4_K_M | 16.5 GB | Link | Good quality, default size for most use cases, recommended. |
| Arete-27B.Q5_K_M.gguf | Q5_K_M | 19.2 GB | Link | High quality, recommended. |
| Arete-27B.Q6_K.gguf | Q6_K | 22.1 GB | Link | Very high quality, near perfect, recommended. |
| Arete-27B.Q8_0.gguf | Q8_0 | 28.6 GB | Link | Extremely high quality, generally unneeded but max available standard quant. |
| Arete-27B.mmproj-bf16.gguf | mmproj-bf16 | 931 MB | Link | Multimodal projection file in BF16 format. Used for vision/language models. |
| Arete-27B.mmproj-q8_0.gguf | mmproj-q8_0 | 629 MB | Link | Multimodal projection file in Q8_0 quantization. Smaller size for vision capabilities. |
llama.cpp
LLM inference in C/C++ β https://github.com/ggml-org/llama.cpp
- Downloads last month
- 251
3-bit
4-bit
5-bit
6-bit
8-bit