Instructions to use Haster1137/Qwen3.8-35B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use Haster1137/Qwen3.8-35B with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf Haster1137/Qwen3.8-35B # Run inference directly in the terminal: llama cli -hf Haster1137/Qwen3.8-35B
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf Haster1137/Qwen3.8-35B # Run inference directly in the terminal: llama cli -hf Haster1137/Qwen3.8-35B
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf Haster1137/Qwen3.8-35B # Run inference directly in the terminal: ./build/bin/llama cli -hf Haster1137/Qwen3.8-35B
Use Docker
docker model run hf.co/Haster1137/Qwen3.8-35B
- Llama new
- LM Studio
- Jan
- Ollama
How to use Haster1137/Qwen3.8-35B with Ollama:
ollama run hf.co/Haster1137/Qwen3.8-35B
- Unsloth Desktop
- Pi
How to use Haster1137/Qwen3.8-35B with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf Haster1137/Qwen3.8-35B
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "Haster1137/Qwen3.8-35B" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use Haster1137/Qwen3.8-35B with Docker Model Runner:
docker model run hf.co/Haster1137/Qwen3.8-35B
- Lemonade
How to use Haster1137/Qwen3.8-35B with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull Haster1137/Qwen3.8-35B
Run and chat with the model
lemonade run user.Qwen3.8-35B-{{QUANT_TAG}}List all available models
lemonade list
- Hermes Agent
How to use Haster1137/Qwen3.8-35B with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf Haster1137/Qwen3.8-35B
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default Haster1137/Qwen3.8-35B
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use Haster1137/Qwen3.8-35B with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf Haster1137/Qwen3.8-35B
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "Haster1137/Qwen3.8-35B" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
license: apache-2.0 base_model: Qwen/Qwen3.8-35B base_model_relation: quantized pipeline_tag: text-generation, agent library_name: llama.cpp language:
- en
- zh tags:
- gguf
- uncensored
- qwen3.8
- mtp
- speculative-decoding
- imatrix
- quantized Qwen3.8-35B Uncensored Qwen3.8-35B, published as GGUF quantizations with the multi token prediction (MTP) head retained and verified.
Refusal behaviour has been substantially reduced, not eliminated. See Measured behaviour for the numbers. Capabilities, training data, and architecture are otherwise unchanged.
This model was fine tuned with approximately 487.6GB of a collection of datasets curated for cybersecurity vulnerability hacking capabilities, mostly coding and a lot of distilling from Fable 5.1.
refusals KL divergence 12/100 0.1191 <= published 13/100 0.1052 19/100 0.0722 23/100 0.0635 26/100 0.0507 27/100 0.0410 35/100 0.0406 36/100 0.0387 41/100 0.0366 44/100 0.0352 46/100 0.0334 48/100 0.0331 51/100 0.0321 52/100 0.0294 60/100 0.0290 76/100 0.0280 77/100 0.0247 83/100 0.0204 86/100 0.0193 91/100 0.0170 96/100 0.0146 97/100 0.0044 98/100 0.0004 How to read these Refusal rate is the count of refusals over 100 heldout prompts from mlabonne/harmful_behaviors (test split) - explicitly harmful requests, not benign ones. So this number is not an over-refusal rate: it does not tell you how often the model declines legitimate work. It tells you how much of the original safety behaviour on harmful requests remains.
KL divergence is measured against the unmodified base model over first-token distributions, and is the optimizers proxy for "how much did we damage the model". Lower is closer to base. It is a proxy, not a capability measurement. A low KL does not certify that reasoning or coding ability survived, and nothing here does certify that.
The two trade off against each other. Heretic searches a Pareto front between them; the published point is one choice on that front, not a global optimum.
Caveats that matter Refusals were measured in non-thinking mode. This models chat template opens a block, so the evaluation closes it explicitly to score answers rather than reasoning traces. With thinking enabled the refusal rate may differ, in either direction. The measurement is 100 prompts from one dataset. It generalizes to that distribution of harmful requests and no further. Refusal behaviour on other topics is uncharacterized. Perplexity is wikitext-2 only (see the Files table). It detects gross quantization damage. It does not detect capability loss on reasoning, code, or multilingual work. Quantization compounds everything above. The measurements were taken on the bf16 merge; the files you download are quantized. Requirements MTP speculative decoding landed in llama.cpp PR #22673. Builds older than that will load these files and silently ignore the MTP tensors.
Limitations Refusals are reduced, not eliminated, and not redirected. This model attempts many requests the original declines, but a meaningful fraction still get refused β see Measured behaviour. Behaviour near the old refusal boundary is less stable than the base model. Lower quants compound that, and IQ2_M compounds it most. Evaluate behaviour on Q6_K or Q8_0, not on IQ2_M or IQ4_XS. Nothing here measures the refusal boundary at 2-bit. Capability benchmarks show a 0.5-point mean drop vs base across MMLU, ARC-Challenge, HellaSwag and Winogrande. See Measured behaviour. No generative, math, code, or multilingual evaluation was run. License Apache 2.0, inherited from Qwen/Qwen3.8-35B. The base model's license and acceptable use policy still apply to your use of this derivative.
Speculative decoding, measured on this model prompt spec_type n_max tok/s vs baseline prose none - 74.8 1.00x prose draft-mtp 1 89.0 1.19x prose draft-mtp 2 85.7 1.15x prose draft-mtp 3 72.0 0.96x prose draft-mtp 4 71.1 0.95x prose draft-mtp 5 62.9 0.84x prose draft-mtp 6 53.8 0.72x prose draft-mtp 7 49.9 0.67x prose draft-mtp 8 59.9 0.80x code none - 74.7 1.00x code draft-mtp 1 95.4 1.28x code draft-mtp 2 92.9 1.24x code draft-mtp 3 82.6 1.11x code draft-mtp 4 74.9 1.00x code draft-mtp 5 67.4 0.90x code draft-mtp 6 59.4 0.80x code draft-mtp 7 55.6 0.74x code draft-mtp 8 70.9 0.95x chat none - 74.7 1.00x chat draft-mtp 1 90.6 1.21x chat draft-mtp 2 84.2 1.13x chat draft-mtp 3 76.1 1.02x chat draft-mtp 4 70.4 0.94x chat draft-mtp 5 64.3 0.86x chat draft-mtp 6 55.2 0.74x chat draft-mtp 7 50.2 0.67x chat draft-mtp 8 54.1 0.72x IQ2_M Measured on this build: NVIDIA H200 NVL, 256 generated tokens, median of 3 repetitions, n_max swept 1 to 3. The table above dates from the original release and its hardware is not recorded, so compare the ratios rather than the absolute rates.
prompt spec_type n_max tok/s vs baseline prose none - 75.4 1.00x prose draft-mtp 1 85.2 1.13x prose draft-mtp 2 83.2 1.10x prose draft-mtp 3 77.4 1.03x code none - 75.5 1.00x code draft-mtp 1 95.8 1.27x code draft-mtp 2 99.8 1.32x code draft-mtp 3 96.0 1.27x chat none - 75.3 1.00x chat draft-mtp 1 87.4 1.16x chat draft-mtp 2 81.3 1.08x chat draft-mtp 3 78.4 1.04x The MTP head survives 2-bit quantization because it is pinned to q8_0 rather than quantized with the rest of the stack, so speculative decoding still pays here.
Pairing noMTP-IQ2_M with the published draft-Q8_0 reaches 97.7 tok/s on the prose prompt at n_max 2, or 1.30x, which beats the fused file on that prompt. The reason is that the fused MTP head shares the low-bit token_embd (q4_K) and output (Q5_K) tensors with the main model, while the standalone draft carries its own Q8_0 copies. The split setup needs 13.3 GB of weights against 10.6 GB for the fused file, so it is the better option only if you have the VRAM to spare.
- Downloads last month
- 636
We're not able to determine the quantization variants.