Instructions to use Thox-ai/ThoxMini-3B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Thox-ai/ThoxMini-3B with PEFT:

from peft import PeftModel
from transformers import AutoModelForCausalLM

base_model = AutoModelForCausalLM.from_pretrained("unsloth/Llama-3.2-3B-Instruct")
model = PeftModel.from_pretrained(base_model, "Thox-ai/ThoxMini-3B")

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use Thox-ai/ThoxMini-3B with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Thox-ai/ThoxMini-3B:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf Thox-ai/ThoxMini-3B:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Thox-ai/ThoxMini-3B:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf Thox-ai/ThoxMini-3B:Q4_K_M

Build from source code

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf Thox-ai/ThoxMini-3B:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama cli -hf Thox-ai/ThoxMini-3B:Q4_K_M

Use Docker

docker model run hf.co/Thox-ai/ThoxMini-3B:Q4_K_M

Llama new
LM Studio
Jan

vLLM

How to use Thox-ai/ThoxMini-3B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Thox-ai/ThoxMini-3B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Thox-ai/ThoxMini-3B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Thox-ai/ThoxMini-3B:Q4_K_M

Ollama
How to use Thox-ai/ThoxMini-3B with Ollama:
```
ollama run hf.co/Thox-ai/ThoxMini-3B:Q4_K_M
```
Unsloth Desktop
Docker Model Runner
How to use Thox-ai/ThoxMini-3B with Docker Model Runner:
```
docker model run hf.co/Thox-ai/ThoxMini-3B:Q4_K_M
```

Lemonade

How to use Thox-ai/ThoxMini-3B with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull Thox-ai/ThoxMini-3B:Q4_K_M

Run and chat with the model

lemonade run user.ThoxMini-3B-Q4_K_M

List all available models

lemonade list

Atomic Chat

ThoxMini-3B — ThoxMini Assistant (LoRA Adapter + GGUF)

Summary

ThoxMini-3B is a THOX assistant model for the ThoxMini tier: a LoRA fine-tune (rank 16) on a THOX-authored corpus of 1,565 instruction examples, published as an adapter with a bundled Q4_K_M GGUF for direct use. Access is public and ungated (ungated 2026-10-31 on the CTO's standing directive: no gating unless licensing requires it). The Llama 3.2 license travels with this model and its terms govern use.

Your AI. Your Data. Your Rules.™

Files

File	Size	Purpose
`thoxmini-3b-Q4_K_M.gguf`	2,019,373,952 B	GGUF model build
`adapter_model.safetensors`	97,307,544 B	LoRA adapter weights
`tokenizer.json`	17,209,920 B	Tokenizer files
`tokenizer_config.json`	50,657 B	Tokenizer files
`.gitattributes`	1,630 B	Repository file
`adapter_config.json`	1,463 B	JSON metadata
`GATED_ACCESS.md`	1,262 B	Documentation
`chat_template.jinja`	551 B	Chat template

Run it

Bundled GGUF (llama.cpp / Ollama)

ollama pull Thox-ai/thoxmini-3b
ollama run Thox-ai/thoxmini-3b

hf download Thox-ai/ThoxMini-3B thoxmini-3b-Q4_K_M.gguf --local-dir ./ThoxMini-3B

This repo is public and ungated: download directly; hf auth login is optional for authenticated pulls.

Adapter (Transformers + PEFT)

Load the base model named in this card's metadata, then attach the adapter from this repo:

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base = "<base model named in this card's metadata>"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, device_map="auto")
model = PeftModel.from_pretrained(model, "Thox-ai/ThoxMini-3B")

Intended use

On-device and edge text generation within the THOX stack, for ThoxMini-class hardware.
Out of scope: medical, legal, financial, safety-critical, or other high-stakes decisions without human review; unattended control of physical devices without a confirmation step; any use prohibited by the license or its acceptable use policy.

Training data and procedure

THOX fine-tune of meta-llama/Llama-3.2-3B-Instruct. The corpus and procedure are described in this card as published at release (see Summary and Evaluation); no training beyond what this card describes is claimed for this repository.

Evaluation

No THOX GATE49 result is published for this model.

Limitations

Small and quantized models make factual mistakes and can produce confident but wrong output; verify important answers.
Behaviour and bias reflect the training data, which is mostly English; quality in other languages is lower.
Quantization trades accuracy for size.
This build has not been qualified on ThoxMini device hardware.

Provenance

Published by THOX.ai as Thox-ai/ThoxMini-3B. Upstream base: meta-llama/Llama-3.2-3B-Instruct. This card was conformed to THOX Model Card Standard v1.0 on 2026-10-31 (frontmatter, tags, header, files, and section structure); the technical content is as published at release.

License

This model is licensed under the Llama 3.2 Community License, Copyright © Meta Platforms, Inc. All Rights Reserved. Built with Llama. Use must comply with the license's Acceptable Use Policy. The THOX fine-tune and additions are covered under the same terms.

THOX.ai — Your AI. Your Data. Your Rules.™

Downloads last month: 27

GGUF

Model size

3B params

Architecture

llama

Hardware compatibility

4-bit

Model tree for Thox-ai/ThoxMini-3B

Base model

meta-llama/Llama-3.2-3B-Instruct

Adapter

(877)

this model

Thox-ai
/

ThoxMini-3B