Text Generation
NeMo
ONNX
Safetensors
GGUF
English
Persian
local-llm
quantized
rtx-3060
consumer-hardware
code
agent
agents
rag
embeddings
reranker
text-to-speech
speech-to-text
multimodal
vision
voice-cloning
persian
farsi
uncensored
abliterated
imatrix
conversational
Instructions to use mohammadjafari-dev/local-ai-toolkit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- NeMo
How to use mohammadjafari-dev/local-ai-toolkit with NeMo:
# tag did not correspond to a valid NeMo domain.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use mohammadjafari-dev/local-ai-toolkit with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf mohammadjafari-dev/local-ai-toolkit:UD-Q4_K_XL # Run inference directly in the terminal: llama cli -hf mohammadjafari-dev/local-ai-toolkit:UD-Q4_K_XL
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf mohammadjafari-dev/local-ai-toolkit:UD-Q4_K_XL # Run inference directly in the terminal: llama cli -hf mohammadjafari-dev/local-ai-toolkit:UD-Q4_K_XL
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf mohammadjafari-dev/local-ai-toolkit:UD-Q4_K_XL # Run inference directly in the terminal: ./build/bin/llama cli -hf mohammadjafari-dev/local-ai-toolkit:UD-Q4_K_XL
Use Docker
docker model run hf.co/mohammadjafari-dev/local-ai-toolkit:UD-Q4_K_XL
- Llama new
- LM Studio
- Jan
- vLLM
How to use mohammadjafari-dev/local-ai-toolkit with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "mohammadjafari-dev/local-ai-toolkit" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "mohammadjafari-dev/local-ai-toolkit", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/mohammadjafari-dev/local-ai-toolkit:UD-Q4_K_XL
- Ollama
How to use mohammadjafari-dev/local-ai-toolkit with Ollama:
ollama run hf.co/mohammadjafari-dev/local-ai-toolkit:UD-Q4_K_XL
- Unsloth Desktop
- Pi
How to use mohammadjafari-dev/local-ai-toolkit with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf mohammadjafari-dev/local-ai-toolkit:UD-Q4_K_XL
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "mohammadjafari-dev/local-ai-toolkit:UD-Q4_K_XL" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use mohammadjafari-dev/local-ai-toolkit with Docker Model Runner:
docker model run hf.co/mohammadjafari-dev/local-ai-toolkit:UD-Q4_K_XL
- Lemonade
How to use mohammadjafari-dev/local-ai-toolkit with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull mohammadjafari-dev/local-ai-toolkit:UD-Q4_K_XL
Run and chat with the model
lemonade run user.local-ai-toolkit-UD-Q4_K_XL
List all available models
lemonade list
- Hermes Agent
How to use mohammadjafari-dev/local-ai-toolkit with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf mohammadjafari-dev/local-ai-toolkit:UD-Q4_K_XL
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default mohammadjafari-dev/local-ai-toolkit:UD-Q4_K_XL
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use mohammadjafari-dev/local-ai-toolkit with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf mohammadjafari-dev/local-ai-toolkit:UD-Q4_K_XL
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "mohammadjafari-dev/local-ai-toolkit:UD-Q4_K_XL" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
mohammad commited on
Upload README.md
Browse files
README.md
CHANGED
|
@@ -62,7 +62,7 @@ pipeline_tag: text-generation
|
|
| 62 |
<img src="https://img.shields.io/badge/🎬 Video_Gen-🔜-yellow" alt="Video Gen">
|
| 63 |
</p>
|
| 64 |
|
| 65 |
-
<i>
|
| 66 |
|
| 67 |
</div>
|
| 68 |
|
|
@@ -70,9 +70,13 @@ pipeline_tag: text-generation
|
|
| 70 |
|
| 71 |
## 📖 About This Toolkit
|
| 72 |
|
| 73 |
-
This repository is a **
|
| 74 |
|
| 75 |
-
The selection process was **
|
|
|
|
|
|
|
|
|
|
|
|
|
| 76 |
|
| 77 |
### 🎯 Design Philosophy
|
| 78 |
|
|
@@ -80,8 +84,8 @@ Each category follows a dual-tier architecture:
|
|
| 80 |
|
| 81 |
| Tier | Purpose | What It Means |
|
| 82 |
|------|---------|---------------|
|
| 83 |
-
| 🏆 **Quality** |
|
| 84 |
-
| ⚡ **Speed** |
|
| 85 |
|
| 86 |
> **Note on Coding models:** The Coding Agents category includes two quality-tier models (Architect & Executor + Detective & Debugger). The "speed" variants in that category, while labeled for speed, are still powerful enough to serve as competent rapid-response code assistants — they simply trade some reasoning depth for significantly faster token generation.
|
| 87 |
|
|
@@ -201,7 +205,7 @@ The "brains" of your agent network. These models decide which worker to call, wh
|
|
| 201 |
|-------|------|-------------|------|
|
| 202 |
| [Agents-A1 35B MoE](https://huggingface.co/Chungulus/Agents-A1-IQ4_XS-imatrix-gguf-fable5-calibrated) | 35B (MoE) | IQ4_XS | **CEO Agent** — high-level autonomous planning and task decomposition. Calibrated with imatrix for superior quantization fidelity |
|
| 203 |
| [GLM-4.7-Flash](https://huggingface.co/unsloth/GLM-4.7-Flash-GGUF) | Flash | Q4_K_XL | **Router** — ultra-fast model for routing decisions, intent classification, and dispatching tasks to appropriate workers |
|
| 204 |
-
| [Hermes 4.3 36B](https://huggingface.co/NousResearch/Hermes-4.3-36B-GGUF) | 36B | Q4_K_M | **Quality Orchestrator** —
|
| 205 |
|
| 206 |
### ⚙️ Agentic Workers
|
| 207 |
|
|
@@ -272,7 +276,7 @@ RAG (Retrieval-Augmented Generation) is how you give your AI **access to your ow
|
|
| 272 |
- **Sparse retrieval** — BM25-style keyword matching (great for exact term matches)
|
| 273 |
- **ColBERT retrieval** — late interaction for fine-grained token-level relevance scoring
|
| 274 |
|
| 275 |
-
This multi-modal retrieval approach means you
|
| 276 |
|
| 277 |
### 🔄 2. Reranker — [BGE-Reranker-V2-M3](https://huggingface.co/Heisenberg0314/bge-reranker-v2-m3)
|
| 278 |
|
|
@@ -376,7 +380,7 @@ A complete speech understanding pipeline that starts with audio preprocessing (c
|
|
| 376 |
| 🎙️ Vocal Isolation | [Demucs V4 (ONNX)](https://huggingface.co/MrCitron/demucs-v4-onnx) | Studio-quality vocal separation — removes background music, noise, and other speakers. Essential for noisy recordings |
|
| 377 |
| 🔇 Silence/Noise Cutter | [Silero VAD (ONNX)](https://huggingface.co/istupakov/silero-vad-onnx) | Voice Activity Detection — automatically cuts silence and non-speech segments, reducing processing time and improving transcription accuracy |
|
| 378 |
|
| 379 |
-
### 🏆 Quality Tier —
|
| 380 |
|
| 381 |
| Scope | Model | Size | Highlights |
|
| 382 |
|-------|-------|------|------------|
|
|
@@ -503,9 +507,6 @@ These are **abliterated** (uncensored) versions of the models found in other cat
|
|
| 503 |
|
| 504 |
All models in this toolkit are sourced from Hugging Face. During the initial download process, direct URLs were not recorded. The original sources were **recovered by matching SHA-256 hashes** against Hugging Face repositories. While some links may point to slightly different file versions within the same quantization family, the hash-verified content is identical to what is hosted in this toolkit.
|
| 505 |
|
| 506 |
-
<details>
|
| 507 |
-
<summary>🎯 Click to expand full source URL table</summary>
|
| 508 |
-
|
| 509 |
### 1 — Universal Foundation LLMs
|
| 510 |
|
| 511 |
| File | Source |
|
|
@@ -592,8 +593,6 @@ All models in this toolkit are sourced from Hugging Face. During the initial dow
|
|
| 592 |
| Huihui-Qwen3-VL-8B-Instruct-abliterated.Q8_0.gguf | [mradermacher/Huihui-Qwen3-VL-8B-Instruct-abliterated-GGUF](https://huggingface.co/mradermacher/Huihui-Qwen3-VL-8B-Instruct-abliterated-GGUF/blob/main/Huihui-Qwen3-VL-8B-Instruct-abliterated.Q8_0.gguf) |
|
| 593 |
| Huihui-Qwen3-VL-8B-Instruct-abliterated.mmproj-f16.gguf | [mradermacher/Huihui-Qwen3-VL-8B-Instruct-abliterated-GGUF](https://huggingface.co/mradermacher/Huihui-Qwen3-VL-8B-Instruct-abliterated-GGUF/blob/main/Huihui-Qwen3-VL-8B-Instruct-abliterated.mmproj-f16.gguf) |
|
| 594 |
|
| 595 |
-
</details>
|
| 596 |
-
|
| 597 |
---
|
| 598 |
|
| 599 |
## 🛠️ Recommended Tools for Running These Models
|
|
@@ -635,6 +634,6 @@ All models in this toolkit are the work of their respective creators and are use
|
|
| 635 |
|
| 636 |
---
|
| 637 |
|
| 638 |
-
<i>
|
| 639 |
|
| 640 |
</div>
|
|
|
|
| 62 |
<img src="https://img.shields.io/badge/🎬 Video_Gen-🔜-yellow" alt="Video Gen">
|
| 63 |
</p>
|
| 64 |
|
| 65 |
+
<i>AI-assisted model selection · Quantized for consumer hardware · A suggested starting point</i>
|
| 66 |
|
| 67 |
</div>
|
| 68 |
|
|
|
|
| 70 |
|
| 71 |
## 📖 About This Toolkit
|
| 72 |
|
| 73 |
+
This repository is a **suggested toolkit** of open-source AI models assembled to run entirely locally on consumer-grade hardware — specifically an **NVIDIA RTX 3060 (12GB VRAM) with 32GB system RAM**. Each model was chosen to fit within these constraints while offering strong output quality for the VRAM budget.
|
| 74 |
|
| 75 |
+
The selection process was **AI-assisted** — I consulted several AI assistants to weigh benchmarks, quantization trade-offs, and model capabilities, then settled on the common recommendations you see here. This is **not a definitively optimal combination**: better models may already exist, and stronger ones are released regularly. Treat it as a well-reasoned starting point rather than a final answer — your own testing on your own hardware is the real benchmark.
|
| 76 |
+
|
| 77 |
+
### 🔄 Updates & Longevity
|
| 78 |
+
|
| 79 |
+
This toolkit is **periodically refreshed** as stronger or more efficient models are released. When a model gets clearly outclassed, it gets swapped out — so the lineup you see today may differ from a few months down the line.
|
| 80 |
|
| 81 |
### 🎯 Design Philosophy
|
| 82 |
|
|
|
|
| 84 |
|
| 85 |
| Tier | Purpose | What It Means |
|
| 86 |
|------|---------|---------------|
|
| 87 |
+
| 🏆 **Quality** | Strong output | A high-quality model that fits in 12GB VRAM with room for context |
|
| 88 |
+
| ⚡ **Speed** | Fast iteration | A fast model that fits comfortably, ideal for rapid prototyping and quick tasks |
|
| 89 |
|
| 90 |
> **Note on Coding models:** The Coding Agents category includes two quality-tier models (Architect & Executor + Detective & Debugger). The "speed" variants in that category, while labeled for speed, are still powerful enough to serve as competent rapid-response code assistants — they simply trade some reasoning depth for significantly faster token generation.
|
| 91 |
|
|
|
|
| 205 |
|-------|------|-------------|------|
|
| 206 |
| [Agents-A1 35B MoE](https://huggingface.co/Chungulus/Agents-A1-IQ4_XS-imatrix-gguf-fable5-calibrated) | 35B (MoE) | IQ4_XS | **CEO Agent** — high-level autonomous planning and task decomposition. Calibrated with imatrix for superior quantization fidelity |
|
| 207 |
| [GLM-4.7-Flash](https://huggingface.co/unsloth/GLM-4.7-Flash-GGUF) | Flash | Q4_K_XL | **Router** — ultra-fast model for routing decisions, intent classification, and dispatching tasks to appropriate workers |
|
| 208 |
+
| [Hermes 4.3 36B](https://huggingface.co/NousResearch/Hermes-4.3-36B-GGUF) | 36B | Q4_K_M | **Quality Orchestrator** — a capable model for complex multi-step reasoning, tool use, and agent coordination. Excellent function calling |
|
| 209 |
|
| 210 |
### ⚙️ Agentic Workers
|
| 211 |
|
|
|
|
| 276 |
- **Sparse retrieval** — BM25-style keyword matching (great for exact term matches)
|
| 277 |
- **ColBERT retrieval** — late interaction for fine-grained token-level relevance scoring
|
| 278 |
|
| 279 |
+
This multi-modal retrieval approach means you combine semantic understanding and keyword precision in a single model. It supports 100+ languages out of the box.
|
| 280 |
|
| 281 |
### 🔄 2. Reranker — [BGE-Reranker-V2-M3](https://huggingface.co/Heisenberg0314/bge-reranker-v2-m3)
|
| 282 |
|
|
|
|
| 380 |
| 🎙️ Vocal Isolation | [Demucs V4 (ONNX)](https://huggingface.co/MrCitron/demucs-v4-onnx) | Studio-quality vocal separation — removes background music, noise, and other speakers. Essential for noisy recordings |
|
| 381 |
| 🔇 Silence/Noise Cutter | [Silero VAD (ONNX)](https://huggingface.co/istupakov/silero-vad-onnx) | Voice Activity Detection — automatically cuts silence and non-speech segments, reducing processing time and improving transcription accuracy |
|
| 382 |
|
| 383 |
+
### 🏆 Quality Tier — High Accuracy
|
| 384 |
|
| 385 |
| Scope | Model | Size | Highlights |
|
| 386 |
|-------|-------|------|------------|
|
|
|
|
| 507 |
|
| 508 |
All models in this toolkit are sourced from Hugging Face. During the initial download process, direct URLs were not recorded. The original sources were **recovered by matching SHA-256 hashes** against Hugging Face repositories. While some links may point to slightly different file versions within the same quantization family, the hash-verified content is identical to what is hosted in this toolkit.
|
| 509 |
|
|
|
|
|
|
|
|
|
|
| 510 |
### 1 — Universal Foundation LLMs
|
| 511 |
|
| 512 |
| File | Source |
|
|
|
|
| 593 |
| Huihui-Qwen3-VL-8B-Instruct-abliterated.Q8_0.gguf | [mradermacher/Huihui-Qwen3-VL-8B-Instruct-abliterated-GGUF](https://huggingface.co/mradermacher/Huihui-Qwen3-VL-8B-Instruct-abliterated-GGUF/blob/main/Huihui-Qwen3-VL-8B-Instruct-abliterated.Q8_0.gguf) |
|
| 594 |
| Huihui-Qwen3-VL-8B-Instruct-abliterated.mmproj-f16.gguf | [mradermacher/Huihui-Qwen3-VL-8B-Instruct-abliterated-GGUF](https://huggingface.co/mradermacher/Huihui-Qwen3-VL-8B-Instruct-abliterated-GGUF/blob/main/Huihui-Qwen3-VL-8B-Instruct-abliterated.mmproj-f16.gguf) |
|
| 595 |
|
|
|
|
|
|
|
| 596 |
---
|
| 597 |
|
| 598 |
## 🛠️ Recommended Tools for Running These Models
|
|
|
|
| 634 |
|
| 635 |
---
|
| 636 |
|
| 637 |
+
<i>Assembled with AI-assisted model selection · Curated for RTX 3060 12GB · Updated as stronger models arrive</i>
|
| 638 |
|
| 639 |
</div>
|