mohammad commited on
Commit
c404781
Β·
verified Β·
1 Parent(s): 47555ea

Upload README.md

Browse files
Files changed (1) hide show
  1. README.md +11 -3
README.md CHANGED
@@ -80,12 +80,12 @@ This toolkit is **periodically refreshed** as stronger or more efficient models
80
 
81
  ### 🎯 Design Philosophy
82
 
83
- Each category follows a dual-tier architecture:
84
 
85
  | Tier | Purpose | What It Means |
86
  |------|---------|---------------|
87
- | πŸ† **Quality** | Strong output | A high-quality model that fits in 12GB VRAM with room for context |
88
- | ⚑ **Speed** | Fast iteration | A fast model that fits comfortably, ideal for rapid prototyping and quick tasks |
89
 
90
  > **Note on Coding models:** The Coding Agents category includes two quality-tier models (Architect & Executor + Detective & Debugger). The "speed" variants in that category, while labeled for speed, are still powerful enough to serve as competent rapid-response code assistants β€” they simply trade some reasoning depth for significantly faster token generation.
91
 
@@ -438,6 +438,14 @@ Multimodal models are the frontier of local AI β€” they can analyze images, desc
438
  | πŸ† Quality | [Qwen3-Omni 30B-A3B](https://huggingface.co/ggml-org/Qwen3-Omni-30B-A3B-Instruct-GGUF) | 30B (MoE, 3B active) | Q4_K_M | **Full omni-model** β€” understands text, images, AND audio natively. The most capable multimodal model in this collection. MoE architecture keeps VRAM usage manageable |
439
  | ⚑ Speed | [Gemma 4 E4B](https://huggingface.co/unsloth/gemma-4-E4B-it-GGUF) | ~4B (effective) | Q4_K_XL | Speculative decoding with MTP draft model for faster generation. Excellent for quick image analysis, screenshot understanding, and visual Q&A. Despite its small effective size, Gemma 4 punches well above its weight |
440
 
 
 
 
 
 
 
 
 
441
  **Why Qwen3-Omni for quality?** It's one of the very few models that genuinely handles text + image + audio in a unified architecture. Most "multimodal" models only handle text and images β€” Qwen3-Omni adds audio understanding, making it a true all-in-one model for multimedia tasks.
442
 
443
  <details>
 
80
 
81
  ### 🎯 Design Philosophy
82
 
83
+ Each category follows a dual-tier architecture. Both tiers target the **same 12GB VRAM + 32GB RAM machine** β€” the difference is *how* that budget is used:
84
 
85
  | Tier | Purpose | What It Means |
86
  |------|---------|---------------|
87
+ | πŸ† **Quality** | Best achievable quality | The highest-quality model that can still run within the 12GB VRAM + 32GB RAM budget, using **GPU+CPU offload** (some layers spill into system RAM). Slower, but maximizes output quality |
88
+ | ⚑ **Speed** | Fastest, no offload | A capable model that fits **entirely within 12GB VRAM** β€” leaving room for context too β€” so it runs fully on-GPU with no CPU offload overhead. Prioritizes throughput and low latency |
89
 
90
  > **Note on Coding models:** The Coding Agents category includes two quality-tier models (Architect & Executor + Detective & Debugger). The "speed" variants in that category, while labeled for speed, are still powerful enough to serve as competent rapid-response code assistants β€” they simply trade some reasoning depth for significantly faster token generation.
91
 
 
438
  | πŸ† Quality | [Qwen3-Omni 30B-A3B](https://huggingface.co/ggml-org/Qwen3-Omni-30B-A3B-Instruct-GGUF) | 30B (MoE, 3B active) | Q4_K_M | **Full omni-model** β€” understands text, images, AND audio natively. The most capable multimodal model in this collection. MoE architecture keeps VRAM usage manageable |
439
  | ⚑ Speed | [Gemma 4 E4B](https://huggingface.co/unsloth/gemma-4-E4B-it-GGUF) | ~4B (effective) | Q4_K_XL | Speculative decoding with MTP draft model for faster generation. Excellent for quick image analysis, screenshot understanding, and visual Q&A. Despite its small effective size, Gemma 4 punches well above its weight |
440
 
441
+ > ⚠️ **CRITICAL β€” Read before relying on "multimodal":** The models shipped here are **GGUF quantized** files. In their current state, tooling only supports the **vision (image)** modality from these quantized files β€” **audio input does not work** on the quantized versions.
442
+ >
443
+ > True text + image + audio multimodality is currently only available from the **original, uncompressed (FP16/BF16) models** β€” not from the quantized GGUF files in this toolkit. The quantized variants effectively behave as **vision-language models (VLM)**: they can see images and read text, but they **cannot process audio**.
444
+ >
445
+ > This is a tooling limitation, not a model defect β€” current inference engines (llama.cpp, Ollama, LM Studio, etc.) cannot yet route audio through quantized GGUF multimodal pipelines. If you need genuine audio understanding, download and run the **original uncompressed weights** from the source repos linked below.
446
+ >
447
+ > I learned this the hard way after downloading β€” the quantized files simply won't run as full multimodal. Hence this warning.
448
+
449
  **Why Qwen3-Omni for quality?** It's one of the very few models that genuinely handles text + image + audio in a unified architecture. Most "multimodal" models only handle text and images β€” Qwen3-Omni adds audio understanding, making it a true all-in-one model for multimedia tasks.
450
 
451
  <details>