| # Quantization Methodology |
|
|
| ## Overview |
|
|
| Sixpert K1 is released in Q4_K_M GGUF format. This document details the quantization methodology, quality benchmarks, and guidance for users selecting quantization levels. |
|
|
| ## What is Q4_K_M? |
|
|
| Q4_K_M is a 4-bit K-quantization method that provides: |
|
|
| - **4-bit weights** with block-wise quantization |
| - **Per-block scales** for fine-grained accuracy |
| - **K-quant optimization** that preserves important weight groups |
| - **Medium quality tier** balancing speed and accuracy |
|
|
| ## Quantization Comparison |
|
|
| | Method | Bits | File Size | Quality | Speed | |
| |---|---|---|---|---| |
| | FP16 (original) | 16 | ~17.4 GB | Maximum | Slowest | |
| | Q8_0 | 8 | ~9.0 GB | Near-lossless | Fast | |
| | Q6_K | 6 | ~6.8 GB | Excellent | Very Fast | |
| | Q5_K_M | 5 | ~5.8 GB | Great | Very Fast | |
| | **Q4_K_M** | **4** | **~5.0 GB** | **Good** | **Fastest** | |
| | Q4_0 | 4 | ~4.6 GB | Acceptable | Fast | |
| | Q3_K_M | 3 | ~3.8 GB | Lower | Fast | |
| |
| ## Quality Retention |
| |
| Benchmarks comparing Q4_K_M to FP16 baseline: |
| |
| | Benchmark | FP16 Score | Q4_K_M Score | Retention | |
| |---|---|---|---| |
| | MMLU | 72.1 | 70.8 | 98.2% | |
| | HumanEval | 68.4 | 66.1 | 96.6% | |
| | GSM8K | 82.3 | 80.5 | 97.8% | |
| | TruthfulQA | 61.2 | 59.8 | 97.7% | |
| | MATH | 54.7 | 52.9 | 96.7% | |
| |
| ## Conversion Commands |
| |
| To convert to other quantization levels: |
| |
| ```bash |
| # Install llama.cpp |
| git clone https://github.com/ggerganov/llama.cpp |
| cd llama.cpp && make |
| |
| # Quantize to Q8_0 |
| ./llama-quantize SixpertK1.gguf SixpertK1-Q8_0.gguf Q8_0 |
|
|
| # Quantize to Q6_K |
| ./llama-quantize SixpertK1.gguf SixpertK1-Q6_K.gguf Q6_K |
| |
| # Quantize to Q5_K_M |
| ./llama-quantize SixpertK1.gguf SixpertK1-Q5_K_M.gguf Q5_K_M |
| ``` |
| |
| ## GGUF Format Details |
| |
| The GGUF (GPT-Generated Unified Format) specification used: |
| |
| - **Version**: 3 |
| - **Metadata**: Includes model architecture, tokenizer, and training info |
| - **Alignment**: 512-byte aligned for mmap compatibility |
| - **Metadata KV**: Contains all model hyperparameters |
| |
| ## Recommendations |
| |
| | Hardware | Recommended Quant | |
| |---|---| |
| | Apple M1/M2 (8GB) | Q4_K_M (this release) | |
| | Apple M1/M2 (16GB+) | Q6_K or Q8_0 | |
| | NVIDIA RTX 3060 (12GB) | Q6_K or Q8_0 | |
| | NVIDIA RTX 4060 (8GB) | Q4_K_M (this release) | |
| | NVIDIA RTX 3090 (24GB) | Q8_0 or FP16 | |
| | CPU-only (16GB RAM) | Q4_K_M (this release) | |
| | CPU-only (32GB+ RAM) | Q6_K or Q8_0 | |
|
|
