Hugging Face's logo
None's picture

None

CompressedGemma
4
emanubiz's profile picture ironbcc's profile picture nakwMt's profile picture
¡

AI & ML interests

None yet

Recent Activity

reacted to savoji's post with 🔥 about 10 hours ago
A 284B-parameter DeepSeek-V4-Flash now runs on a 64 GB desktop with 0 GPUs: llama-modchip shrinks the checkpoint to 56.56 GiB (1.71 bpw) and decodes at 9.1 to 14.7 tok/s. The standard UD-IQ1_S build is 76.87 GiB, above the ~62.5 GiB usable on a 64 GB Linux box. The repo mixes precisions (routed experts mostly IQ1_S, MLA projections in a custom Fractal-64 type) and patches the CPU kernels. The model plus a 256K KV cache then sits at 57.61 to 58.9 GiB RSS with zero swap. The repo was created on 2026-10-29 and already has 744 stars and 470 points on Hacker News. 📊 The numbers • Size: 56.56 GiB, 20.31 GiB smaller than UD-IQ1_S • Prefill: 25.0 to 44.36 tok/s (+77%) on a 16-core Ryzen 9 9955HX • Decode: 5.91 tok/s unpatched, 9.06 to 9.95 tok/s at Top-6, 11.5 to 14.7 tok/s with the n-gram drafter ⚠️ Caveats • "Lossless" means bit-exact against the same re-quantized checkpoint. Versus UD-IQ1_S the author reports 0.9964 top-logit cosine similarity • The 98.5/100 quality score comes from the author's own gate, and the headline speeds come from one machine • The custom tensor type 40 is rejected by stock llama.cpp, Ollama and LM Studio 🛠️ Try it • Run scripts/check_cpu_readiness.py from the repo: it needs no model download • Run ./modchip bench for speed, then ./modchip eval to check the quality claims yourself Repo: https://github.com/synchopate/llama-modchip HN thread: https://news.ycombinator.com/item?id=50411961 Would you accept a custom runtime for 9 to 15 tok/s from a 284B model on a 64 GB box? 📚 Read it elsewhere • The deep dive on Medium: https://medium.com/p/e47ab3581f58 • The long read on Substack: https://sajjadpakdamansavoji.substack.com/p/llama-modchip-runs-a-284b-deepseek • The thread on X: https://x.com/saavoji/status/2119858773280141526 • Every version on sajjadpsavoji.github.io: https://sajjadpsavoji.github.io/blog#llama-modchip
reacted to Ternii's post with 🔥 about 15 hours ago
Published TR Quantisation versions of Qwen3.6-27B, Qwen3.6-35B-A3B and Qwen3.8-Flash-Next: weights packed into trits (2.25-3 per weight), no retraining. - Mac: Qwen3.6-35B-A3B in a 16.5 GB GGUF that fits a 24 GB Mac, ~120 tok/s sustained on an M5 Max. Needs our llama.cpp build (ready-built for macOS and Linux arm64). - vLLM on one DGX Spark: 35B at 180.9 tok/s for 1 user vs 128.0 for NVIDIA's NVFP4, 17.4 vs 21.1 GiB, no significant accuracy difference on the core benchmarks. - Trade-offs on each card: slower prompt processing, lower multilingual maths, and 4-bit GGUF stays closer to the original text. https://huggingface.co/collections/Ternii/tr-quantisation
repliedto Ternii's post about 15 hours ago
Published TR Quantisation versions of Qwen3.6-27B, Qwen3.6-35B-A3B and Qwen3.8-Flash-Next: weights packed into trits (2.25-3 per weight), no retraining. - Mac: Qwen3.6-35B-A3B in a 16.5 GB GGUF that fits a 24 GB Mac, ~120 tok/s sustained on an M5 Max. Needs our llama.cpp build (ready-built for macOS and Linux arm64). - vLLM on one DGX Spark: 35B at 180.9 tok/s for 1 user vs 128.0 for NVIDIA's NVFP4, 17.4 vs 21.1 GiB, no significant accuracy difference on the core benchmarks. - Trade-offs on each card: slower prompt processing, lower multilingual maths, and 4-bit GGUF stays closer to the original text. https://huggingface.co/collections/Ternii/tr-quantisation
View all activity

Organizations

None yet