None

CompressedGemma

AI & ML interests

None yet

Recent Activity

reacted to savoji's post with 🔥 about 10 hours ago

A 284B-parameter DeepSeek-V4-Flash now runs on a 64 GB desktop with 0 GPUs: llama-modchip shrinks the checkpoint to 56.56 GiB (1.71 bpw) and decodes at 9.1 to 14.7 tok/s. The standard UD-IQ1_S build is 76.87 GiB, above the ~62.5 GiB usable on a 64 GB Linux box. The repo mixes precisions (routed experts mostly IQ1_S, MLA projections in a custom Fractal-64 type) and patches the CPU kernels. The model plus a 256K KV cache then sits at 57.61 to 58.9 GiB RSS with zero swap. The repo was created on 2026-10-29 and already has 744 stars and 470 points on Hacker News. 📊 The numbers • Size: 56.56 GiB, 20.31 GiB smaller than UD-IQ1_S • Prefill: 25.0 to 44.36 tok/s (+77%) on a 16-core Ryzen 9 9955HX • Decode: 5.91 tok/s unpatched, 9.06 to 9.95 tok/s at Top-6, 11.5 to 14.7 tok/s with the n-gram drafter ⚠️ Caveats • "Lossless" means bit-exact against the same re-quantized checkpoint. Versus UD-IQ1_S the author reports 0.9964 top-logit cosine similarity • The 98.5/100 quality score comes from the author's own gate, and the headline speeds come from one machine • The custom tensor type 40 is rejected by stock llama.cpp, Ollama and LM Studio 🛠️ Try it • Run scripts/check_cpu_readiness.py from the repo: it needs no model download • Run ./modchip bench for speed, then ./modchip eval to check the quality claims yourself Repo: https://github.com/synchopate/llama-modchip HN thread: https://news.ycombinator.com/item?id=50411961 Would you accept a custom runtime for 9 to 15 tok/s from a 284B model on a 64 GB box? 📚 Read it elsewhere • The deep dive on Medium: https://medium.com/p/e47ab3581f58 • The long read on Substack: https://sajjadpakdamansavoji.substack.com/p/llama-modchip-runs-a-284b-deepseek • The thread on X: https://x.com/saavoji/status/2119858773280141526 • Every version on sajjadpsavoji.github.io: https://sajjadpsavoji.github.io/blog#llama-modchip

reacted to Ternii's post with 🔥 about 15 hours ago

Published TR Quantisation versions of Qwen3.6-27B, Qwen3.6-35B-A3B and Qwen3.8-Flash-Next: weights packed into trits (2.25-3 per weight), no retraining. - Mac: Qwen3.6-35B-A3B in a 16.5 GB GGUF that fits a 24 GB Mac, ~120 tok/s sustained on an M5 Max. Needs our llama.cpp build (ready-built for macOS and Linux arm64). - vLLM on one DGX Spark: 35B at 180.9 tok/s for 1 user vs 128.0 for NVIDIA's NVFP4, 17.4 vs 21.1 GiB, no significant accuracy difference on the core benchmarks. - Trade-offs on each card: slower prompt processing, lower multilingual maths, and 4-bit GGUF stays closer to the original text. https://huggingface.co/collections/Ternii/tr-quantisation

repliedto Ternii's post about 15 hours ago

View all activity

Organizations

None yet

reacted to savoji's post with 🔥 about 10 hours ago

Post

915

A 284B-parameter DeepSeek-V4-Flash now runs on a 64 GB desktop with 0 GPUs: llama-modchip shrinks the checkpoint to 56.56 GiB (1.71 bpw) and decodes at 9.1 to 14.7 tok/s.

The standard UD-IQ1_S build is 76.87 GiB, above the ~62.5 GiB usable on a 64 GB Linux box. The repo mixes precisions (routed experts mostly IQ1_S, MLA projections in a custom Fractal-64 type) and patches the CPU kernels. The model plus a 256K KV cache then sits at 57.61 to 58.9 GiB RSS with zero swap. The repo was created on 2026-10-29 and already has 744 stars and 470 points on Hacker News.

📊 The numbers
• Size: 56.56 GiB, 20.31 GiB smaller than UD-IQ1_S
• Prefill: 25.0 to 44.36 tok/s (+77%) on a 16-core Ryzen 9 9955HX
• Decode: 5.91 tok/s unpatched, 9.06 to 9.95 tok/s at Top-6, 11.5 to 14.7 tok/s with the n-gram drafter

⚠️ Caveats
• "Lossless" means bit-exact against the same re-quantized checkpoint. Versus UD-IQ1_S the author reports 0.9964 top-logit cosine similarity
• The 98.5/100 quality score comes from the author's own gate, and the headline speeds come from one machine
• The custom tensor type 40 is rejected by stock llama.cpp, Ollama and LM Studio

🛠️ Try it
• Run scripts/check_cpu_readiness.py from the repo: it needs no model download
• Run ./modchip bench for speed, then ./modchip eval to check the quality claims yourself

Repo:
https://github.com/synchopate/llama-modchip

HN thread:
https://news.ycombinator.com/item?id=50411961

Would you accept a custom runtime for 9 to 15 tok/s from a 284B model on a 64 GB box?

📚 Read it elsewhere
• The deep dive on Medium: https://medium.com/p/e47ab3581f58
• The long read on Substack: https://sajjadpakdamansavoji.substack.com/p/llama-modchip-runs-a-284b-deepseek
• The thread on X: https://x.com/saavoji/status/2119858773280141526
• Every version on sajjadpsavoji.github.io: https://sajjadpsavoji.github.io/blog#llama-modchip

1 reply

reacted to Ternii's post with 🔥 about 15 hours ago

Post

3,843

Published TR Quantisation versions of Qwen3.6-27B, Qwen3.6-35B-A3B and Qwen3.8-Flash-Next: weights packed into trits (2.25-3 per weight), no retraining.

- Mac: Qwen3.6-35B-A3B in a 16.5 GB GGUF that fits a 24 GB Mac, ~120 tok/s sustained on an M5 Max. Needs our llama.cpp build (ready-built for macOS and Linux arm64).
- vLLM on one DGX Spark: 35B at 180.9 tok/s for 1 user vs 128.0 for NVIDIA's NVFP4, 17.4 vs 21.1 GiB, no significant accuracy difference on the core benchmarks.
- Trade-offs on each card: slower prompt processing, lower multilingual maths, and 4-bit GGUF stays closer to the original text.

https://huggingface.co/collections/Ternii/tr-quantisation

2 replies

replied to Ternii's post about 15 hours ago

Hi Wai.

Given that Ternii translates standard FP16/BF16 weights down to variable trit densities (TR2.25 to TR3) without retraining, I wanted to reach out and share a highly relevant optimization from my pipeline that could directly address the multilingual and mathematical regressions noted in your model cards.

I introduced a Symbolic Lattice Algebraic Solver that uses an exact algebraic pinning strategy to drop net DC bias (mean error shift) within quantized blocks to absolute zero.

Because ternary integer codes are inherently symmetric, my DC pinning method can surely be used by your pipeline.

By forcing the residual error vector to be perfectly orthogonal to the intercept vector, you can eliminate the subtle, accumulating distortion that degrades deep reasoning tasks in low-bit states.