中文 | English

AngelSlim

A more accessible, comprehensive, and efficient toolkit for large model compression.

✒️ TechnicalReport   |    📖 Documentation   |   🤗 Hugging Face   |   🤖 ModelScope

💬 WeChat |   🫨 Discord

Hy4-flash GGUF

Five GGUF builds of Hy4-flash: tencent/Hy4-flash

file model size bpw (overall) bpw (excl. Engram table)
Hy4-flash-Q4_K_M.gguf 156.79 GiB 6.05 5.27
Hy4-flash-SPARK-4Bit.gguf 151.3 GiB 5.84 4.78
Hy4-flash-SPARK-4Bit-Fast.gguf 148.6 GiB 5.74 4.63
Hy4-flash-Q3_K.gguf 144.70 GiB 5.58 4.62
Hy4-flash-IQ2-AH.gguf 125.73 GiB 4.85 3.60

Hy4-flash GGUF benchmark results


1. What these are

Hy4-flash-Q4_K_M.gguf — standard 4-bit, the safe default. The routed experts follow llama.cpp's Q4_K_M recipe: gate/up are Q4_K, and down gets Q6_K on 19 layers via llama.cpp's fixed rule and Q4_K on the other 22. Most non-expert weights are Q8_0. Use this unless you are memory-constrained.

Hy4-flash-SPARK-4Bit.gguf — the SPARK-4Bit build (NVIDIA's 4-bit floating-point format). Suited for Spark devices.

Hy4-flash-SPARK-4Bit-Fast.gguf — a variant of the SPARK-4Bit build, 2.7 GiB smaller. Also suited for Spark devices.

Hy4-flash-Q3_K.gguf — mixed Q3_K/Q4_K experts. The routed-expert gate/up projections keep Q4_K on 19 layers chosen by quantization sensitivity and drop to Q3_K on the other 22; down is Q4_K. Everything outside the routed experts is the same as Q4_K_M.

Hy4-flash-IQ2-AH.gguf — the smallest of the five. The routed-expert gate/up projections run at 2.0625 bpw (IQ2_XXS) and down at Q4_K, while most non-expert weights stay in BF16 rather than Q8_0.

Strata version coming soon

2. SPEED

Mac M5 Max 128GB

format Prompt tok/s (4K / 16K / 32K) Decode tok/s (4K / 16K / 32K)
Q4_K_M 629.8 / 448.1 / 418.6 48.13 / 45.18 / 44.15
Q3_K 743.2 / 563.0 / 520.6 50.34 / 49.47 / 47.37

For code generation, Q3_K reaches up to 70 tok/s.

RTX Spark 128GB

format Prompt tok/s (4K / 16K / 32K) Decode tok/s (4K / 16K / 32K)
SPARK-4Bit-Fast 799 / 717 / 707 31.95 / 33.14 / 33.39
SPARK-4Bit 765 / 686 / 666 26.28 / 30.23 / 35.38

For code generation, SPARK-4Bit-Fast reaches up to 40 tok/s.

3. How to use

General Usage

The support is in PR #32517 and not merged yet, so check out the PR branch:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git fetch origin pull/32517/head:hy4-flash
git checkout hy4-flash

Build:

cmake -B build                # add -DGGML_CUDA=ON for NVIDIA GPUs; Metal is on by default on macOS
cmake --build build --config Release -j

Chat in the terminal:

./build/bin/llama-cli -m Hy4-flash-Q4_K_M-00001-of-00002.gguf -c 8192 -ngl 99 --jinja

OpenAI-compatible server (http://localhost:8080):

./build/bin/llama-server -m Hy4-flash-Q4_K_M-00001-of-00002.gguf -c 8192 -ngl 99 --jinja --port 8080

Image input, with the mmproj file:

./build/bin/llama-mtmd-cli -m Hy4-flash-Q4_K_M-00001-of-00002.gguf --mmproj mmproj-Hy4-flash-bf16.gguf \
    --image photo.png -p "Describe this image." -c 8192 -ngl 99 --jinja

Mac M5 Max 128GB

cd llama.cpp
git checkout 2cbca389f          
git apply hy4-flash-mac-minimal.patch
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_METAL=ON && cmake --build build -j

RTX Spark 128GB

See README_SPARK-4Bit.md

Downloads last month
-
GGUF
Model size
223B params
Architecture
hy_v4_flash
Hardware compatibility
Log In to add your hardware

4-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for AngelSlim/Hy4-flash-GGUF

Quantized
(1)
this model

Collection including AngelSlim/Hy4-flash-GGUF

Paper for AngelSlim/Hy4-flash-GGUF