We introduce Hy4 flash GGUF from Nace.AI: Tencent's Hy4-flash in four GGUF sizes, starting at 87.8 GB and 3.21 bits per weight.

Results

Hy4-flash quantization quality across 37 measured variants

37 measured Hy4-flash variants on the same 11 texts, 5,621 scored tokens. Chart sizes use text-weight payload, including required Engram tables, excluding vision/MTP. Complete weight-file sizes and exact scores are in the comparison data.

GGUF Bits per weight Download size LiveCodeBench
IQ3_XXS 3.21 87.8 GB 39/100
Q3_K_M 3.9 107.49 GB
IQ4_XS-4.48bpw 4.48 122.58 GB
Q4_K_M 4.6 127.22 GB

IQ3_XXS has lower KL than AtomicChat's 84.7 GB IQ2 file on these texts. KL divergence measures how far the next-token probabilities are from the native Hy4-flash. Lower is closer.

IQ4_XS-4.48bpw has the lowest KL of every measured file at its size or smaller.

Nace Weight Space Quantization

Precision goes where Hy4-flash needs it.

Nace Weight Space Quantization assigns precision by weight class and rebalances weights before compression. The IQ3_XXS variant stores its Engram lookup table as Q4_0. Q3_K_M and Q4_K_M store it as IQ4_NL. IQ4_XS-4.48bpw is requantized from Q8, retaining higher-precision tensors and an IQ4_NL Engram table.

Run it

Use Little0o0/llama.cpp at 0a054d0, the CUDA build used for the IQ3_XXS LiveCodeBench run.

hf download nace-ai/Hy4-flash-GGUF \
  IQ3_XXS/Hy4-flash-IQ3_XXS.gguf --local-dir models

git clone https://github.com/Little0o0/llama.cpp.git hy4-llama.cpp
git -C hy4-llama.cpp checkout \
  0a054d0b02454c998e068381d15f3f6615bdd987
cmake -S hy4-llama.cpp -B hy4-llama.cpp/build \
  -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build hy4-llama.cpp/build --target llama-cli -j

hy4-llama.cpp/build/bin/llama-cli \
  -m models/IQ3_XXS/Hy4-flash-IQ3_XXS.gguf \
  -c 4096 -ngl 999 -n 32 --temp 0 --top-k 1 -s 1 \
  -st --simple-io --no-display-prompt --no-warmup \
  --chat-template-kwargs '{"reasoning_effort":"no_think"}' \
  -p "Name three primary colors."

IQ3_XXS is one file, including its Engram table. For Q3_K_M or Q4_K_M, download Hy4-flash-Q3_K_M-*.gguf or Hy4-flash-Q4_K_M-*.gguf from this repo and pass the first shard to llama-cli. It loads the other two shards from the same folder.

For IQ4_XS-4.48bpw, download all GGUF shards from its folder and pass the first shard to llama-cli. Keep all shards in the same folder.

License

Tencent Hy License, Version 1.0, including its Section 10 commercial terms for large Model-as-a-Service operators. See also NOTICE and MODIFICATIONS.md.

Downloads last month
2,179
GGUF
Model size
219B params
Architecture
hy_v4_flash
Hardware compatibility
Log In to add your hardware

3-bit

4-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for nace-ai/Hy4-flash-GGUF

Quantized
(44)
this model

Space using nace-ai/Hy4-flash-GGUF 1

Collection including nace-ai/Hy4-flash-GGUF