q-qwen2.5-1.5b-instruct-q4: Qwen2.5-1.5B-Instruct as one streamable .holo

This is Qwen/Qwen2.5-1.5B-Instruct-GGUF at revision 91cad51170dc346986eccefdc2dd33a9da36ead9, file qwen2.5-1.5b-instruct-q4_0.gguf (1,066,227,232 bytes), re-laid for the Hologram engine so it runs in a browser on WebGPU.

What changed and what did not:

  • 197 Q4_0 weight tensors pass through: the same 4-bit codes, with each 32-weight group's f16 scale widened to f32.
  • 141 f32 tensors (norms and the q, k, v biases) pass through.
  • One tensor is requantised: the output head, Q6_K in the source, is dequantised and requantised to the same 4-bit format (one f32 scale per 32 weights).

Experimental. It runs on the Hologram AI engine and on the original Q engine; tools that read GGUF do not read this format.

File

File What it is
q-qwen2.5-1.5b-instruct-q4.v1.holo One file, 953,978,079 bytes: header, embedded tokenizer (the source GGUF header), then gzip'd tensor blocks in the order the engine reads them, each named by the BLAKE3 of its stored bytes.
  • sha256 of the file: 84b00c7b445488f27d4df0fafc9e74d4ccbd4e478390f243b59c9d765fe6eefc
  • manifest root: sha256:2147533113fbe3634ae3e1fceaf2fe0f6d72dbd948e6ab78c6a9f6924fc1348b
  • shape: Qwen2, 28 layers, width 1536, 12 heads, 2 key-value heads, head size 128, FFN 8960, vocabulary 151,936, q/k/v biases, output head separate from the embedding, rotary base 1e6.

Format

The header manifest is holo-2bit/1 with mode: "q4", bits: 4, attn_bias: true, qk_norm: false. A q4 block is [codes N*K/2 bytes][f32 scale per 32 weights]; a weight is scale * (code - 8).

Measured

One laptop (Radeon RDNA-3 integrated GPU, Chromium, WebGPU), file served locally, greedy decoding, every run cold:

Hologram AI engine Original Q engine
First token, 41-token prompt 156 ms 545 ms
First token, ~430-token prompts 1.85 s 5.0 s
Tokens per second, ~430-token prompts 70.7 73.6

Both engines produced the same tokens on every prompt measured.

Licence

Apache 2.0, as the source. LICENSE is copied unchanged from the source repository; NOTICE.txt states the modifications.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for HOLOGRAMTECH/q-qwen2.5-1.5b-instruct-q4

Finetuned
(4)
this model