engival/YuE2-3B-EAGLE3-draft-GGUF

YuE2-3B EAGLE-3 draft head (GGUF)

An EAGLE-3 draft head for the semantic (audio-token) phase of YuE2-3B, for speculative decoding in yue2.cpp.

It works, but the gain is modest: about 1.1× on the semantic phase with exact sampling, up to ~1.35× with relaxed acceptance. The semantic phase is roughly a third of a render, so a whole song gets about 5–10 % faster. The ceiling comes from the task, not from this head: at temperature 1.0 the target puts only ~26 % of its probability on its top codec token, so no draft can be accepted much more often than about half the time.

file	type	size	sha256
YuE2-3B-EAGLE3-draft-Q8_0.gguf	Q8_0	154 MB	54e2219f7d021373d05bcd4c932b7bf7d362e705b7f10e59195ea6af9d374d4b

This is the exact file every number below was measured with. Quantizing costs no acceptance (Q4_K_M was tested too), and Q8_0 was the fastest at K=2.

Usage

Draft support lives on the yue2.cpp branch feat/eagle3-draft. It is an experiment and is not merged into master. Build that branch as described in its README, then add --draft to yue2 song, yue2 ar or yue2 batch:

yue2 song --request request.json --out song.flac \
  --ar yue2-ar-q8_0.gguf --nar yue2-nar-f16.gguf --vae yue2-vae-f32.gguf \
  --draft YuE2-3B-EAGLE3-draft-Q8_0.gguf --draft-k 2

flag	default	meaning
--draft FILE	–	this GGUF
--draft-k N	–	tokens drafted per round; 2 is best on the GPUs tested (larger K loses)
--draft-lambda L	1	1 = exact: the output has the same distribution as without a draft. L > 1 relaxes acceptance (lossy, faster)
--draft-q shaped|raw	shaped	apply the target's sampler shaping to the draft's distribution
--draft-window N	0	0 = whole-song context for the draft; no measurable difference
--draft-trace	off	write draft_trace.npy with per-token acceptance data

Limits: unguided songs only (requests with guidance / cfg_scale ≠ 1, semantic_keep or handover are rejected), --parallel 1, no --greedy. The head was trained on features from the Q8_0 YuE2-3B AR GGUF made by yue2.cpp's converter (yue2.source_sha256 = 1d55c42c…1e59e9); other target quantizations are untested.

Results

Exact sampling (λ = 1), Intel Arc Pro B70, Vulkan

AR stage only, 7 held-out songs (35.7 k semantic tokens per variant).

head	K	semantic tok/s	vs no draft	accept d1	d2
none	–	122.7	1.000×
F32	2	123.8	1.009×	0.546	0.446
F16	2	136.2	1.110×	0.549	0.457
Q8_0	2	141.0	1.149×	0.544	0.459
Q8_0	3	127.0	1.036×	0.555	0.451

A drafted round costs ~1–1.4 ms per head call regardless of weight type (launch/sync bound), and verifying 3+ rows is noticeably dearer than 1, which is why K > 2 loses.

Exact vs lossy, AMD Radeon RX 7900 XTX, Vulkan

Full yue2 song, Q8_0 head, K = 2, the same 7 songs.

variant	semantic tok/s	vs no draft	tokens/round	accept d1
no draft	225.8	1.000×	–	–
exact (λ 1)	249.4	1.105×	1.79	0.546
λ 1.5	271.9	1.204×	2.00	0.645
λ 2	289.0	1.280×	2.12	0.703
λ 3	306.5	1.357×	2.26	0.763

With the same seed every variant renders the same score (the draft only touches the semantic phase). In a side-by-side listening test, λ 3 could not be called worse than exact: the songs differ slightly in timbre and accompaniment, and lyrics drift then re-sync at section boundaries. Use λ > 1 at your own judgement.

Model

• One Qwen3-style decoder layer (hidden 2048, 16 heads / 8 KV heads, FFN 6144), standard EAGLE-3 layout (general.architecture = eagle3).
• Input: the target's hidden states from layers 2, 14 and 25, fused by fc (6144 → 2048), plus the target's token embeddings.
• Output vocabulary: the 32 769 semantic ids (32 768 codec tokens + end of music), mapped back to the target vocabulary by d2t. The draft cannot propose text tokens; the semantic phase never emits them.

Training

• Data: YuE2-3B's own renders (no external audio): about 580 takes, 3.26 M semantic positions. Part are the author's songs, the rest songs with LLM-written lyrics across a wide range of genres, made for this purpose. Lead vocals are about 2:1 male, and almost all lyrics are English (YuE2 also sings Mandarin; that is barely covered). 7 songs held out.
• Labels: the target's sampler-shaped distribution at each position (repetition penalty, top-k 100, top-p 0.95), not just the sampled token.
• Objective: cross-entropy against those labels, unrolled 4 draft steps (training-time test), chunk 2048, AdamW, F32.
• Schedule: 20 epochs at lr 1e-4, then a 7-epoch warm restart at lr 5e-5 with extra data. About 9 GPU-hours on one 7900 XTX.
• Trainer: written from scratch on ggml / ggml_opt and runs on Vulkan (no PyTorch, no CUDA). It will be published at github.com/engival/yue2-eagle3-train.

Held-out acceptance from the trainer (scheme B, Σ min(p, q)) by draft depth 0–3: 0.486 / 0.384 / 0.316 / 0.261. Online acceptance in yue2.cpp came out a few points higher. More data was still helping but slowly (+38 % data → +0.004 at depth 0), and the head is close to what one layer can do here.

License

CC BY-NC 4.0, non-commercial. This head is a derivative of the YuE2-3B weights (it was trained on their hidden states and output distributions), so it carries their licence. The yue2.cpp code is MIT; that does not change the licence of these weights.

Citation

If you use this, cite YuE2 (see the base model card) and EAGLE-3:

@article{li2025eagle3,
  title   = {EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test},
  author  = {Li, Yuhui and Wei, Fangyun and Zhang, Chao and Zhang, Hongyang},
  journal = {arXiv preprint arXiv:2503.01840},
  year    = {2025}
}