Upload README.md with huggingface_hub
Browse files
README.md
ADDED
|
@@ -0,0 +1,84 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
tags:
|
| 4 |
+
- dflash2
|
| 5 |
+
- speculative-decoding
|
| 6 |
+
- draft-model
|
| 7 |
+
- qwen3
|
| 8 |
+
base_model:
|
| 9 |
+
- incoai/Qwen3.8-27B-DFlash2
|
| 10 |
+
---
|
| 11 |
+
|
| 12 |
+
# Arete DFlash2 Drafter (Z-Lab base finetuned)
|
| 13 |
+
|
| 14 |
+
DFlash2 draft model for speculative decoding, trained by SargeDev on the
|
| 15 |
+
GB10 (2x GB10 cluster, vLLM 0.28).
|
| 16 |
+
|
| 17 |
+
## What this is
|
| 18 |
+
|
| 19 |
+
A DFlash2 drafter: a small (1.92B) draft model that proposes tokens for a
|
| 20 |
+
larger verifier (target) model. The verifier scores the drafts in one forward
|
| 21 |
+
pass and accepts a prefix — lossless speedups, identical output.
|
| 22 |
+
|
| 23 |
+
Two drafters ship, both in z-lab export format (`DFlash2DraftModel` +
|
| 24 |
+
`dflash_config`), drop-in for vLLM `--speculative-config`:
|
| 25 |
+
|
| 26 |
+
| repo | base | training |
|
| 27 |
+
|---|---|---|
|
| 28 |
+
| `SargeDev/arete-dflash2-scratch` | random init | 5 epochs on Arete-27B-Uncensored regen data |
|
| 29 |
+
| `SargeDev/arete-dflash2-incoai` | [incoai/Qwen3.8-27B-DFlash2](https://huggingface.co/incoai/Qwen3.8-27B-DFlash2) (Z-Lab DFlash2, 1.92B) | 7 epochs finetune on the same data |
|
| 30 |
+
|
| 31 |
+
## Targets
|
| 32 |
+
|
| 33 |
+
Both drafters are **drop-in compatible** with all three (greedy losslessness
|
| 34 |
+
5/5 on every combo — verified):
|
| 35 |
+
|
| 36 |
+
| target | repo |
|
| 37 |
+
|---|---|
|
| 38 |
+
| Arete-27B-Uncensored (our uncensored build, bf16) | [SargeDev/Arete-27B-Uncensored-V2](https://huggingface.co/SargeDev/Arete-27B-Uncensored-V2) |
|
| 39 |
+
| Arete-27B (censored, original) | [NousResearch/Arete-27B](https://huggingface.co/NousResearch/Arete-27B) |
|
| 40 |
+
| Qwen3.8-27B | [Qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B) |
|
| 41 |
+
|
| 42 |
+
## Benchmarks (vLLM 0.28, GB10, single target serve)
|
| 43 |
+
|
| 44 |
+
Accept length = mean accepted tokens per draft block (7-token blocks, greedy).
|
| 45 |
+
Greedy losslessness: 5/5 prompts identical to the un-drafted target output.
|
| 46 |
+
|
| 47 |
+
| drafter | target | accept_len | draft rate |
|
| 48 |
+
|---|---|---|---|
|
| 49 |
+
| scratch | Arete-27B-Uncensored | 2.03 | 14.7% |
|
| 50 |
+
| scratch | Arete-27B (censored) | 1.91 | 13.1% |
|
| 51 |
+
| scratch | Qwen3.8-27B | 1.78 | 11.2% |
|
| 52 |
+
| **incoai** | **Arete-27B-Uncensored** | **3.00** | **28.6%** |
|
| 53 |
+
| **incoai** | **Arete-27B (censored)** | **2.97** | **28.2%** |
|
| 54 |
+
| **incoai** | **Qwen3.8-27B** | **3.80** | **40.0%** |
|
| 55 |
+
|
| 56 |
+
The incoai drafter (Z-Lab base finetuned on our regen data) is the strongest
|
| 57 |
+
combo everywhere — including on the base Qwen3.8-27B, supporting cross-model
|
| 58 |
+
transfer of DFlash2 drafters.
|
| 59 |
+
|
| 60 |
+
## Usage (vLLM)
|
| 61 |
+
|
| 62 |
+
```bash
|
| 63 |
+
vllm serve <target-model> \
|
| 64 |
+
--speculative-config '{"method": "dflash", "model": "SargeDev/arete-dflash2-incoai", "num_speculative_tokens": 7}'
|
| 65 |
+
```
|
| 66 |
+
|
| 67 |
+
## Training data
|
| 68 |
+
|
| 69 |
+
6,903 rows generated on-policy from Arete-27B-Uncensored (loop rows
|
| 70 |
+
quarantined), hidden states extracted from layers 6/20/34/48/62/64 of the
|
| 71 |
+
target. Corpus: `SargeDev/jev-distill-corpus-v3` lineage.
|
| 72 |
+
|
| 73 |
+
## Provenance
|
| 74 |
+
|
| 75 |
+
- Z-Lab drafter base: [z-lab/Qwen3.8-27B-DFlash2](https://huggingface.co/z-lab/Qwen3.8-27B-DFlash2)
|
| 76 |
+
- incoai finetuned variant: [incoai/Qwen3.8-27B-DFlash2](https://huggingface.co/incoai/Qwen3.8-27B-DFlash2)
|
| 77 |
+
- Target: [SargeDev/Arete-27B-Uncensored-V2](https://huggingface.co/SargeDev/Arete-27B-Uncensored-V2) (our uncensored build of [NousResearch/Arete-27B](https://huggingface.co/NousResearch/Arete-27B))
|
| 78 |
+
- Training corpus: `SargeDev/jev-distill-corpus-v3`
|
| 79 |
+
- Trainer: [vllm-project/speculators](https://github.com/vllm-project/speculators) 0.9.0a0, DFlash2 (dflash2) speculator type
|
| 80 |
+
|
| 81 |
+
## License
|
| 82 |
+
|
| 83 |
+
Apache-2.0 (base drafter). Arete-27B is cc-by-nc-4.0; the uncensored
|
| 84 |
+
SargeDev build follows the same license as its base.
|