File size: 1,617 Bytes
849a848 249726d 849a848 249726d 849a848 249726d 849a848 249726d 849a848 249726d 849a848 249726d 849a848 249726d 849a848 249726d 849a848 249726d 849a848 249726d | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 | ---
tags:
- masked-diffusion
- diffusion
- qwen2
- code
language:
- en
pipeline_tag: text-generation
---
# dlm-code0.6b-exp
**Experimental** masked-diffusion language model for code, from the [thunder-fast](https://github.com/AdrianTuci1/thunder-fast) project.
This is a **discrete masked-diffusion (MDM)** adaptation of a Qwen2-based pretrained code model (~0.5B). It is *not* an autoregressive next-token model: generation is a progressive un-masking loop under **bidirectional attention**, so a whole output window is filled in parallel over 24 diffusion steps.
## Details
| | |
|---|---|
| Backbone | Qwen2 (`model_type: qwen2`), 24 layers, hidden 896, 14 attention / 2 KV heads, GQA |
| Vocabulary | 151,936 + `<M>` mask token (id 151665) |
| Diffusion | discrete `[MASK]`, 24-step progressive un-masking, entropy-based position confidence (`alg_temp` 0.6, `top_k` 500) |
| Context | window-based (256 tokens) — not left-to-right; supports infilling |
| Weights | bf16 (`model.safetensors`) |
## How to run
This checkpoint is intended to be driven by a **diffusion decoder** (bidirectional attention + the iterative un-masking loop), e.g. the thunder-fast runtime. Loading it as a plain causal `Qwen2ForCausalLM` will *not* reproduce diffusion generation.
```
# thunder-fast runtime (ggml engine) or the reference DiffusionLM
# e.g. runtime --model dlm-code0.6b-exp.gguf --prompt "..."
```
## Status
Experimental / work-in-progress (`.exp`). It is a diffusion code-generation checkpoint published for the thunder-fast project; see the repo for training, the masking schedule, and evaluation details.
|