add model card

README.md

@@ -1,60 +1,39 @@
 ---
-license: apache-2.0
-language:
-- code
-library_name: transformers
 tags:
 - masked-diffusion
-- code-generation
+- diffusion
 - qwen2
+- code
+language:
+- en
+pipeline_tag: text-generation
 ---
 
-## thunder-dlm-0.6b — Masked Diffusion Model for Code Generation
+# dlm-code0.6b-exp
 
-This repository contains the weights and custom code for the **staticlabs/thunder-dlm-0.6b** model, a masked diffusion model for code generation based on the Qwen2 architecture.
+**Experimental** masked-diffusion language model for code, from the [thunder-fast](https://github.com/AdrianTuci1/thunder-fast) project.
 
-This model uses bidirectional attention and must be used with the custom `diffusion_generate` method.
+This is a **discrete masked-diffusion (MDM)** adaptation of a Qwen2-based pretrained code model (~0.5B). It is *not* an autoregressive next-token model: generation is a progressive un-masking loop under **bidirectional attention**, so a whole output window is filled in parallel over 24 diffusion steps.
 
-## How to Use
+## Details
 
-First, make sure you have the latest `transformers` library installed.
+| | |
+|---|---|
+| Backbone | Qwen2 (`model_type: qwen2`), 24 layers, hidden 896, 14 attention / 2 KV heads, GQA |
+| Vocabulary | 151,936 + `<M>` mask token (id 151665) |
+| Diffusion | discrete `[MASK]`, 24-step progressive un-masking, entropy-based position confidence (`alg_temp` 0.6, `top_k` 500) |
+| Context | window-based (256 tokens) — not left-to-right; supports infilling |
+| Weights | bf16 (`model.safetensors`) |
 
-```bash
-pip install transformers torch huggingface_hub
-```
+## How to run
 
-You can then use the model for generation. Note: You must pass trust_remote_code=True to load the custom model architecture.
-```python
-import torch
-from transformers import AutoTokenizer, AutoModelForCausalLM
-
-model_id = "staticlabs/thunder-dlm-0.6b"
-device = "cuda" if torch.cuda.is_available() else "cpu"
-
-# trust_remote_code=True is essential
-tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
-model = AutoModelForCausalLM.from_pretrained(
-    model_id,
-    torch_dtype=torch.bfloat16,
-    trust_remote_code=True
-).to(device)
-
-prompt = "def fibonacci(n):"
-input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(device)
-
-# The model will use the generation_config.json from the repo by default
-# You can also override parameters here
-outputs = model.diffusion_generate(
-    inputs=input_ids,
-    max_new_tokens=100,
-    steps=16,
-    temperature=0.8
-)
-
-# Decode the output
-prompt_len = input_ids.shape[1]
-generated_text = tokenizer.decode(outputs.sequences[0][prompt_len:], skip_special_tokens=True)
-
-print("--- Generated Code ---")
-print(generated_text)
+This checkpoint is intended to be driven by a **diffusion decoder** (bidirectional attention + the iterative un-masking loop), e.g. the thunder-fast runtime. Loading it as a plain causal `Qwen2ForCausalLM` will *not* reproduce diffusion generation.
+
+```
+# thunder-fast runtime (ggml engine) or the reference DiffusionLM
+# e.g. runtime --model dlm-code0.6b-exp.gguf --prompt "..."
 ```
+
+## Status
+
+Experimental / work-in-progress (`.exp`). It is a diffusion code-generation checkpoint published for the thunder-fast project; see the repo for training, the masking schedule, and evaluation details.