| --- |
| language: |
| - en |
| license: openrail |
| tags: |
| - diffusion-llm |
| - parallel-generation |
| - custom-transformer |
| - cropmark |
| datasets: |
| - OpenAssistant/oasst1 |
| metrics: |
| - cosine_similarity |
| base_model: |
| - darwinkernelpanic/DiffReaper-5 |
| --- |
| |
| # DiffReaper-5L |
|
|
| DiffReaper-5L is a **larger** version of DiffReaper-5, with **2048-dim embeddings** and a **24-layer Transformer**. |
|
|
| ## Model Details |
|
|
| - **Architecture:** 24-layer Custom Transformer with Time Embedding. |
| - **Task:** Conditioned Text Diffusion (Prompt-Response). |
| - **Training Objective:** Cosine Similarity Regression. |
| - **Sampling:** 10-step iterative parallel denoising. |
|
|
| ## Usage (Inference) |
|
|
| To run inference: |
|
|
| ```python |
| import torch |
| # Assuming DiffReaperModel is defined as in train_diffreaper_5l.py |
| |
| model = DiffReaperModel(vocab_size=50257, n_embd=2048, n_head=32, n_layer=24).to("cuda") |
| model.load_state_dict(torch.load("diffreaper5l_latest.pt")) |
| model.eval() |
| ``` |
|
|
| ## Fine-tuning |
|
|
| To fine-tune on a custom dataset, ensure your data loader provides **Prompt** + **Response** pairs. Use the same Cosine Similarity loss. |
