Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -1,3 +1,19 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
# tiny-dllm
|
| 2 |
|
| 3 |
A Masked Diffusion Language Model built from scratch in PyTorch — for learning and robotics research.
|
|
@@ -130,17 +146,6 @@ Logits [B, T, vocab_size]
|
|
| 130 |
|
| 131 |
Both trained from scratch on TinyShakespeare with identical model size and steps.
|
| 132 |
|
| 133 |
-
## Trained Checkpoint
|
| 134 |
-
|
| 135 |
-
A checkpoint trained to 50,000 steps on TinyShakespeare is available on HuggingFace:
|
| 136 |
-
|
| 137 |
-
```bash
|
| 138 |
-
from huggingface_hub import hf_hub_download
|
| 139 |
-
path = hf_hub_download(repo_id="sutharsan311/tiny-dllm", filename="dllm_step50000.pt")
|
| 140 |
-
```
|
| 141 |
-
|
| 142 |
-
Hardware: NVIDIA RTX 3050 (4GB VRAM) — ~2 hours training time.
|
| 143 |
-
|
| 144 |
## Roadmap
|
| 145 |
|
| 146 |
- [x] Character-level tokenizer
|
|
@@ -150,7 +155,7 @@ Hardware: NVIDIA RTX 3050 (4GB VRAM) — ~2 hours training time.
|
|
| 150 |
- [x] Tamil Unicode tokenizer
|
| 151 |
- [x] Train on Thirukkural + Sangam poetry
|
| 152 |
- [x] Tamil Wikipedia downloader (API + full dump)
|
| 153 |
-
- [
|
| 154 |
- [ ] GPT baseline comparison (dLLM vs GPT on same data)
|
| 155 |
- [ ] Blog post: dLLM vs GPT on TinyShakespeare
|
| 156 |
- [ ] Fill-in-the-blanks (conditional generation)
|
|
|
|
| 1 |
+
---
|
| 2 |
+
language:
|
| 3 |
+
- en
|
| 4 |
+
- ta
|
| 5 |
+
license: mit
|
| 6 |
+
tags:
|
| 7 |
+
- diffusion
|
| 8 |
+
- language-model
|
| 9 |
+
- pytorch
|
| 10 |
+
- shakespeare
|
| 11 |
+
- tamil
|
| 12 |
+
- nlp
|
| 13 |
+
- from-scratch
|
| 14 |
+
library_name: pytorch
|
| 15 |
+
---
|
| 16 |
+
|
| 17 |
# tiny-dllm
|
| 18 |
|
| 19 |
A Masked Diffusion Language Model built from scratch in PyTorch — for learning and robotics research.
|
|
|
|
| 146 |
|
| 147 |
Both trained from scratch on TinyShakespeare with identical model size and steps.
|
| 148 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 149 |
## Roadmap
|
| 150 |
|
| 151 |
- [x] Character-level tokenizer
|
|
|
|
| 155 |
- [x] Tamil Unicode tokenizer
|
| 156 |
- [x] Train on Thirukkural + Sangam poetry
|
| 157 |
- [x] Tamil Wikipedia downloader (API + full dump)
|
| 158 |
+
- [ ] Train to 50k steps
|
| 159 |
- [ ] GPT baseline comparison (dLLM vs GPT on same data)
|
| 160 |
- [ ] Blog post: dLLM vs GPT on TinyShakespeare
|
| 161 |
- [ ] Fill-in-the-blanks (conditional generation)
|