| license: mit | |
| datasets: | |
| - Amirjalaly/ooast_prompts | |
| language: | |
| - en | |
| tags: | |
| - DDM | |
| - lightwieght | |
| # TalosDDM | |
| 254,272 parameters | |
| 64 hidden size | |
| 2 layers | |
| 4 attention heads | |
| 2 KV heads | |
| 1,024 vocabulary | |
| 512 max sequence length | |
| FP32 | |
| PyTorch 2.11.0+cu128 | |
| CUDA 12.8.1 | |
| NVIDIA Tesla T4 | |
| ### Training Loss | |
| Step 0 — 6.9177 | |
| Step 25 — 5.3568 | |
| Step 50 — 3.9134 | |
| Step 75 — 2.6434 | |
| Step 100 — 1.6195 | |
| Step 125 — 1.0475 | |
| Step 150 — 0.5186 | |
| Step 175 — 0.2652 | |
| ### MoE Configuration | |
| 128 hidden size | |
| 3 layers | |
| 8 attention heads | |
| 4 KV heads | |
| 16 head dimension | |
| 2,048 vocabulary | |
| 16 experts | |
| 3 active experts | |
| 1 shared expert | |