File size: 610 Bytes
f165395 68bfd12 f165395 68bfd12 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 | ---
license: mit
datasets:
- Amirjalaly/ooast_prompts
language:
- en
tags:
- DDM
- lightwieght
---
# TalosDDM
254,272 parameters
64 hidden size
2 layers
4 attention heads
2 KV heads
1,024 vocabulary
512 max sequence length
FP32
PyTorch 2.11.0+cu128
CUDA 12.8.1
NVIDIA Tesla T4
### Training Loss
Step 0 — 6.9177
Step 25 — 5.3568
Step 50 — 3.9134
Step 75 — 2.6434
Step 100 — 1.6195
Step 125 — 1.0475
Step 150 — 0.5186
Step 175 — 0.2652
### MoE Configuration
128 hidden size
3 layers
8 attention heads
4 KV heads
16 head dimension
2,048 vocabulary
16 experts
3 active experts
1 shared expert
|