-
Update to files
# TalosDDM
254,272 parameters
64 hidden size
2 layers
4 attention heads
2 KV heads
1,024 vocabulary
512 max sequence length
FP32
PyTorch 2.11.0+cu128
CUDA 12.8.1
NVIDIA Tesla T4
### Training Loss
Step 0 — 6.9177
Step 25 — 5.3568
Step 50 — 3.9134
Step 75 — 2.6434
Step 100 — 1.6195
Step 125 — 1.0475
Step 150 — 0.5186
Step 175 — 0.2652
### MoE Configuration
128 hidden size
3 layers
8 attention heads
4 KV heads
16 head dimension
2,048 vocabulary
16 experts
3 active experts
1 shared expert -
TalosDDM upload
# TalosDDM
**Talos Dynamic Data Model (TalosDDM)** is a lightweight experimental language-model architecture developed as part of the Talos project.
## Model Data
| Parameter | Value |
| ----------------------- | --------------------: |
| Architecture | TalosGPT |
| Parameters | 254,272 |
| Hidden Size | 64 |
| Layers | 2 |
| Attention Heads | 4 |
| KV Heads | 2 |
| Vocabulary Size | 1,024 |
| Maximum Sequence Length | 512 |
| Attention Backend | PlainAttentionBackend |
| Precision | FP32 |
| Initial Training Device | NVIDIA Tesla T4 |
| Framework | PyTorch |
| PyTorch Version | 2.11.0+cu128 |
## Training
Initial training used a small experiment… -
initial commit
