---
library_name: talos
tags:
- talos
- causal-language-model
- custom-architecture
- small-language-model
---

# Talos 10M

Talos 10M is a custom causal language model developed by Promethean Studios.

## Model

- Parameters: 9,952,320
- Hidden size: 448
- Layers: 3
- Attention heads: 28
- KV heads: 14
- Head dimension: 16
- FFN intermediate size: 1,792
- Context length: 512 tokens
- Model vocabulary: 1,024
- Training tokenizer vocabulary: 512
- Architecture: TalosGPT
- Attention: Full attention
- RoPE theta: 10,000

## Training

- Training steps: 100,000
- Validation loss: 1.1523912596702575
- Batch size: 8
- Sequence length: 512
- Maximum learning rate: 3e-4
- Minimum learning rate: 3e-5
- Warmup steps: 1,000
- Weight decay: 0.1
- Seed: 1337

## Files

`talos_10m_step_100000.pt`
- PyTorch training checkpoint containing the trained Talos 10M weights.

`tokenizer.json`
- Talos-native ByteLevelBPE tokenizer used to produce the training tokens.

`config.json`
- Model and training configuration.

## Notes

Talos is a custom architecture and is not a Transformers-native model.

The model was trained with a 512-vocabulary tokenizer while the model architecture contains 1,024 vocabulary slots. Training data used token IDs through 509.