TalosDDM upload

# TalosDDM

**Talos Dynamic Data Model (TalosDDM)** is a lightweight experimental language-model architecture developed as part of the Talos project.

## Model Data

| Parameter | Value |
| ----------------------- | --------------------: |
| Architecture | TalosGPT |
| Parameters | 254,272 |
| Hidden Size | 64 |
| Layers | 2 |
| Attention Heads | 4 |
| KV Heads | 2 |
| Vocabulary Size | 1,024 |
| Maximum Sequence Length | 512 |
| Attention Backend | PlainAttentionBackend |
| Precision | FP32 |
| Initial Training Device | NVIDIA Tesla T4 |
| Framework | PyTorch |
| PyTorch Version | 2.11.0+cu128 |

## Training

Initial training used a small experimental dataset containing short Talos/AI-related text.

Observed training loss:

| Step | Loss |
| ---: | -----: |
| 0 | 6.9177 |
| 25 | 5.3568 |
| 50 | 3.9134 |
| 75 | 2.6434 |
| 100 | 1.6195 |
| 125 | 1.0475 |
| 150 | 0.5186 |
| 175 | 0.2652 |

A separate CUDA smoke test produced:

```text
Step 0: 6.9209
Step 1: 6.9158
Step 2: 6.8067
```

## Experimental MoE Configuration

The repository also contains an experimental Mixture-of-Experts code path:

| Parameter | Value |
| --------------- | ----: |
| Hidden Size | 128 |
| Layers | 3 |
| Attention Heads | 8 |
| KV Heads | 4 |
| Head Dimension | 16 |
| Vocabulary Size | 2,048 |
| Total Experts | 16 |
| Active Experts | 3 |
| Shared Experts | 1 |

The MoE configuration is currently a code-path experiment and is not the primary 254K-parameter model configuration.

## Runtime

The model has been tested on CUDA and CPU environments.

CUDA test environment:

```text
GPU: NVIDIA Tesla T4
CUDA: Available
CUDA Toolkit: 12.8.1
PyTorch: 2.11.0+cu128
```

CPU test environment:

```text
Architecture: ARM64
CPU cores: 8
PyTorch: 2.14.0+cpu
Device: CPU
```

## Status

**Experimental / Research**

Current architecture scale: **254K parameters**

Primary development areas:

* End-to-end training
* Inference validation
* KV-cache validation
* Training throughput measurements
* Prefill/decode latency
* Memory usage
* Parameter and active-parameter measurements
* Quantization experiments
* Scaling toward ~1M parameters

talos-main.zip

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3e1863068867af9e10fbe0325548d01cac6264a6a6f4c49b941d74a3eda68a5e
+size 297704