File size: 1,469 Bytes
42c70da | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 | ---
datasets:
- browndw/human-ai-parallel-corpus
language:
- en
library_name: pytorch
license: mit
pipeline_tag: text-generation
tags:
- anlp
- m26-assignment2
- part2
- optimizers
- mars
---
# anlp-a2-optimizers-mars
Advanced NLP (Monsoon 2026), Assignment 2.
Part 2, optimizer **mars**. 10.4M params, trained on 42090496 tokens = 0.33x Chinchilla. Best checkpoint by validation loss; test loss 4.437, ppl 84.507, acc 0.273, bleu 0.592.
## Run
| item | value |
|---|---|
| parameters | 10.4M |
| training step | 5138 |
| wandb | https://wandb.ai/winterdew-iiit-hyderabad/anlp-a2-part2/runs/3kigbqvb |
## Hyperparameters
| setting | value |
|---|---|
| optimizer | Mars |
| max_grad_norm | 1.0 |
| weight_decay | 0.1, 0.0 |
| betas | (0.9, 0.95) |
| gamma | 0.025 |
| eps | 1e-08 |
| lr | 0.0003 |
| param_groups | 2 |
| trainable_params | 10391808 |
| schedule | linear warmup 2% then cosine to 0% of peak |
| batch_size | 32 |
| grad_accum | 1 |
| tokens_per_step | 8192 |
| n_ctx | 256 |
| total_steps | 5137 |
| warmup_steps | 102 |
| trained_tokens | 42090496 |
| chinchilla_ratio | 0.3342 |
| precision | fp16 |
| seed | 26 |
## Model config
```json
{
"vocab_size": 16000,
"n_ctx": 256,
"rope_base": 10000.0,
"d_model": 256,
"n_heads": 8,
"n_layers": 8,
"d_ff": 1024,
"dropout": 0.1,
"tie_embeddings": true,
"use_moe": false,
"n_experts": 4,
"n_active": 1,
"n_shared": 0,
"match_active": false,
"moe_aux_weight": 0.01
}
```
|