---
datasets:
- browndw/human-ai-parallel-corpus
language:
- en
library_name: pytorch
license: mit
pipeline_tag: text-generation
tags:
- anlp
- m26-assignment2
- part2
- optimizers
- lion
---

# anlp-a2-optimizers-lion

Advanced NLP (Monsoon 2026), Assignment 2.

Part 2, optimizer **lion**. 10.4M params, trained on 42090496 tokens = 0.33x Chinchilla. Best checkpoint by validation loss; test loss 4.682, ppl 107.952, acc 0.253, bleu 0.472.

## Run

| item | value |
|---|---|
| parameters | 10.4M |
| training step | 5138 |
| wandb | https://wandb.ai/winterdew-iiit-hyderabad/anlp-a2-part2/runs/5zxuma69 |

## Hyperparameters

| setting | value |
|---|---|
| optimizer | Lion |
| max_grad_norm | 1.0 |
| weight_decay | 1.0, 0.0 |
| betas | (0.9, 0.99) |
| lr | 3e-05 |
| param_groups | 2 |
| trainable_params | 10391808 |
| schedule | linear warmup 2% then cosine to 0% of peak |
| batch_size | 32 |
| grad_accum | 1 |
| tokens_per_step | 8192 |
| n_ctx | 256 |
| total_steps | 5137 |
| warmup_steps | 102 |
| trained_tokens | 42090496 |
| chinchilla_ratio | 0.3342 |
| precision | fp16 |
| seed | 26 |

## Model config

```json
{
  "vocab_size": 16000,
  "n_ctx": 256,
  "rope_base": 10000.0,
  "d_model": 256,
  "n_heads": 8,
  "n_layers": 8,
  "d_ff": 1024,
  "dropout": 0.1,
  "tie_embeddings": true,
  "use_moe": false,
  "n_experts": 4,
  "n_active": 1,
  "n_shared": 0,
  "match_active": false,
  "moe_aux_weight": 0.01
}
```