| --- |
| datasets: |
| - browndw/human-ai-parallel-corpus |
| language: |
| - en |
| library_name: pytorch |
| license: mit |
| pipeline_tag: text-generation |
| tags: |
| - anlp |
| - m26-assignment2 |
| - part2 |
| - optimizers |
| - mars |
| --- |
| |
| # anlp-a2-optimizers-mars |
|
|
| Advanced NLP (Monsoon 2026), Assignment 2. |
|
|
| Part 2, optimizer **mars**. 10.4M params, trained on 42090496 tokens = 0.33x Chinchilla. Best checkpoint by validation loss; test loss 4.437, ppl 84.507, acc 0.273, bleu 0.592. |
|
|
| ## Run |
|
|
| | item | value | |
| |---|---| |
| | parameters | 10.4M | |
| | training step | 5138 | |
| | wandb | https://wandb.ai/winterdew-iiit-hyderabad/anlp-a2-part2/runs/3kigbqvb | |
|
|
| ## Hyperparameters |
|
|
| | setting | value | |
| |---|---| |
| | optimizer | Mars | |
| | max_grad_norm | 1.0 | |
| | weight_decay | 0.1, 0.0 | |
| | betas | (0.9, 0.95) | |
| | gamma | 0.025 | |
| | eps | 1e-08 | |
| | lr | 0.0003 | |
| | param_groups | 2 | |
| | trainable_params | 10391808 | |
| | schedule | linear warmup 2% then cosine to 0% of peak | |
| | batch_size | 32 | |
| | grad_accum | 1 | |
| | tokens_per_step | 8192 | |
| | n_ctx | 256 | |
| | total_steps | 5137 | |
| | warmup_steps | 102 | |
| | trained_tokens | 42090496 | |
| | chinchilla_ratio | 0.3342 | |
| | precision | fp16 | |
| | seed | 26 | |
|
|
| ## Model config |
|
|
| ```json |
| { |
| "vocab_size": 16000, |
| "n_ctx": 256, |
| "rope_base": 10000.0, |
| "d_model": 256, |
| "n_heads": 8, |
| "n_layers": 8, |
| "d_ff": 1024, |
| "dropout": 0.1, |
| "tie_embeddings": true, |
| "use_moe": false, |
| "n_experts": 4, |
| "n_active": 1, |
| "n_shared": 0, |
| "match_active": false, |
| "moe_aux_weight": 0.01 |
| } |
| ``` |
|
|
