---
datasets:
- belumind/en-vi-ja-curated-500k-triplets
language:
- en
- vi
- ja
library_name: pytorch
license: mit
pipeline_tag: translation
tags:
- anlp
- m26-assignment2
- part1
- mixture-of-experts
- variant2
---

# anlp-a2-MoE-var2

Advanced NLP (Monsoon 2026), Assignment 2.

Part 1, FFN variant 2. 8.8M total / 6.5M active params, trained on 120000840 tokens. Best checkpoint by validation loss; test loss 2.125, ppl 8.376, acc 0.614, bleu 21.893.

## Run

| item | value |
|---|---|
| parameters | 8.8M |
| training step | 89078 |
| wandb | https://wandb.ai/winterdew-iiit-hyderabad/anlp-a2-part1/runs/t2ck2i7t |

## Hyperparameters

| setting | value |
|---|---|
| FFN variant | 2 — 4 experts, 1 active per token, total-param matched |
| params (total / active) | 8.8M / 6.5M |
| FFN hidden width | 256 per expert |
| MoE aux weight | 0.01 |
| d_model / n_layers / n_heads | 256 / 6 / 8 |
| context length | 256 |
| positional encoding | RoPE (base 10000) |
| tokenizer | sentencepiece unigram, vocab 16000, character coverage 0.9995, byte fallback |
| optimizer | AdamW(betas=(0.9, 0.95), eps=1e-8) |
| learning rate | 0.0003 |
| lr schedule | linear warmup 1740 steps, cosine to 0.1x |
| weight decay | 0.1 (matrices only) |
| gradient clip | 1.0 |
| batch size | 32 x 1 accum |
| dropout | 0.1 |
| precision | fp16 |
| token budget | 120,000,000 (120,000,840 seen) |
| seed | 26 |

## Model config

```json
{
  "vocab_size": 16000,
  "n_ctx": 256,
  "rope_base": 10000.0,
  "d_model": 256,
  "n_heads": 8,
  "n_layers": 6,
  "d_ff": 1024,
  "dropout": 0.1,
  "tie_embeddings": true,
  "use_moe": true,
  "n_experts": 4,
  "n_active": 1,
  "n_shared": 0,
  "match_active": false,
  "moe_aux_weight": 0.01
}
```