| ---
|
| library_name: pytorch
|
| tags:
|
| - anlp-a2
|
| - mixture-of-experts
|
| - translation
|
| language:
|
| - vi
|
| - ja
|
| - en
|
| datasets:
|
| - belumind/en-vi-ja-curated-500k-triplets
|
| ---
|
|
|
| # a2-task1-v4_moe_shared
|
|
|
| ANLP Assignment 2, Task 1 (Mixture of Experts): Vietnamese/Japanese to English translation, trained from scratch on `belumind/en-vi-ja-curated-500k-triplets`.
|
|
|
| **FFN variant:** MoE, 1 always-on shared expert + 3 routed experts (width 512), top-1 routed. Total params matched to the dense model.
|
|
|
| Decoder-only transformer: d_model 512, 8 layers, 8 heads, context 256, learned absolute positions, pre-LayerNorm, bias-free GELU FFN, 32k BPE vocabulary with the embedding tied to the LM head.
|
|
|
| | total params | active params | train tokens | test ppl | ppl vi->en | ppl ja->en | BLEU | BLEU vi->en | BLEU ja->en | chrF |
|
| |---|---|---|---|---|---|---|---|---|---|
|
| | 41,710,592 | 33,321,984 | 73.0M | 4.620 | 3.946 | 5.410 | 38.72 | 43.60 | 33.77 | 60.50 |
|
|
|
| Input layout: `[bos] <2en> source [sep] target [eos]` (loss on the target only); greedy decoding.
|
|
|
| Files: `model.pt` (a `torch.save` dict: `model` state_dict, `step`, `tokens_seen`, `config`), `tokenizer.json` (HF `tokenizers`), `config.yaml` (training config), `metrics.json` (full eval output).
|
|
|
| Loading (with the assignment code on the path):
|
|
|
| ```python
|
| import torch
|
| from src.part1.config import TransformerConfig, resolve_ffn_dims
|
| from src.part1.model import build_model
|
|
|
| ckpt = torch.load('model.pt', map_location='cpu', weights_only=False)
|
| model = build_model(resolve_ffn_dims(TransformerConfig(**ckpt['config'])))
|
| model.load_state_dict(ckpt['model'])
|
| ```
|
|
|
