anlp-a2-part1-moe_top2_active
Decoder-only transformer trained from scratch to translate Vietnamese and Japanese to English
(belumind/en-vi-ja-curated-500k-triplets). FFN variant: moe_top2_active.
- total params: 24,098,688 | active params per token: 17,020,800
- trained on 39,003,133 tokens
- W&B run: https://wandb.ai/dunkronit-iiit-hyderabad/anlp-a2-part1/runs/moe_top2_active-db95d986
Input format: <vi> source <en> or <ja> source <en>; the model continues with English and <eos>.
Load with src.part1.model.Transformer.from_pretrained("DunkRonit/anlp-a2-part1-moe_top2_active") from the assignment repo.
- Downloads last month
- 11