ANLP Assignment 2 (part1): moe_4e2a
Decoder-only transformer with Mixture-of-Experts feed-forward, 4 experts, top-2 routing.
Trained from scratch for IIIT-H Advanced NLP, Monsoon 2026. Architecture, training
code and evaluation live in the assignment repository. This repository holds the
checkpoint and the tokenizer it was trained with, which every model in the assignment
shares.
Architecture
| Setting |
Value |
d_model |
512 |
n_layers |
8 |
n_heads |
8 |
n_kv_heads |
8 |
n_ctx |
256 |
vocab_size |
32000 |
ffn |
moe_4e2a |
d_ff |
512 |
n_routed_experts |
4 |
n_shared_experts |
0 |
top_k |
2 |
norm |
rmsnorm |
Parameters
| Count |
Value |
| Total |
41,607,680 |
| Active per token |
33,202,688 |
| Feed-forward (total / active) |
16,809,984 / 8,404,992 |
Test set results
| Metric |
Value |
| Perplexity (both directions) |
9.85 |
| BLEU vi->en |
33.61 |
| BLEU ja->en |
24.51 |
| BLEU mean |
29.06 |
Training
- Scored target tokens: 33,393,292
- Best validation loss: 2.0348
- Best validation perplexity: 7.65
Loading
import torch
payload = torch.load("model.pt", map_location="cpu", weights_only=False)
state_dict, config = payload["model"], payload["config"]
from tokenizers import Tokenizer
tokenizer = Tokenizer.from_file("tokenizer.json")