ANLP Assignment 2 (part1): moe_active_matched

Decoder-only transformer with Mixture-of-Experts, 4 experts, top-2 routing, active parameters matched to the dense baseline.

Trained from scratch for IIIT-H Advanced NLP, Monsoon 2026. Architecture, training code and evaluation live in the assignment repository. This repository holds the checkpoint and the tokenizer it was trained with, which every model in the assignment shares.

Architecture

Setting Value
d_model 512
n_layers 8
n_heads 8
n_kv_heads 8
n_ctx 256
vocab_size 32000
ffn moe_active_matched
d_ff 1024
n_routed_experts 4
n_shared_experts 0
top_k 2
norm rmsnorm

Parameters

Count Value
Total 58,401,280
Active per token 41,599,488
Feed-forward (total / active) 33,603,584 / 16,801,792

Test set results

Metric Value
Perplexity (both directions) 8.81
BLEU vi->en 35.06
BLEU ja->en 25.17
BLEU mean 30.12

Training

  • Scored target tokens: 33,393,292
  • Best validation loss: 1.9321
  • Best validation perplexity: 6.90

Loading

import torch
payload = torch.load("model.pt", map_location="cpu", weights_only=False)
state_dict, config = payload["model"], payload["config"]

from tokenizers import Tokenizer
tokenizer = Tokenizer.from_file("tokenizer.json")
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support