parth12-ui/anlp-a2-optim-mars

ANLP Assignment 2, Part 2: mars

Dense decoder-only transformer (33.4M parameters, 8 layers, d_model 512, context 256) pretrained from scratch for next-token prediction on browndw/human-ai-parallel-corpus, for 1x the dataset (39,075,840 tokens), using a from-scratch mars optimizer (category: variance-reduced AdamW).

Optimizer settings: lr=0.001, betas=[0.95, 0.99], eps=1e-08, gamma=0.025, weight_decay=0.1.

metric at 1x dataset	value
validation loss	3.9895
validation perplexity	54.03
test BLEU (greedy 64-token continuation)	1.38

train_log.jsonl holds validation loss and test BLEU every 0.1x dataset tokens.

Loading

import json
from safetensors.torch import load_model
from model_src.config import TransformerConfig
from model_src.model import Transformer
model = Transformer(TransformerConfig.from_dict(json.load(open("config.json"))))
load_model(model, "model.safetensors")