parth12-ui/anlp-a2-optim-lion

ANLP Assignment 2, Part 2: lion

Dense decoder-only transformer (33.4M parameters, 8 layers, d_model 512, context 256) pretrained from scratch for next-token prediction on browndw/human-ai-parallel-corpus, for 1x the dataset (39,075,840 tokens), using a from-scratch lion optimizer (category: memory-efficient).

Optimizer settings: lr=0.0002, betas=[0.9, 0.99], weight_decay=0.3.

metric at 1x dataset	value
validation loss	4.1811
validation perplexity	65.44
test BLEU (greedy 64-token continuation)	1.10

train_log.jsonl holds validation loss and test BLEU every 0.1x dataset tokens.

Loading

import json
from safetensors.torch import load_model
from model_src.config import TransformerConfig
from model_src.model import Transformer
model = Transformer(TransformerConfig.from_dict(json.load(open("config.json"))))
load_model(model, "model.safetensors")