skimo2022/anlp-a2-p2-mars

Advanced NLP Assignment 2, Part 2: mars.

Decoder-only Transformer (Part-1 dense architecture) pretrained on browndw/human-ai-parallel-corpus with a from-scratch optimizer: mars.

train_summary.json

{
 "train_time_s": 1106.750611782074,
 "final": {
  "step": 2385,
  "tokens": 39075840,
  "frac": 1.0,
  "val_loss": 3.7112768657135464,
  "val_ppl": 40.9060046697299,
  "bleu": 3.0067028339103503,
  "bleu_human_ref": 0.8166480937413516,
  "eval_time_s": 12.587995290756226
 },
 "params": {
  "total": 27269632,
  "active": 27269632,
  "non_embedding_total": 18881024,
  "non_embedding_active": 18881024
 },
 "budget_tokens": 39068808,
 "peak_mem_gb": 6.795574784
}

Load with Transformer.load(<local snapshot dir>) from src/common/model.py of the course repo.