Part 2 optimizer checkpoints
Each optimizer folder contains saved PyTorch checkpoints, tokenizer, settings, recorded metrics, predictions, and plots. Exact model dimensions and dataset fractions are recorded separately in each settings.json.
These are custom PyTorch models. Load them with the included source code. The checkpoints are not in the Transformers AutoModel format.
import sys
from pathlib import Path
import torch
from huggingface_hub import snapshot_download
folder = Path(snapshot_download(repo_id="walke007/anlp-assignment2-part2"))
sys.path.insert(0, str(folder / "source"))
from part2.model import DecoderOnlyTransformer, TransformerConfig
from part2.data import load_tokenizer
checkpoint = torch.load(folder / 'muon' / "final.pt", map_location="cpu", weights_only=True)
model = DecoderOnlyTransformer(TransformerConfig(**checkpoint["config"]), False, 0, 0, False)
model.load_state_dict(checkpoint["model_state_dict"])
model.eval()
tokenizer = load_tokenizer(str(folder / 'muon' / "tokenizer"))
The final checkpoints include optimizer/scaler state for resuming training. The training data itself is not included. BLEU measures greedy held-out text continuation against its reference, as defined in each settings.json.
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐ Ask for provider support