ANLP Assignment 3 models (2025801013)
Checkpoints for the two parts of the assignment.
- Part 1, cipher decryption. Language models written from scratch in plain PyTorch that read a ciphertext and write its plaintext: a Transformer, Mamba, DeltaNet and a Mamba-DeltaNet hybrid.
- Part 2, fake news detection. Graph neural networks (Root-Only MLP, GCN, GraphSAGE, GAT and Bi-GCN) that classify UPFD GossipCop propagation graphs.
The checkpoints hold weights and settings only. The model code is in the assignment's code repository, and loading a checkpoint needs that code (see How to use). The training logs are on WandB: 2025801013-anlp-assignment3.
Files
| Path | What it is |
|---|---|
part1/release_2026_11_11/<run>/best.pt |
Part 1 runs trained on the second data release (2026-11-11). These are the default. |
part1/<run>/best.pt |
The same runs on the first data release, earlier first-release runs (transformer, 20,000 steps; transformer_characters; mamba_short), and the runs on a synthetic substitution cipher (control_*, ablation_*). |
part1/with_decay_on_a/<run>/best.pt |
Earlier Mamba and hybrid runs, from before weight decay was removed from Mamba's A. Kept for comparison only. |
part1/vocabulary.json |
Vocabulary for runs with subword output (1,000 BPE merges on the plaintext). |
part1/vocabulary_characters.json |
Vocabulary for runs with character output (names ending in _characters). |
part2/<model>_<features>/seed_<n>.pt |
Part 2 runs, 5 seeds (0 to 4) for each model and feature set. |
Each best.pt in Part 1 is the checkpoint with the lowest validation loss. Each Part 2 checkpoint is the epoch with the best validation F1.
Part 1: cipher decryption
Task. The plaintext is lowercase English with spaces, 27 characters. The ciphertext uses 81 symbols: 63 main symbols and 18 noise symbols that carry no character. Each model reads the cipher of a window of 512 plaintext characters, then writes the plaintext one token at a time. Decoding is greedy, and it writes exactly one character per real cipher symbol.
Models. About 5 million parameters each, all trained for the same budget of 2,500 steps of about 16,384 tokens. Each architecture has a run with subword output (<arch>) and one with character output (<arch>_characters).
| Run | Architecture | Parameters | Validation char. accuracy | Test char. accuracy |
|---|---|---|---|---|
transformer_common |
Transformer, 6 layers | 5,711,872 | 7.47% | 7.53% |
transformer_common_characters |
Transformer, 6 layers | 5,199,872 | 9.64% | 9.59% |
mamba |
Mamba, 12 layers | 5,819,648 | 8.42% | 8.38% |
mamba_characters |
Mamba, 12 layers | 5,307,648 | 8.14% | 8.11% |
deltanet |
DeltaNet | 5,342,720 | 8.36% | 8.45% |
deltanet_characters |
DeltaNet | 4,830,720 | 7.97% | 8.00% |
hybrid |
Mamba and DeltaNet blocks, alternating | 5,501,696 | 8.53% | not evaluated yet |
hybrid_characters |
Mamba and DeltaNet blocks, alternating | 4,989,696 | 8.36% | not evaluated yet |
hybrid_mamba_then_deltanet |
Mamba blocks, then DeltaNet blocks | 5,501,696 | 8.38% | not evaluated yet |
hybrid_mamba_then_deltanet_characters |
Mamba blocks, then DeltaNet blocks | 4,989,696 | 8.89% | not evaluated yet |
Second data release, 1,153 validation and 1,318 test texts. Character accuracy is the share of positions where the predicted character equals the reference.
Default model. part1/release_2026_11_11/transformer_common_characters/best.pt, the best run on validation. The assignment's entry point downloads it from here by default.
Limitations. These models do not decrypt the cipher. Every run scores below a baseline that ignores the cipher and writes only spaces (16.87% on test). On the second release the models use the cipher slightly: shuffling the cipher symbols raises their loss by +0.0005 to +0.017 per token, with z from 3.4 to 18.0. On the first release this effect was not distinguishable from zero.
Memory and speed. Decoding the test split with the default model peaked at 1.17 GB of GPU memory allocated (7.93 GB reserved by PyTorch's cache), at about 3.2 s per text on a laptop RTX 5050. The Transformer keeps no cache of earlier computations. Mamba with subwords decodes at about 0.2 s per text.
Part 2: fake news detection on UPFD GossipCop
Each model classifies a news propagation graph as fake or real. Settings: 2 layers, hidden size 128, max pooling, AdamW (learning rate 0.001, weight decay 0.01), 200 epochs, batch 128. GAT uses 4 heads; Bi-GCN uses 20% edge drop.
Test results, mean ± standard deviation over 5 seeds, on 3,826 test graphs:
| Checkpoint folder | Model | Features | Parameters | Accuracy | F1 |
|---|---|---|---|---|---|
mlp_bert |
Root-Only MLP | bert | 115,202 | 0.7301 ± 0.0093 | 0.7334 ± 0.0034 |
gcn_bert |
GCN | bert | 115,202 | 0.9551 ± 0.0040 | 0.9553 ± 0.0038 |
sage_bert |
GraphSAGE | bert | 229,890 | 0.9660 ± 0.0019 | 0.9661 ± 0.0020 |
gat_bert |
GAT | bert | 115,714 | 0.9671 ± 0.0069 | 0.9672 ± 0.0069 |
bigcn_bert |
Bi-GCN | bert | 427,522 | 0.9428 ± 0.0106 | 0.9433 ± 0.0105 |
sage_profile |
GraphSAGE | profile | 35,842 | 0.9478 ± 0.0022 | 0.9484 ± 0.0021 |
sage_spacy |
GraphSAGE | spacy | 110,082 | 0.9612 ± 0.0023 | 0.9613 ± 0.0025 |
sage_content |
GraphSAGE | content | 112,642 | 0.9780 ± 0.0024 | 0.9781 ± 0.0023 |
How to use
The commands below run from the root of the assignment's code repository, after uv sync.
With the entry points
bash scripts/part1/test_assn.sh --heldout
bash scripts/part2/test_gnn.sh --ta
test_assn.sh --heldoutreadsdata/part1/heldout/cipher.jsonl, one{"id", "ciphertext"}per line. It downloads the default model from this repository and writesoutput/part1/heldout_result_2025801013.jsonl, one{"id", "text"}per line in input order.--split validationor--split testdecodes a split instead.--config config/part1/<run>.yamlpicks another run, and--oldpicks its first-release checkpoint.--localusesmodels/part1/...on disk instead of this repository.
test_gnn.sh --tascores all 40 Part 2 checkpoints. It usesmodels/part2if present and downloads from this repository otherwise, then writesoutput/part2/results_ta.json.
In Python, Part 1
Save this in the root of the code repository and run it with PYTHONPATH=. uv run python example.py.
import torch
from huggingface_hub import hf_hub_download
from src.cipher.data.cipher_windower import CipherWindower
from src.cipher.inference.generator_factory import GeneratorFactory
from src.cipher.inference.length_rules import LengthRuleFactory
from src.cipher.inference.window_decryptor import WindowDecryptor
from src.cipher.models.checkpoint_loader import CheckpointLoader
from src.cipher.tokenization.vocabulary import Vocabulary
repository = "samvaran/2025801013-anlp-a3"
checkpoint_path = hf_hub_download(repository, "part1/release_2026_11_11/transformer_common_characters/best.pt")
vocabulary_path = hf_hub_download(repository, "part1/vocabulary_characters.json")
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
vocabulary = Vocabulary.load(vocabulary_path)
model = CheckpointLoader.load(checkpoint_path, vocabulary, device)
length_rule = LengthRuleFactory.create(LengthRuleFactory.DEFAULT, vocabulary, device)
generator = GeneratorFactory.create(model, vocabulary, length_rule, 1600, 24000, device)
decryptor = WindowDecryptor(generator, vocabulary, CipherWindower(512))
ciphers = ["<one ciphertext per string>"]
print(decryptor.decrypt(ciphers))
Use part1/vocabulary.json for runs without _characters in their name. 1600 is the model's maximum length, 24000 the tokens per decoding batch, and 512 the window size, as in the run configs.
In Python, Part 2
This needs the UPFD data under data/part2/upfd, which torch_geometric downloads on first use.
import torch
from huggingface_hub import hf_hub_download
from torch_geometric.datasets import UPFD
from torch_geometric.loader import DataLoader
from src.gnn.models.model_config import ModelConfig
from src.gnn.models.model_factory import ModelFactory
checkpoint_path = hf_hub_download("samvaran/2025801013-anlp-a3", "part2/sage_bert/seed_0.pt")
checkpoint = torch.load(checkpoint_path, map_location="cpu")
test_graphs = UPFD("data/part2/upfd", "gossipcop", checkpoint["feature_type"], "test")
model = ModelFactory.create(checkpoint["model_kind"], test_graphs.num_features, ModelConfig(**checkpoint["settings"]["model"]))
model.load_state_dict(checkpoint["model_state"])
model.eval()
correct = 0
with torch.no_grad():
for graph_batch in DataLoader(test_graphs, batch_size=128):
predicted = model(graph_batch).argmax(dim=1)
correct += int((predicted == graph_batch.y).sum())
print(f"test accuracy {correct / len(test_graphs):.4f}")
This prints test accuracy 0.9647, the saved score of that seed.
Checkpoint contents
- Part 1:
model_state(the weights),model_config(architecture settings),stepandvalidation_loss. - Part 2:
model_state,model_kind,feature_type,seed,best_epoch,best_validation_f1, andsettings(dataset, batch size, model and training settings).