ANLP Assignment 3 models (2025801013)

Checkpoints for the two parts of the assignment.

  • Part 1, cipher decryption. Language models written from scratch in plain PyTorch that read a ciphertext and write its plaintext: a Transformer, Mamba, DeltaNet and a Mamba-DeltaNet hybrid.
  • Part 2, fake news detection. Graph neural networks (Root-Only MLP, GCN, GraphSAGE, GAT and Bi-GCN) that classify UPFD GossipCop propagation graphs.

The checkpoints hold weights and settings only. The model code is in the assignment's code repository, and loading a checkpoint needs that code (see How to use). The training logs are on WandB: 2025801013-anlp-assignment3.

Files

Path What it is
part1/release_2026_11_11/<run>/best.pt Part 1 runs trained on the second data release (2026-11-11). These are the default.
part1/<run>/best.pt The same runs on the first data release, earlier first-release runs (transformer, 20,000 steps; transformer_characters; mamba_short), and the runs on a synthetic substitution cipher (control_*, ablation_*).
part1/with_decay_on_a/<run>/best.pt Earlier Mamba and hybrid runs, from before weight decay was removed from Mamba's A. Kept for comparison only.
part1/vocabulary.json Vocabulary for runs with subword output (1,000 BPE merges on the plaintext).
part1/vocabulary_characters.json Vocabulary for runs with character output (names ending in _characters).
part2/<model>_<features>/seed_<n>.pt Part 2 runs, 5 seeds (0 to 4) for each model and feature set.

Each best.pt in Part 1 is the checkpoint with the lowest validation loss. Each Part 2 checkpoint is the epoch with the best validation F1.

Part 1: cipher decryption

Task. The plaintext is lowercase English with spaces, 27 characters. The ciphertext uses 81 symbols: 63 main symbols and 18 noise symbols that carry no character. Each model reads the cipher of a window of 512 plaintext characters, then writes the plaintext one token at a time. Decoding is greedy, and it writes exactly one character per real cipher symbol.

Models. About 5 million parameters each, all trained for the same budget of 2,500 steps of about 16,384 tokens. Each architecture has a run with subword output (<arch>) and one with character output (<arch>_characters).

Run Architecture Parameters Validation char. accuracy Test char. accuracy
transformer_common Transformer, 6 layers 5,711,872 7.47% 7.53%
transformer_common_characters Transformer, 6 layers 5,199,872 9.64% 9.59%
mamba Mamba, 12 layers 5,819,648 8.42% 8.38%
mamba_characters Mamba, 12 layers 5,307,648 8.14% 8.11%
deltanet DeltaNet 5,342,720 8.36% 8.45%
deltanet_characters DeltaNet 4,830,720 7.97% 8.00%
hybrid Mamba and DeltaNet blocks, alternating 5,501,696 8.53% not evaluated yet
hybrid_characters Mamba and DeltaNet blocks, alternating 4,989,696 8.36% not evaluated yet
hybrid_mamba_then_deltanet Mamba blocks, then DeltaNet blocks 5,501,696 8.38% not evaluated yet
hybrid_mamba_then_deltanet_characters Mamba blocks, then DeltaNet blocks 4,989,696 8.89% not evaluated yet

Second data release, 1,153 validation and 1,318 test texts. Character accuracy is the share of positions where the predicted character equals the reference.

Default model. part1/release_2026_11_11/transformer_common_characters/best.pt, the best run on validation. The assignment's entry point downloads it from here by default.

Limitations. These models do not decrypt the cipher. Every run scores below a baseline that ignores the cipher and writes only spaces (16.87% on test). On the second release the models use the cipher slightly: shuffling the cipher symbols raises their loss by +0.0005 to +0.017 per token, with z from 3.4 to 18.0. On the first release this effect was not distinguishable from zero.

Memory and speed. Decoding the test split with the default model peaked at 1.17 GB of GPU memory allocated (7.93 GB reserved by PyTorch's cache), at about 3.2 s per text on a laptop RTX 5050. The Transformer keeps no cache of earlier computations. Mamba with subwords decodes at about 0.2 s per text.

Part 2: fake news detection on UPFD GossipCop

Each model classifies a news propagation graph as fake or real. Settings: 2 layers, hidden size 128, max pooling, AdamW (learning rate 0.001, weight decay 0.01), 200 epochs, batch 128. GAT uses 4 heads; Bi-GCN uses 20% edge drop.

Test results, mean ± standard deviation over 5 seeds, on 3,826 test graphs:

Checkpoint folder Model Features Parameters Accuracy F1
mlp_bert Root-Only MLP bert 115,202 0.7301 ± 0.0093 0.7334 ± 0.0034
gcn_bert GCN bert 115,202 0.9551 ± 0.0040 0.9553 ± 0.0038
sage_bert GraphSAGE bert 229,890 0.9660 ± 0.0019 0.9661 ± 0.0020
gat_bert GAT bert 115,714 0.9671 ± 0.0069 0.9672 ± 0.0069
bigcn_bert Bi-GCN bert 427,522 0.9428 ± 0.0106 0.9433 ± 0.0105
sage_profile GraphSAGE profile 35,842 0.9478 ± 0.0022 0.9484 ± 0.0021
sage_spacy GraphSAGE spacy 110,082 0.9612 ± 0.0023 0.9613 ± 0.0025
sage_content GraphSAGE content 112,642 0.9780 ± 0.0024 0.9781 ± 0.0023

How to use

The commands below run from the root of the assignment's code repository, after uv sync.

With the entry points

bash scripts/part1/test_assn.sh --heldout
bash scripts/part2/test_gnn.sh --ta
  • test_assn.sh --heldout reads data/part1/heldout/cipher.jsonl, one {"id", "ciphertext"} per line. It downloads the default model from this repository and writes output/part1/heldout_result_2025801013.jsonl, one {"id", "text"} per line in input order.
    • --split validation or --split test decodes a split instead.
    • --config config/part1/<run>.yaml picks another run, and --old picks its first-release checkpoint.
    • --local uses models/part1/... on disk instead of this repository.
  • test_gnn.sh --ta scores all 40 Part 2 checkpoints. It uses models/part2 if present and downloads from this repository otherwise, then writes output/part2/results_ta.json.

In Python, Part 1

Save this in the root of the code repository and run it with PYTHONPATH=. uv run python example.py.

import torch
from huggingface_hub import hf_hub_download

from src.cipher.data.cipher_windower import CipherWindower
from src.cipher.inference.generator_factory import GeneratorFactory
from src.cipher.inference.length_rules import LengthRuleFactory
from src.cipher.inference.window_decryptor import WindowDecryptor
from src.cipher.models.checkpoint_loader import CheckpointLoader
from src.cipher.tokenization.vocabulary import Vocabulary

repository = "samvaran/2025801013-anlp-a3"
checkpoint_path = hf_hub_download(repository, "part1/release_2026_11_11/transformer_common_characters/best.pt")
vocabulary_path = hf_hub_download(repository, "part1/vocabulary_characters.json")

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
vocabulary = Vocabulary.load(vocabulary_path)
model = CheckpointLoader.load(checkpoint_path, vocabulary, device)
length_rule = LengthRuleFactory.create(LengthRuleFactory.DEFAULT, vocabulary, device)
generator = GeneratorFactory.create(model, vocabulary, length_rule, 1600, 24000, device)
decryptor = WindowDecryptor(generator, vocabulary, CipherWindower(512))

ciphers = ["<one ciphertext per string>"]
print(decryptor.decrypt(ciphers))

Use part1/vocabulary.json for runs without _characters in their name. 1600 is the model's maximum length, 24000 the tokens per decoding batch, and 512 the window size, as in the run configs.

In Python, Part 2

This needs the UPFD data under data/part2/upfd, which torch_geometric downloads on first use.

import torch
from huggingface_hub import hf_hub_download
from torch_geometric.datasets import UPFD
from torch_geometric.loader import DataLoader

from src.gnn.models.model_config import ModelConfig
from src.gnn.models.model_factory import ModelFactory

checkpoint_path = hf_hub_download("samvaran/2025801013-anlp-a3", "part2/sage_bert/seed_0.pt")
checkpoint = torch.load(checkpoint_path, map_location="cpu")

test_graphs = UPFD("data/part2/upfd", "gossipcop", checkpoint["feature_type"], "test")
model = ModelFactory.create(checkpoint["model_kind"], test_graphs.num_features, ModelConfig(**checkpoint["settings"]["model"]))
model.load_state_dict(checkpoint["model_state"])
model.eval()

correct = 0
with torch.no_grad():
    for graph_batch in DataLoader(test_graphs, batch_size=128):
        predicted = model(graph_batch).argmax(dim=1)
        correct += int((predicted == graph_batch.y).sum())
print(f"test accuracy {correct / len(test_graphs):.4f}")

This prints test accuracy 0.9647, the saved score of that seed.

Checkpoint contents

  • Part 1: model_state (the weights), model_config (architecture settings), step and validation_loss.
  • Part 2: model_state, model_kind, feature_type, seed, best_epoch, best_validation_f1, and settings (dataset, batch size, model and training settings).
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support