Surya4/anlp-a3-part2

Propagation-Graph Classifiers on UPFD GossipCop

Checkpoints for Part B of the ANLP (Monsoon 2026) Assignment 3. The task is binary classification of news propagation trees: does a news story spread as fake or real, given the source post and the retweet graph around it.

Nine model configurations are included, each trained with five seeds:

• the architecture comparison on BERT node features: root-only MLP, GCN, GraphSAGE, GAT, Bi-GCN;
• the feature ablation on GraphSAGE: profile, spaCy, BERT and content features.

Data

GossipCop subset of UPFD, loaded through PyG's UPFD class with the shipped 20 / 10 / 70 split and never re-split.

Split	Graphs	Nodes
Train	1,092	63,482
Validation	546	30,791
Test	3,826	219,989

Node 0 of every graph is the source news post; the remaining nodes are users who retweeted it. Edges are stored parent-to-child (verified: every edge points from a shallower to a deeper node, and node 0 has no incoming edge). Labels are binary and almost balanced on test (1,910 / 1,916).

Feature types:

feature	Dimension	Contents
profile	10	user profile attributes
spacy	300	user tweet embeddings, spaCy
bert	768	user tweet embeddings, BERT
content	310	spaCy (300) + profile (10)

Each feature type is standardised per dimension with training-split statistics. The transform is identical across feature types.

Models

All five architectures share the same readout style: 128 hidden units, two message-passing layers, dropout 0.5, mean pooling over nodes, and a linear classifier.

• Root-Only MLP — uses only node 0's feature vector, discarding every user node and edge. The text-only baseline.
• GCN — the standard first-order graph convolution (Kipf & Welling).
• GraphSAGE — mean-aggregator neighbourhood sampling.
• GAT — four attention heads on the first layer, one on the second.
• Bi-GCN — Bian et al., Rumor Detection on Social Media with Bi-Directional Graph Convolutional Networks (AAAI 2020). One GCN on the parent-to-child graph and one on its transpose, root feature concatenated at every layer, DropEdge 0.2, mean pooling per direction, then concatenation.

Training: Adam, learning rate 1e-3, weight decay 5e-4, batch 32, at most 100 epochs, early stopping on validation macro-F1 with patience 20. The checkpoint evaluated on test is the epoch with the best validation macro-F1, as the assignment requires.

Results

Test-set mean ± standard deviation over five seeds. Precision, recall and F1 are macro-averaged over the two classes.

Configuration	Accuracy	Precision	Recall	F1	Parameters
mlp_bert	73.72 ± 0.42	74.11 ± 0.40	73.73 ± 0.42	73.61 ± 0.44	115,202
gcn_bert	84.47 ± 0.44	84.58 ± 0.38	84.47 ± 0.44	84.46 ± 0.45	115,202
sage_bert	93.87 ± 0.53	93.88 ± 0.53	93.87 ± 0.53	93.87 ± 0.53	229,890
gat_bert	95.49 ± 0.32	95.50 ± 0.32	95.49 ± 0.32	95.49 ± 0.32	115,714
bigcn_bert	95.28 ± 0.17	95.30 ± 0.15	95.29 ± 0.17	95.28 ± 0.17	427,522
sage_profile	92.93 ± 0.66	93.00 ± 0.62	92.93 ± 0.66	92.93 ± 0.66	35,842
sage_spacy	93.59 ± 0.15	93.60 ± 0.15	93.59 ± 0.15	93.59 ± 0.15	110,082
sage_content	95.87 ± 0.36	95.88 ± 0.36	95.87 ± 0.36	95.86 ± 0.36	112,642

sage_bert is the same run as the BERT row of the architecture comparison and is therefore reported once in the table; the file list still contains both names because the same checkpoint serves both questions.

Interpretation

Three things stand out.

1. The graph helps a lot. The root-only MLP reaches 73.7% using only the source post. Every graph model is at least 10 points better, and the best ones reach 95.5%. The retweet structure carries information the article text alone does not.

2. Attention and bidirectional propagation are the strongest inductive biases. GAT (95.49%) and Bi-GCN (95.28%) are within 0.2 points of each other and clearly ahead of GCN (84.47%) and GraphSAGE (93.87%). GCN's fixed isotropic averaging is the weakest graph model.

3. content is the best feature set, not bert. With GraphSAGE fixed, content (spaCy + profile, 310 dimensions) reaches 95.87%, above BERT's 93.87% despite being less than half the width. Profile features alone already reach 92.93%, so user-level attributes are highly informative, and combining them with tweet embeddings is better than either alone. Article/user embeddings alone are not the whole signal.

Standard deviations are at most 0.66 points, so these orderings are stable across seeds.

Checkpoint format

Each file is a torch.save dictionary:

import torch

payload = torch.load("sage_bert_seed0.pt", map_location="cpu", weights_only=False)
payload["model"]        # state_dict
payload["config"]       # training and architecture configuration
payload["arch"]         # mlp | gcn | sage | gat | bigcn
payload["feature"]      # bert | profile | spacy | content
payload["seed"]         # 0-4
payload["in_dim"]       # input feature dimension
payload["num_classes"]  # 2
payload["best_epoch"]   # epoch selected by validation macro-F1
payload["best_val"]     # validation metrics at that epoch
payload["test"]         # test metrics of this checkpoint

Reloading requires the model definitions in src/part2/models.py from the accompanying repository. Load with weights_only=False because the payload contains dictionaries and lists.

Limitations

These weights are an academic artefact for comparing graph architectures on a fixed benchmark. They are not a fake-news detector for deployment: the dataset is a single English-language snapshot, the labels come from one source, and the models are evaluated only on the shipped test split.

Training and evaluation logs are in the Weights & Biases project anlp-a3-part2, one run per configuration plus a matching -evals run.