Layout
| Folder | Contents |
|---|---|
part1/<model>/ |
Part A cipher decoders: best.pt, training history, protocol, summary, test and validation metrics |
part2/checkpoints/<model>_<feature>/seed<k>/ |
Part B checkpoints (best validation macro F1), per-run history and result |
part2/ |
results.json (means), summary.json (mean ± std), runs.json (every run) |
results/ |
metric files, cipher analysis, summary tables and figures |
outputs/part1/ |
decrypted validation and test sets for every model |
logs/ |
training, evaluation and analysis logs |
report/ |
the report PDF and its LaTeX source |
Part A: cipher decryption (test set, 1,318 messages)
| Model | Weights | Char acc | Word acc | F1 | Mean Levenshtein | Params |
|---|---|---|---|---|---|---|
| transformer | part1/transformer/best.pt |
0.0904 | 0.0313 | 0.3589 | 663.5 | 8,165,888 |
| mamba | part1/mamba/best.pt |
0.0923 | 0.0291 | 0.3914 | 644.1 | 6,858,752 |
| deltanet | part1/deltanet/best.pt |
0.0915 | 0.0318 | 0.3528 | 665.3 | 8,195,480 |
| hybrid | part1/hybrid/best.pt |
0.0913 | 0.0376 | 0.3764 | 650.0 | 7,527,116 |
Run inference with the assignment code (downloads from this repository by default):
bash scripts/part1/test_assn.sh --model mamba
Part B: UPFD GossipCop (test set, mean ± std over 5 seeds)
| Configuration | Accuracy | Macro F1 |
|---|---|---|
| root_mlp_bert | 0.7383 ± 0.0030 | 0.7369 ± 0.0042 |
| gcn_bert | 0.9479 ± 0.0027 | 0.9479 ± 0.0027 |
| graphsage_bert | 0.9274 ± 0.0027 | 0.9274 ± 0.0027 |
| gat_bert | 0.9658 ± 0.0021 | 0.9658 ± 0.0021 |
| bigcn_bert | 0.9327 ± 0.0033 | 0.9327 ± 0.0033 |
| graphsage_profile | 0.9035 ± 0.0049 | 0.9034 ± 0.0050 |
| graphsage_spacy | 0.9237 ± 0.0071 | 0.9237 ± 0.0072 |
| graphsage_content | 0.9486 ± 0.0071 | 0.9485 ± 0.0071 |
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support