anlp-assignment3 / README.md
sahoosk's picture
Update README.md
177e702 verified
|
Raw History Blame Contribute Delete
2.01 kB
---
tags:
- cipher-decryption
- mamba
- deltanet
- graph-neural-network
- fake-news-detection
library_name: pytorch
---
## Layout
| Folder | Contents |
| --- | --- |
| `part1/<model>/` | Part A cipher decoders: `best.pt`, training history, protocol, summary, test and validation metrics |
| `part2/checkpoints/<model>_<feature>/seed<k>/` | Part B checkpoints (best validation macro F1), per-run history and result |
| `part2/` | `results.json` (means), `summary.json` (mean ± std), `runs.json` (every run) |
| `results/` | metric files, cipher analysis, summary tables and figures |
| `outputs/part1/` | decrypted validation and test sets for every model |
| `logs/` | training, evaluation and analysis logs |
| `report/` | the report PDF and its LaTeX source |
## Part A: cipher decryption (test set, 1,318 messages)
| Model | Weights | Char acc | Word acc | F1 | Mean Levenshtein | Params |
| --- | --- | ---: | ---: | ---: | ---: | ---: |
| transformer | `part1/transformer/best.pt` | 0.0904 | 0.0313 | 0.3589 | 663.5 | 8,165,888 |
| mamba | `part1/mamba/best.pt` | 0.0923 | 0.0291 | 0.3914 | 644.1 | 6,858,752 |
| deltanet | `part1/deltanet/best.pt` | 0.0915 | 0.0318 | 0.3528 | 665.3 | 8,195,480 |
| hybrid | `part1/hybrid/best.pt` | 0.0913 | 0.0376 | 0.3764 | 650.0 | 7,527,116 |
Run inference with the assignment code (downloads from this repository by default):
```bash
bash scripts/part1/test_assn.sh --model mamba
```
## Part B: UPFD GossipCop (test set, mean ± std over 5 seeds)
| Configuration | Accuracy | Macro F1 |
| --- | ---: | ---: |
| root_mlp_bert | 0.7383 ± 0.0030 | 0.7369 ± 0.0042 |
| gcn_bert | 0.9479 ± 0.0027 | 0.9479 ± 0.0027 |
| graphsage_bert | 0.9274 ± 0.0027 | 0.9274 ± 0.0027 |
| gat_bert | 0.9658 ± 0.0021 | 0.9658 ± 0.0021 |
| bigcn_bert | 0.9327 ± 0.0033 | 0.9327 ± 0.0033 |
| graphsage_profile | 0.9035 ± 0.0049 | 0.9034 ± 0.0050 |
| graphsage_spacy | 0.9237 ± 0.0071 | 0.9237 ± 0.0072 |
| graphsage_content | 0.9486 ± 0.0071 | 0.9485 ± 0.0071 |