| --- |
| library_name: pytorch |
| pipeline_tag: translation |
| tags: |
| - transformer |
| - seq2seq |
| - cryptanalysis |
| - from-scratch |
| - anlp |
| --- |
| |
| # C2 -- RoPE instead of sinusoidal absolute |
|
|
| Encoder-decoder transformer written from scratch -- no `nn.Transformer`, no |
| pretrained weights, tokenizer fitted in-repo -- trained to decipher a binary |
| cipher into English plaintext. One row of a five-configuration ablation in |
| which exactly one architectural choice moves at a time. |
|
|
| | | | |
| | --- | --- | |
| | configuration | `C2` -- RoPE instead of sinusoidal absolute | |
| | positional encoding | rope | |
| | attention | mha | |
| | normalisation | layernorm | |
| | tokenisation | unigram LM subword, fitted on the training split | |
| | depth | 2 encoder / 2 decoder layers | |
| | width | dim_model 128, 4 heads, dim_ff 512 | |
| | parameters | 2,763,264 | |
| | windowing | 1024-character source windows at stride 768 (25,654 training windows) | |
| | best epoch | 100 (val loss 0.0993) | |
|
|
| ## Test metrics |
|
|
| | metric | test | test_full | gen_test_win | gen_test | gen_test_full | |
| | --- | --- | --- | --- | --- | --- | |
| | loss | 0.1198 | 5.9206 | -- | -- | -- | |
| | perplexity | 1.1273 | 372.6371 | -- | -- | -- | |
| | token acc | 0.9747 | 0.3038 | 0.8211 | 0.4405 | 0.0425 | |
| | bit acc | -- | -- | 0.9561 | 0.8182 | 0.3518 | |
| | seq acc | 0.5632 | 0.0528 | 0.6598 | 0.2040 | 0.0540 | |
| | levenshtein | -- | -- | 1.5377 | 22.9500 | 505.6000 | |
| | lev / len | -- | -- | 0.0103 | 0.0342 | 0.6313 | |
| | BLEU | -- | -- | 0.9453 | 0.9246 | 0.1027 | |
| | ROUGE-1 | -- | -- | 0.9753 | 0.9620 | 0.3624 | |
| | ROUGE-2 | -- | -- | 0.9555 | 0.9412 | 0.2560 | |
| | ROUGE-L | -- | -- | 0.9753 | 0.9619 | 0.3113 | |
|
|
| - `test` -- teacher-forced, per window |
| - `test_full` -- teacher-forced, per whole line |
| - `gen_test_win` -- greedy, per window |
| - `gen_test` -- greedy, windowed and stitched |
| - `gen_test_full` -- greedy, whole line in one pass |
|
|
| ## Files |
|
|
| - `C2_best.pt` -- weights, optimiser state, config and vocabulary sizes, |
| from the best epoch. |
| - `config.json` -- the full ExperimentConfig this was trained under. |
| - `summary.json` -- the run record: the metrics above plus per-epoch history. |
| - `tokenizer_cipher.json`, `tokenizer_plain.json` -- the fitted unigram |
| vocabularies, when the config uses them. Byte-level configurations have none |
| by construction. |
|
|
| ## Loading it |
|
|
| The checkpoint is a plain `torch.save` dict, not a `transformers` model. The |
| vocabulary sizes travel with it because they are not recoverable from the |
| config -- the tokenizer lands wherever pruning leaves it: |
|
|
| ```python |
| import torch |
| from huggingface_hub import hf_hub_download |
| |
| from src.config import get_config |
| from src.models.base import build_model |
| |
| path = hf_hub_download("winterdewdev/anlp-a1-transformers-C2", "C2_best.pt") |
| state = torch.load(path, map_location="cpu", weights_only=False) |
| |
| cfg = get_config(state["config_name"]) |
| model = build_model(cfg, state["src_vocab_size"], state["tgt_vocab_size"]) |
| model.load_state_dict(state["model"]) |
| model.eval() |
| ``` |
|
|
| Trained with `python -m src.train --config C2`. |
|
|
