File size: 2,992 Bytes
904aed4 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 | ---
library_name: pytorch
pipeline_tag: translation
tags:
- transformer
- seq2seq
- cryptanalysis
- from-scratch
- anlp
---
# C2 -- RoPE instead of sinusoidal absolute
Encoder-decoder transformer written from scratch -- no `nn.Transformer`, no
pretrained weights, tokenizer fitted in-repo -- trained to decipher a binary
cipher into English plaintext. One row of a five-configuration ablation in
which exactly one architectural choice moves at a time.
| | |
| --- | --- |
| configuration | `C2` -- RoPE instead of sinusoidal absolute |
| positional encoding | rope |
| attention | mha |
| normalisation | layernorm |
| tokenisation | unigram LM subword, fitted on the training split |
| depth | 2 encoder / 2 decoder layers |
| width | dim_model 128, 4 heads, dim_ff 512 |
| parameters | 2,763,264 |
| windowing | 1024-character source windows at stride 768 (25,654 training windows) |
| best epoch | 100 (val loss 0.0993) |
## Test metrics
| metric | test | test_full | gen_test_win | gen_test | gen_test_full |
| --- | --- | --- | --- | --- | --- |
| loss | 0.1198 | 5.9206 | -- | -- | -- |
| perplexity | 1.1273 | 372.6371 | -- | -- | -- |
| token acc | 0.9747 | 0.3038 | 0.8211 | 0.4405 | 0.0425 |
| bit acc | -- | -- | 0.9561 | 0.8182 | 0.3518 |
| seq acc | 0.5632 | 0.0528 | 0.6598 | 0.2040 | 0.0540 |
| levenshtein | -- | -- | 1.5377 | 22.9500 | 505.6000 |
| lev / len | -- | -- | 0.0103 | 0.0342 | 0.6313 |
| BLEU | -- | -- | 0.9453 | 0.9246 | 0.1027 |
| ROUGE-1 | -- | -- | 0.9753 | 0.9620 | 0.3624 |
| ROUGE-2 | -- | -- | 0.9555 | 0.9412 | 0.2560 |
| ROUGE-L | -- | -- | 0.9753 | 0.9619 | 0.3113 |
- `test` -- teacher-forced, per window
- `test_full` -- teacher-forced, per whole line
- `gen_test_win` -- greedy, per window
- `gen_test` -- greedy, windowed and stitched
- `gen_test_full` -- greedy, whole line in one pass
## Files
- `C2_best.pt` -- weights, optimiser state, config and vocabulary sizes,
from the best epoch.
- `config.json` -- the full ExperimentConfig this was trained under.
- `summary.json` -- the run record: the metrics above plus per-epoch history.
- `tokenizer_cipher.json`, `tokenizer_plain.json` -- the fitted unigram
vocabularies, when the config uses them. Byte-level configurations have none
by construction.
## Loading it
The checkpoint is a plain `torch.save` dict, not a `transformers` model. The
vocabulary sizes travel with it because they are not recoverable from the
config -- the tokenizer lands wherever pruning leaves it:
```python
import torch
from huggingface_hub import hf_hub_download
from src.config import get_config
from src.models.base import build_model
path = hf_hub_download("winterdewdev/anlp-a1-transformers-C2", "C2_best.pt")
state = torch.load(path, map_location="cpu", weights_only=False)
cfg = get_config(state["config_name"])
model = build_model(cfg, state["src_vocab_size"], state["tgt_vocab_size"])
model.load_state_dict(state["model"])
model.eval()
```
Trained with `python -m src.train --config C2`.
|