| --- |
| library_name: pytorch |
| pipeline_tag: translation |
| tags: |
| - transformer |
| - seq2seq |
| - cryptanalysis |
| - from-scratch |
| - anlp |
| --- |
| |
| # C1 -- base sinusoidal MHA LayerNorm subword |
|
|
| Encoder-decoder transformer written from scratch -- no `nn.Transformer`, no |
| pretrained weights, tokenizer fitted in-repo -- trained to decipher a binary |
| cipher into English plaintext. One row of a five-configuration ablation in |
| which exactly one architectural choice moves at a time. |
|
|
| | | | |
| | --- | --- | |
| | configuration | `C1` -- base sinusoidal MHA LayerNorm subword | |
| | positional encoding | sinusoidal | |
| | attention | mha | |
| | normalisation | layernorm | |
| | tokenisation | unigram LM subword, fitted on the training split | |
| | depth | 2 encoder / 2 decoder layers | |
| | width | dim_model 128, 4 heads, dim_ff 512 | |
| | parameters | 2,763,264 | |
| | windowing | 1024-character source windows at stride 768 (25,654 training windows) | |
| | best epoch | 98 (val loss 0.0942) | |
|
|
| ## Test metrics |
|
|
| | metric | test | test_full | gen_test_win | gen_test | gen_test_full | |
| | --- | --- | --- | --- | --- | --- | |
| | loss | 0.1108 | 7.0169 | -- | -- | -- | |
| | perplexity | 1.1172 | 1115.3036 | -- | -- | -- | |
| | token acc | 0.9776 | 0.2704 | 0.8477 | 0.4725 | 0.1503 | |
| | bit acc | -- | -- | 0.9586 | 0.8195 | 0.2315 | |
| | seq acc | 0.5972 | 0.0482 | 0.7002 | 0.2520 | 0.0520 | |
| | levenshtein | -- | -- | 0.9238 | 18.9620 | 472.0240 | |
| | lev / len | -- | -- | 0.0075 | 0.0283 | 0.6332 | |
| | BLEU | -- | -- | 0.9566 | 0.9368 | 0.0575 | |
| | ROUGE-1 | -- | -- | 0.9800 | 0.9692 | 0.4304 | |
| | ROUGE-2 | -- | -- | 0.9636 | 0.9519 | 0.3962 | |
| | ROUGE-L | -- | -- | 0.9800 | 0.9691 | 0.4292 | |
|
|
| - `test` -- teacher-forced, per window |
| - `test_full` -- teacher-forced, per whole line |
| - `gen_test_win` -- greedy, per window |
| - `gen_test` -- greedy, windowed and stitched |
| - `gen_test_full` -- greedy, whole line in one pass |
|
|
| ## Files |
|
|
| - `C1_best.pt` -- weights, optimiser state, config and vocabulary sizes, |
| from the best epoch. |
| - `config.json` -- the full ExperimentConfig this was trained under. |
| - `summary.json` -- the run record: the metrics above plus per-epoch history. |
| - `tokenizer_cipher.json`, `tokenizer_plain.json` -- the fitted unigram |
| vocabularies, when the config uses them. Byte-level configurations have none |
| by construction. |
|
|
| ## Loading it |
|
|
| The checkpoint is a plain `torch.save` dict, not a `transformers` model. The |
| vocabulary sizes travel with it because they are not recoverable from the |
| config -- the tokenizer lands wherever pruning leaves it: |
|
|
| ```python |
| import torch |
| from huggingface_hub import hf_hub_download |
| |
| from src.config import get_config |
| from src.models.base import build_model |
| |
| path = hf_hub_download("winterdewdev/anlp-a1-transformers-C1", "C1_best.pt") |
| state = torch.load(path, map_location="cpu", weights_only=False) |
| |
| cfg = get_config(state["config_name"]) |
| model = build_model(cfg, state["src_vocab_size"], state["tgt_vocab_size"]) |
| model.load_state_dict(state["model"]) |
| model.eval() |
| ``` |
|
|
| Trained with `python -m src.train --config C1`. |
|
|
