| --- |
| library_name: pytorch |
| pipeline_tag: translation |
| tags: |
| - transformer |
| - seq2seq |
| - cryptanalysis |
| - from-scratch |
| - anlp |
| --- |
| |
| # C4 -- RMSNorm instead of LayerNorm |
|
|
| Encoder-decoder transformer written from scratch -- no `nn.Transformer`, no |
| pretrained weights, tokenizer fitted in-repo -- trained to decipher a binary |
| cipher into English plaintext. One row of a five-configuration ablation in |
| which exactly one architectural choice moves at a time. |
|
|
| | | | |
| | --- | --- | |
| | configuration | `C4` -- RMSNorm instead of LayerNorm | |
| | positional encoding | sinusoidal | |
| | attention | mha | |
| | normalisation | rmsnorm | |
| | tokenisation | unigram LM subword, fitted on the training split | |
| | depth | 2 encoder / 2 decoder layers | |
| | width | dim_model 128, 4 heads, dim_ff 512 | |
| | parameters | 2,761,728 | |
| | windowing | 1024-character source windows at stride 768 (25,654 training windows) | |
| | best epoch | 95 (val loss 0.0932) | |
|
|
| ## Test metrics |
|
|
| | metric | test | test_full | gen_test_win | gen_test | gen_test_full | |
| | --- | --- | --- | --- | --- | --- | |
| | loss | 0.1127 | 7.0180 | -- | -- | -- | |
| | perplexity | 1.1193 | 1116.5630 | -- | -- | -- | |
| | token acc | 0.9778 | 0.2683 | 0.8427 | 0.4516 | 0.1518 | |
| | bit acc | -- | -- | 0.9582 | 0.8168 | 0.2638 | |
| | seq acc | 0.6063 | 0.0528 | 0.6996 | 0.2480 | 0.0500 | |
| | levenshtein | -- | -- | 0.9809 | 17.9300 | 491.9480 | |
| | lev / len | -- | -- | 0.0079 | 0.0263 | 0.6246 | |
| | BLEU | -- | -- | 0.9555 | 0.9372 | 0.1109 | |
| | ROUGE-1 | -- | -- | 0.9793 | 0.9688 | 0.4146 | |
| | ROUGE-2 | -- | -- | 0.9625 | 0.9505 | 0.3715 | |
| | ROUGE-L | -- | -- | 0.9793 | 0.9687 | 0.4112 | |
|
|
| - `test` -- teacher-forced, per window |
| - `test_full` -- teacher-forced, per whole line |
| - `gen_test_win` -- greedy, per window |
| - `gen_test` -- greedy, windowed and stitched |
| - `gen_test_full` -- greedy, whole line in one pass |
|
|
| ## Files |
|
|
| - `C4_best.pt` -- weights, optimiser state, config and vocabulary sizes, |
| from the best epoch. |
| - `config.json` -- the full ExperimentConfig this was trained under. |
| - `summary.json` -- the run record: the metrics above plus per-epoch history. |
| - `tokenizer_cipher.json`, `tokenizer_plain.json` -- the fitted unigram |
| vocabularies, when the config uses them. Byte-level configurations have none |
| by construction. |
|
|
| ## Loading it |
|
|
| The checkpoint is a plain `torch.save` dict, not a `transformers` model. The |
| vocabulary sizes travel with it because they are not recoverable from the |
| config -- the tokenizer lands wherever pruning leaves it: |
|
|
| ```python |
| import torch |
| from huggingface_hub import hf_hub_download |
| |
| from src.config import get_config |
| from src.models.base import build_model |
| |
| path = hf_hub_download("winterdewdev/anlp-a1-transformers-C4", "C4_best.pt") |
| state = torch.load(path, map_location="cpu", weights_only=False) |
| |
| cfg = get_config(state["config_name"]) |
| model = build_model(cfg, state["src_vocab_size"], state["tgt_vocab_size"]) |
| model.load_state_dict(state["model"]) |
| model.eval() |
| ``` |
|
|
| Trained with `python -m src.train --config C4`. |
|
|
