| --- |
| library_name: pytorch |
| pipeline_tag: translation |
| tags: |
| - transformer |
| - seq2seq |
| - cryptanalysis |
| - from-scratch |
| - anlp |
| --- |
| |
| # C5 -- token-free BLT instead of subword |
|
|
| Encoder-decoder transformer written from scratch -- no `nn.Transformer`, no |
| pretrained weights, tokenizer fitted in-repo -- trained to decipher a binary |
| cipher into English plaintext. One row of a five-configuration ablation in |
| which exactly one architectural choice moves at a time. |
|
|
| | | | |
| | --- | --- | |
| | configuration | `C5` -- token-free BLT instead of subword | |
| | positional encoding | sinusoidal | |
| | attention | mha | |
| | normalisation | layernorm | |
| | tokenisation | byte-level (token-free) | |
| | depth | 2 encoder / 2 decoder layers | |
| | width | dim_model 128, 4 heads, dim_ff 512 | |
| | parameters | 2,117,376 | |
| | windowing | 1024-character source windows at stride 768 (25,654 training windows) | |
| | best epoch | 29 (val loss 0.0131) | |
|
|
| ## Test metrics |
|
|
| | metric | test | test_full | gen_test_win | gen_test | gen_test_full | |
| | --- | --- | --- | --- | --- | --- | |
| | loss | 0.0133 | 5.8030 | -- | -- | -- | |
| | perplexity | 1.0134 | 331.3012 | -- | -- | -- | |
| | token acc | 0.9995 | 0.3228 | -- | -- | -- | |
| | bit acc | -- | -- | 0.9997 | 0.9894 | 0.2024 | |
| | seq acc | 0.9575 | 0.0744 | 0.9575 | 0.8040 | 0.0680 | |
| | levenshtein | -- | -- | 0.0854 | 2.0480 | 490.0880 | |
| | lev / len | -- | -- | 0.0007 | 0.0037 | 0.6793 | |
|
|
| - `test` -- teacher-forced, per window |
| - `test_full` -- teacher-forced, per whole line |
| - `gen_test_win` -- greedy, per window |
| - `gen_test` -- greedy, windowed and stitched |
| - `gen_test_full` -- greedy, whole line in one pass |
|
|
| ## Files |
|
|
| - `C5_best.pt` -- weights, optimiser state, config and vocabulary sizes, |
| from the best epoch. |
| - `config.json` -- the full ExperimentConfig this was trained under. |
| - `summary.json` -- the run record: the metrics above plus per-epoch history. |
| - `tokenizer_cipher.json`, `tokenizer_plain.json` -- the fitted unigram |
| vocabularies, when the config uses them. Byte-level configurations have none |
| by construction. |
|
|
| ## Loading it |
|
|
| The checkpoint is a plain `torch.save` dict, not a `transformers` model. The |
| vocabulary sizes travel with it because they are not recoverable from the |
| config -- the tokenizer lands wherever pruning leaves it: |
|
|
| ```python |
| import torch |
| from huggingface_hub import hf_hub_download |
| |
| from src.config import get_config |
| from src.models.base import build_model |
| |
| path = hf_hub_download("winterdewdev/anlp-a1-transformers-C5", "C5_best.pt") |
| state = torch.load(path, map_location="cpu", weights_only=False) |
| |
| cfg = get_config(state["config_name"]) |
| model = build_model(cfg, state["src_vocab_size"], state["tgt_vocab_size"]) |
| model.load_state_dict(state["model"]) |
| model.eval() |
| ``` |
|
|
| Trained with `python -m src.train --config C5`. |
|
|
