| --- |
| library_name: pytorch |
| pipeline_tag: translation |
| tags: |
| - transformer |
| - seq2seq |
| - cryptanalysis |
| - from-scratch |
| - anlp |
| --- |
| |
| # C3 -- grouped-query attention instead of MHA |
|
|
| Encoder-decoder transformer written from scratch -- no `nn.Transformer`, no |
| pretrained weights, tokenizer fitted in-repo -- trained to decipher a binary |
| cipher into English plaintext. One row of a five-configuration ablation in |
| which exactly one architectural choice moves at a time. |
|
|
| | | | |
| | --- | --- | |
| | configuration | `C3` -- grouped-query attention instead of MHA | |
| | positional encoding | sinusoidal | |
| | attention | gqa | |
| | normalisation | layernorm | |
| | tokenisation | unigram LM subword, fitted on the training split | |
| | depth | 2 encoder / 2 decoder layers | |
| | width | dim_model 128, 4 heads, dim_ff 512 | |
| | parameters | 2,664,192 | |
| | windowing | 1024-character source windows at stride 768 (25,654 training windows) | |
| | best epoch | 100 (val loss 0.0970) | |
|
|
| ## Test metrics |
|
|
| | metric | test | test_full | gen_test_win | gen_test | gen_test_full | |
| | --- | --- | --- | --- | --- | --- | |
| | loss | 0.1144 | 7.0727 | -- | -- | -- | |
| | perplexity | 1.1212 | 1179.3361 | -- | -- | -- | |
| | token acc | 0.9765 | 0.2687 | 0.8319 | 0.4398 | 0.1262 | |
| | bit acc | -- | -- | 0.9541 | 0.8034 | 0.3040 | |
| | seq acc | 0.5841 | 0.0550 | 0.6753 | 0.2340 | 0.0540 | |
| | levenshtein | -- | -- | 1.0814 | 20.9620 | 514.4440 | |
| | lev / len | -- | -- | 0.0087 | 0.0311 | 0.6223 | |
| | BLEU | -- | -- | 0.9510 | 0.9292 | 0.1512 | |
| | ROUGE-1 | -- | -- | 0.9773 | 0.9656 | 0.4092 | |
| | ROUGE-2 | -- | -- | 0.9588 | 0.9462 | 0.3617 | |
| | ROUGE-L | -- | -- | 0.9773 | 0.9655 | 0.4042 | |
|
|
| - `test` -- teacher-forced, per window |
| - `test_full` -- teacher-forced, per whole line |
| - `gen_test_win` -- greedy, per window |
| - `gen_test` -- greedy, windowed and stitched |
| - `gen_test_full` -- greedy, whole line in one pass |
|
|
| ## Files |
|
|
| - `C3_best.pt` -- weights, optimiser state, config and vocabulary sizes, |
| from the best epoch. |
| - `config.json` -- the full ExperimentConfig this was trained under. |
| - `summary.json` -- the run record: the metrics above plus per-epoch history. |
| - `tokenizer_cipher.json`, `tokenizer_plain.json` -- the fitted unigram |
| vocabularies, when the config uses them. Byte-level configurations have none |
| by construction. |
|
|
| ## Loading it |
|
|
| The checkpoint is a plain `torch.save` dict, not a `transformers` model. The |
| vocabulary sizes travel with it because they are not recoverable from the |
| config -- the tokenizer lands wherever pruning leaves it: |
|
|
| ```python |
| import torch |
| from huggingface_hub import hf_hub_download |
| |
| from src.config import get_config |
| from src.models.base import build_model |
| |
| path = hf_hub_download("winterdewdev/anlp-a1-transformers-C3", "C3_best.pt") |
| state = torch.load(path, map_location="cpu", weights_only=False) |
| |
| cfg = get_config(state["config_name"]) |
| model = build_model(cfg, state["src_vocab_size"], state["tgt_vocab_size"]) |
| model.load_state_dict(state["model"]) |
| model.eval() |
| ``` |
|
|
| Trained with `python -m src.train --config C3`. |
|
|
