| --- |
| library_name: pytorch |
| tags: |
| - transformer |
| - encoder-decoder |
| - rope |
| - grouped-query-attention |
| - byte-latent-transformer |
| - sequence-to-sequence |
| --- |
| |
| # ANLP Assignment 1 — C1-C5 Checkpoints |
|
|
| This repository contains the best-validation checkpoints for five custom |
| encoder-decoder Transformer configurations trained for encrypted binary |
| sequence to plaintext reconstruction. The Transformer components were |
| implemented from basic PyTorch operations rather than `nn.Transformer` or |
| `nn.MultiheadAttention`. |
|
|
| ## Repository contents |
|
|
| ```text |
| ├── C1/ |
| │ ├── model.pt |
| │ └── config.json |
| ├── C2/ |
| │ ├── model.pt |
| │ └── config.json |
| ├── C3/ |
| │ ├── model.pt |
| │ └── config.json |
| ├── C4/ |
| │ ├── model.pt |
| │ └── config.json |
| ├── C5/ |
| │ ├── model.pt |
| │ └── config.json |
| └── tokenizer/ |
| └── merge_rules.json |
| ``` |
|
|
| Each `config.json` contains the resolved hyperparameters, dataset statistics, |
| best checkpoint epoch, and final test metrics. All sequence-level test metrics |
| were computed using greedy decoding from the best-validation checkpoint. |
|
|
| ## Configurations |
|
|
| | Config | Positional encoding | Attention | Normalization | Input processing | |
| |---|---|---|---|---| |
| | C1 | Sinusoidal | MHA | LayerNorm | Learned binary BPE | |
| | C2 | RoPE | MHA | LayerNorm | Learned binary BPE | |
| | C3 | Sinusoidal | GQA | LayerNorm | Learned binary BPE | |
| | C4 | Sinusoidal | MHA | RMSNorm | Learned binary BPE | |
| | C5 | Sinusoidal | MHA | LayerNorm | Raw-byte BLT | |
|
|
| C2 is the corrected model in which RoPE is applied independently to Q and K in |
| encoder self-attention, decoder self-attention, and decoder cross-attention. |
|
|
| ## Shared training setup |
|
|
| - Data split: 4,000 train / 500 validation / 500 test, seed 42 |
| - Global model dimension: 256 |
| - Encoder/decoder layers: 4/4 |
| - Global attention heads: 8 |
| - FFN dimension: 1,024 |
| - Dropout: 0.1 |
| - Batch size: 2 |
| - Epochs: 20 |
| - Optimizer: AdamW |
| - Learning rate: linear 5% warmup to `3e-4`, then cosine decay to `1e-5` |
| - Weight decay: 0.01 |
| - Gradient clipping: 1.0 |
| - Precision: FP16 mixed precision |
| - Maximum sequence/decode length: 4,096 |
| - Decoding: greedy |
|
|
| ## Final test results |
|
|
| | Config | Bit accuracy | Exact sequence accuracy | Mean Levenshtein | BLEU | ROUGE-1/2/L F1 | |
| |---|---:|---:|---:|---:|---:| |
| | C1 | 66.9379% | 0.20% | 228.092 | 0.247765 | 0.577761 / 0.358195 / 0.566949 | |
| | C2 | 68.8562% | 0.40% | 143.894 | 0.413740 | 0.677726 / 0.497954 / 0.673646 | |
| | C3 | 61.8402% | 0.20% | 315.952 | 0.127038 | 0.481853 / 0.230759 / 0.463845 | |
| | C4 | 64.9490% | 0.20% | 245.870 | 0.226044 | 0.565474 / 0.339020 / 0.554313 | |
| | C5 | 91.6518% | 0.00% | 103.882 | N/A | N/A | |
|
|
| BLEU and ROUGE apply only to the learned-tokenizer configurations C1-C4. |
|
|
| ## Tokenizers |
|
|
| C1-C4 use the same frozen binary BPE tokenizer stored in |
| `tokenizer/merge_rules.json`. It starts from individual binary symbols and uses |
| 1,022 learned variable-length merges. C5 does not use this tokenizer; it uses |
| raw byte IDs with local BLT patch modules. |
|
|
| ## Loading |
|
|
| These are checkpoints for a custom PyTorch architecture, not Transformers |
| `AutoModel` packages. Instantiate the matching configuration using the |
| assignment source code, then load the state dictionary: |
|
|
| ```python |
| import torch |
| |
| checkpoint = torch.load("C2/model.pt", map_location="cpu", weights_only=False) |
| config = checkpoint["config"] |
| |
| # Construct the corresponding custom model using config, then: |
| model.load_state_dict(checkpoint["model_state_dict"]) |
| model.eval() |
| ``` |
|
|
| The checkpoint also includes the optimizer state, selected epoch, validation |
| metrics, and resolved configuration. |
|
|
| ## Limitations |
|
|
| - Results are from one random seed. |
| - Exact reconstruction is difficult because examples can contain thousands of |
| output positions and one error invalidates the complete sequence. |
| - C5 loss and throughput are not directly comparable to C1-C4 because it uses a |
| different vocabulary and raw-byte processing. |
| - These models are intended for coursework and research reproduction, not |
| production or security use. |
|
|
