---
library_name: pytorch
tags:
  - transformer
  - encoder-decoder
  - rope
  - grouped-query-attention
  - byte-latent-transformer
  - sequence-to-sequence
---

# ANLP Assignment 1 — C1-C5 Checkpoints

This repository contains the best-validation checkpoints for five custom
encoder-decoder Transformer configurations trained for encrypted binary
sequence to plaintext reconstruction. The Transformer components were
implemented from basic PyTorch operations rather than `nn.Transformer` or
`nn.MultiheadAttention`.

## Repository contents

```text
├── C1/
│   ├── model.pt
│   └── config.json
├── C2/
│   ├── model.pt
│   └── config.json
├── C3/
│   ├── model.pt
│   └── config.json
├── C4/
│   ├── model.pt
│   └── config.json
├── C5/
│   ├── model.pt
│   └── config.json
└── tokenizer/
    └── merge_rules.json
```

Each `config.json` contains the resolved hyperparameters, dataset statistics,
best checkpoint epoch, and final test metrics. All sequence-level test metrics
were computed using greedy decoding from the best-validation checkpoint.

## Configurations

| Config | Positional encoding | Attention | Normalization | Input processing |
|---|---|---|---|---|
| C1 | Sinusoidal | MHA | LayerNorm | Learned binary BPE |
| C2 | RoPE | MHA | LayerNorm | Learned binary BPE |
| C3 | Sinusoidal | GQA | LayerNorm | Learned binary BPE |
| C4 | Sinusoidal | MHA | RMSNorm | Learned binary BPE |
| C5 | Sinusoidal | MHA | LayerNorm | Raw-byte BLT |

C2 is the corrected model in which RoPE is applied independently to Q and K in
encoder self-attention, decoder self-attention, and decoder cross-attention.

## Shared training setup

- Data split: 4,000 train / 500 validation / 500 test, seed 42
- Global model dimension: 256
- Encoder/decoder layers: 4/4
- Global attention heads: 8
- FFN dimension: 1,024
- Dropout: 0.1
- Batch size: 2
- Epochs: 20
- Optimizer: AdamW
- Learning rate: linear 5% warmup to `3e-4`, then cosine decay to `1e-5`
- Weight decay: 0.01
- Gradient clipping: 1.0
- Precision: FP16 mixed precision
- Maximum sequence/decode length: 4,096
- Decoding: greedy

## Final test results

| Config | Bit accuracy | Exact sequence accuracy | Mean Levenshtein | BLEU | ROUGE-1/2/L F1 |
|---|---:|---:|---:|---:|---:|
| C1 | 66.9379% | 0.20% | 228.092 | 0.247765 | 0.577761 / 0.358195 / 0.566949 |
| C2 | 68.8562% | 0.40% | 143.894 | 0.413740 | 0.677726 / 0.497954 / 0.673646 |
| C3 | 61.8402% | 0.20% | 315.952 | 0.127038 | 0.481853 / 0.230759 / 0.463845 |
| C4 | 64.9490% | 0.20% | 245.870 | 0.226044 | 0.565474 / 0.339020 / 0.554313 |
| C5 | 91.6518% | 0.00% | 103.882 | N/A | N/A |

BLEU and ROUGE apply only to the learned-tokenizer configurations C1-C4.

## Tokenizers

C1-C4 use the same frozen binary BPE tokenizer stored in
`tokenizer/merge_rules.json`. It starts from individual binary symbols and uses
1,022 learned variable-length merges. C5 does not use this tokenizer; it uses
raw byte IDs with local BLT patch modules.

## Loading

These are checkpoints for a custom PyTorch architecture, not Transformers
`AutoModel` packages. Instantiate the matching configuration using the
assignment source code, then load the state dictionary:

```python
import torch

checkpoint = torch.load("C2/model.pt", map_location="cpu", weights_only=False)
config = checkpoint["config"]

# Construct the corresponding custom model using config, then:
model.load_state_dict(checkpoint["model_state_dict"])
model.eval()
```

The checkpoint also includes the optimizer state, selected epoch, validation
metrics, and resolved configuration.

## Limitations

- Results are from one random seed.
- Exact reconstruction is difficult because examples can contain thousands of
  output positions and one error invalidates the complete sequence.
- C5 loss and throughput are not directly comparable to C1-C4 because it uses a
  different vocabulary and raw-byte processing.
- These models are intended for coursework and research reproduction, not
  production or security use.