---
language:
- en
license: mit
library_name: pytorch
tags:
- transformers-from-scratch
- encoder-decoder
- seq2seq
- ablation
- bpe
- rope
- gqa
- rmsnorm
- blt
- anlp
- ciphertext-decryption
metrics:
- accuracy
- bleu
- rouge
- levenshtein
---

# ANLP Assignment 1 — Encoder–Decoder Ablations (C1–C5)

This hub entry hosts checkpoints from a controlled ablation: map encrypted binary sequences (`brown_cipher.txt`) to English plaintext (`brown_plain.txt`). Configuration **C1** is the base model; **C2–C5** each change exactly one component.

---

## Model variants

| Config | Single change vs C1 | Architecture class |
|--------|---------------------|--------------------|
| **C1** | — (base) | `Seq2SeqTransformer` |
| **C2** | RoPE instead of sinusoidal PE | `Seq2SeqTransformer` |
| **C3** | GQA (8 Q, 2 KV) instead of MHA | `Seq2SeqTransformer` |
| **C4** | RMSNorm instead of LayerNorm | `Seq2SeqTransformer` |
| **C5** | Entropy-patched BLT instead of BPE | `BLTSeq2Seq` |

### Shared hyperparameters

| Hyperparameter | Value |
|----------------|-------|
| `d_model` | 256 |
| Heads | 8 (C3: 2 KV heads) |
| Encoder / decoder layers | 4 / 4 |
| `d_ff` | 1024 |
| Dropout | 0.15 |
| Optimizer | AdamW, lr 1e-3, wd 0.01, betas (0.9, 0.98) |
| LR schedule | cosine + 400 warmup steps |
| Batch size | 64 |
| Chunk size | 256 characters (aligned cipher/plain) |
| Max epochs | 100 (C5 early-stopped later; best @ epoch 146) |
| Early stopping | patience 10 on val CE |
| Seed | 42 |
| Precision | AMP (fp16) |

**Tokenization**

- **C1–C4:** from-scratch BPE, vocab 1024 per side. Cipher: pack 8 bits → byte, then learn merges. Plain: character BPE. Artifacts: `cipher_bpe.json`, `plain_bpe.json`.
- **C5:** no BPE. Packed bytes + entropy patches (len 1–8, mean ≈ 4.2), hashed 2/3/4-gram features. Artifacts: `entropy_ngram.json`, `entropy_patcher.json`.

---

## Intended use

- **Primary:** course ablation / reproducibility for ANLP Assignment 1.  
- **Task:** greedy decode ciphertext chunks → plaintext chunks.  
- **Not intended for:** production MT, open-domain generation, or claiming a general “decrypt any cipher” system. The mapping is specific to this homework dataset.

---

## How to load weights

Checkpoints are raw PyTorch state dicts (`pytorch_model.bin`) plus `config.json`. They are **not** Hugging Face `transformers.AutoModel` checkpoints—load with the assignment code.

```python
import json
import torch
from pathlib import Path

# after cloning / downloading this repo's `src/` onto PYTHONPATH
from models.transformer import Seq2SeqTransformer
from models.blt import BLTSeq2Seq
from tokenizer import BPETokenizer, PAD

hf_dir = Path("path/to/downloaded/hf")  # folder with config.json + pytorch_model.bin
cfg = json.loads((hf_dir / "config.json").read_text())

if cfg.get("blt"):
    model = BLTSeq2Seq(
        d_model=cfg["d_model"],
        n_heads=cfg["n_heads"],
        n_kv_heads=cfg["n_kv_heads"],
        n_encoder_layers=cfg["n_encoder_layers"],
        n_decoder_layers=cfg["n_decoder_layers"],
        d_ff=cfg["d_ff"],
        dropout=cfg["dropout"],
        patch_size=cfg["patch_size"],
        n_local_layers=cfg["n_local_layers"],
        n_local_heads=cfg["n_local_heads"],
        pad_id=PAD,
    )
else:
    model = Seq2SeqTransformer(
        src_vocab=cfg["src_vocab"],
        tgt_vocab=cfg["tgt_vocab"],
        d_model=cfg["d_model"],
        n_heads=cfg["n_heads"],
        n_kv_heads=cfg["n_kv_heads"],
        n_encoder_layers=cfg["n_encoder_layers"],
        n_decoder_layers=cfg["n_decoder_layers"],
        d_ff=cfg["d_ff"],
        dropout=cfg["dropout"],
        pos_encoding=cfg["pos_encoding"],
        attn_kind=cfg["attn_kind"],
        norm_kind=cfg["norm_kind"],
        pad_id=PAD,
    )

state = torch.load(hf_dir / "pytorch_model.bin", map_location="cpu")
model.load_state_dict(state)
model.eval()
```

---

## Training data

- **Source:** assignment corpus `brown_cipher.txt` / `brown_plain.txt` (5000 line pairs).  
- **Split:** document-level 80/10/10, seed 42, **then** aligned 256-character chunks (~11.5k / 1.4k / 1.4k train/val/test).  
- **Property:** `len(cipher)/8 == len(plain)` so packed cipher byte `i` aligns with plaintext character `i`.

---

## Evaluation

Greedy decoding only. Test metrics (1389 chunks):

| Config | Params | Best val CE | Epochs | Bit acc. (%) | Seq. acc. (%) | Lev. | BLEU |
|--------|--------|-------------|--------|--------------|---------------|------|------|
| C1 | 8.15M | 0.0294 | 100 | 97.06 | 73.43 | 0.58 | 97.87 |
| C2 | 8.15M | 0.0188 | 82 | **99.01** | **85.89** | 0.81 | **98.75** |
| C3 | 6.97M | 0.0336 | 100 | 96.34 | 69.62 | 0.73 | 97.34 |
| C4 | 8.14M | 0.0283 | 94 | 97.15 | 73.79 | 0.54 | 97.86 |
| C5 | 11.70M | 0.0699 | 156 | 91.64 | 26.21 | 3.33 | n/a* |

\*BLEU/ROUGE are secondary for token-free C5; greedy C5 BLEU ≈ 89.3 if computed for reference.

**Ranking:** C2 > C1 ≈ C4 > C3 > C5.

**Note on C5:** bit accuracy stays relatively high while sequence (exact-match) accuracy is lower because errors are mostly scattered typos on long chunks (mean Lev ≈ 3.3). Parallel patch decoding and entropy cuts on predicted bytes amplify that gap.

---


**C1–C4**

| File | Description |
|------|-------------|
| `pytorch_model.bin` | `state_dict` of `Seq2SeqTransformer` |
| `config.json` | Architecture + hparams |
| `cipher_bpe.json` | Source BPE |
| `plain_bpe.json` | Target BPE |
| `README.md` | This model card |

**C5**

| File | Description |
|------|-------------|
| `pytorch_model.bin` | `state_dict` of `BLTSeq2Seq` |
| `config.json` | Architecture + hparams + `entropy_threshold` |
| `entropy_ngram.json` | Smoothed byte n-gram entropy model |
| `entropy_patcher.json` | Threshold / target patch size |
| `README.md` | This model card |

---

## Limitations

- Single seed (42).  
- C5 is not capacity-matched to C1 (extra byte/n-gram parameters).  
- Results are specific to this index-aligned local cipher, not general MT.  
- Not an `transformers` Auto* model; requires the course `src/` code to run.  
- Do not treat published weights as a cryptanalysis tool beyond this homework setup.

---

### References

- Vaswani et al., Attention Is All You Need, NeurIPS 2017.  
- Sennrich et al., Neural Machine Translation of Rare Words with Subword Units, ACL 2016.  
- Su et al., RoFormer (RoPE), 2021.  
- Ainslie et al., GQA, 2023.  
- Zhang & Sennrich, RMSNorm, NeurIPS 2019.  
- Pagnoni et al., Byte Latent Transformer, ACL 2025. https://aclanthology.org/2025.acl-long.453