Upload MODEL_CARD.md

MODEL_CARD.md

@@ -0,0 +1,201 @@
+---
+language:
+- en
+license: mit
+library_name: pytorch
+tags:
+- transformers-from-scratch
+- encoder-decoder
+- seq2seq
+- ablation
+- bpe
+- rope
+- gqa
+- rmsnorm
+- blt
+- anlp
+- ciphertext-decryption
+datasets:
+- custom (brown_cipher / brown_plain, ANLP Assignment 1)
+metrics:
+- accuracy
+- bleu
+- rouge
+- levenshtein
+pipeline_tag: text2text-generation
+---
+
+# ANLP Assignment 1 — Encoder–Decoder Ablations (C1–C5)
+
+This hub entry hosts checkpoints from a controlled ablation: map encrypted binary sequences (`brown_cipher.txt`) to English plaintext (`brown_plain.txt`). Configuration **C1** is the base model; **C2–C5** each change exactly one component.
+
+---
+
+## Model variants
+
+| Config | Single change vs C1 | Architecture class |
+|--------|---------------------|--------------------|
+| **C1** | — (base) | `Seq2SeqTransformer` |
+| **C2** | RoPE instead of sinusoidal PE | `Seq2SeqTransformer` |
+| **C3** | GQA (8 Q, 2 KV) instead of MHA | `Seq2SeqTransformer` |
+| **C4** | RMSNorm instead of LayerNorm | `Seq2SeqTransformer` |
+| **C5** | Entropy-patched BLT instead of BPE | `BLTSeq2Seq` |
+
+### Shared hyperparameters
+
+| Hyperparameter | Value |
+|----------------|-------|
+| `d_model` | 256 |
+| Heads | 8 (C3: 2 KV heads) |
+| Encoder / decoder layers | 4 / 4 |
+| `d_ff` | 1024 |
+| Dropout | 0.15 |
+| Optimizer | AdamW, lr 1e-3, wd 0.01, betas (0.9, 0.98) |
+| LR schedule | cosine + 400 warmup steps |
+| Batch size | 64 |
+| Chunk size | 256 characters (aligned cipher/plain) |
+| Max epochs | 100 (C5 early-stopped later; best @ epoch 146) |
+| Early stopping | patience 10 on val CE |
+| Seed | 42 |
+| Precision | AMP (fp16) |
+
+**Tokenization**
+
+- **C1–C4:** from-scratch BPE, vocab 1024 per side. Cipher: pack 8 bits → byte, then learn merges. Plain: character BPE. Artifacts: `cipher_bpe.json`, `plain_bpe.json`.
+- **C5:** no BPE. Packed bytes + entropy patches (len 1–8, mean ≈ 4.2), hashed 2/3/4-gram features. Artifacts: `entropy_ngram.json`, `entropy_patcher.json`.
+
+---
+
+## Intended use
+
+- **Primary:** course ablation / reproducibility for ANLP Assignment 1.  
+- **Task:** greedy decode ciphertext chunks → plaintext chunks.  
+- **Not intended for:** production MT, open-domain generation, or claiming a general “decrypt any cipher” system. The mapping is specific to this homework dataset.
+
+---
+
+## How to load weights
+
+Checkpoints are raw PyTorch state dicts (`pytorch_model.bin`) plus `config.json`. They are **not** Hugging Face `transformers.AutoModel` checkpoints—load with the assignment code.
+
+```python
+import json
+import torch
+from pathlib import Path
+
+# after cloning / downloading this repo's `src/` onto PYTHONPATH
+from models.transformer import Seq2SeqTransformer
+from models.blt import BLTSeq2Seq
+from tokenizer import BPETokenizer, PAD
+
+hf_dir = Path("path/to/downloaded/hf")  # folder with config.json + pytorch_model.bin
+cfg = json.loads((hf_dir / "config.json").read_text())
+
+if cfg.get("blt"):
+    model = BLTSeq2Seq(
+        d_model=cfg["d_model"],
+        n_heads=cfg["n_heads"],
+        n_kv_heads=cfg["n_kv_heads"],
+        n_encoder_layers=cfg["n_encoder_layers"],
+        n_decoder_layers=cfg["n_decoder_layers"],
+        d_ff=cfg["d_ff"],
+        dropout=cfg["dropout"],
+        patch_size=cfg["patch_size"],
+        n_local_layers=cfg["n_local_layers"],
+        n_local_heads=cfg["n_local_heads"],
+        pad_id=PAD,
+    )
+else:
+    model = Seq2SeqTransformer(
+        src_vocab=cfg["src_vocab"],
+        tgt_vocab=cfg["tgt_vocab"],
+        d_model=cfg["d_model"],
+        n_heads=cfg["n_heads"],
+        n_kv_heads=cfg["n_kv_heads"],
+        n_encoder_layers=cfg["n_encoder_layers"],
+        n_decoder_layers=cfg["n_decoder_layers"],
+        d_ff=cfg["d_ff"],
+        dropout=cfg["dropout"],
+        pos_encoding=cfg["pos_encoding"],
+        attn_kind=cfg["attn_kind"],
+        norm_kind=cfg["norm_kind"],
+        pad_id=PAD,
+    )
+
+state = torch.load(hf_dir / "pytorch_model.bin", map_location="cpu")
+model.load_state_dict(state)
+model.eval()
+```
+
+---
+
+## Training data
+
+- **Source:** assignment corpus `brown_cipher.txt` / `brown_plain.txt` (5000 line pairs).  
+- **Split:** document-level 80/10/10, seed 42, **then** aligned 256-character chunks (~11.5k / 1.4k / 1.4k train/val/test).  
+- **Property:** `len(cipher)/8 == len(plain)` so packed cipher byte `i` aligns with plaintext character `i`.
+
+---
+
+## Evaluation
+
+Greedy decoding only. Test metrics (1389 chunks):
+
+| Config | Params | Best val CE | Epochs | Bit acc. (%) | Seq. acc. (%) | Lev. | BLEU |
+|--------|--------|-------------|--------|--------------|---------------|------|------|
+| C1 | 8.15M | 0.0294 | 100 | 97.06 | 73.43 | 0.58 | 97.87 |
+| C2 | 8.15M | 0.0188 | 82 | **99.01** | **85.89** | 0.81 | **98.75** |
+| C3 | 6.97M | 0.0336 | 100 | 96.34 | 69.62 | 0.73 | 97.34 |
+| C4 | 8.14M | 0.0283 | 94 | 97.15 | 73.79 | 0.54 | 97.86 |
+| C5 | 11.70M | 0.0699 | 156 | 91.64 | 26.21 | 3.33 | n/a* |
+
+\*BLEU/ROUGE are secondary for token-free C5; greedy C5 BLEU ≈ 89.3 if computed for reference.
+
+**Ranking:** C2 > C1 ≈ C4 > C3 > C5.
+
+**Note on C5:** bit accuracy stays relatively high while sequence (exact-match) accuracy is lower because errors are mostly scattered typos on long chunks (mean Lev ≈ 3.3). Parallel patch decoding and entropy cuts on predicted bytes amplify that gap.
+
+---
+
+
+**C1–C4**
+
+| File | Description |
+|------|-------------|
+| `pytorch_model.bin` | `state_dict` of `Seq2SeqTransformer` |
+| `config.json` | Architecture + hparams |
+| `cipher_bpe.json` | Source BPE |
+| `plain_bpe.json` | Target BPE |
+| `README.md` | This model card |
+
+**C5**
+
+| File | Description |
+|------|-------------|
+| `pytorch_model.bin` | `state_dict` of `BLTSeq2Seq` |
+| `config.json` | Architecture + hparams + `entropy_threshold` |
+| `entropy_ngram.json` | Smoothed byte n-gram entropy model |
+| `entropy_patcher.json` | Threshold / target patch size |
+| `README.md` | This model card |
+
+---
+
+## Limitations
+
+- Single seed (42).  
+- C5 is not capacity-matched to C1 (extra byte/n-gram parameters).  
+- Results are specific to this index-aligned local cipher, not general MT.  
+- Not an `transformers` Auto* model; requires the course `src/` code to run.  
+- Do not treat published weights as a cryptanalysis tool beyond this homework setup.
+
+---
+
+### References
+
+- Vaswani et al., Attention Is All You Need, NeurIPS 2017.  
+- Sennrich et al., Neural Machine Translation of Rare Words with Subword Units, ACL 2016.  
+- Su et al., RoFormer (RoPE), 2021.  
+- Ainslie et al., GQA, 2023.  
+- Zhang & Sennrich, RMSNorm, NeurIPS 2019.  
+- Pagnoni et al., Byte Latent Transformer, ACL 2025. https://aclanthology.org/2025.acl-long.453
+