YAML Metadata Warning:empty or missing yaml metadata in repo card
Check out the documentation for more information.
ANLP Assignment 1: Transformers from Scratch, Ablations, and BLT
Encoder-decoder transformer built from basic PyTorch ops that learns to map
encrypted binary sequences (brown_cipher.txt) to plaintext English
(brown_plain.txt), plus a five-way ablation study over positional
encodings (Sinusoidal vs RoPE), attention (MHA vs GQA), normalization
(LayerNorm vs RMSNorm), and tokenization (subword vs token-free BLT).
C1–C4 use independently learned, from-scratch BPE tokenizers: 512 source tokens over ciphertext bytes and 1,024 target tokens over plaintext. Merged tokens retain their underlying byte/character offsets for positional encoding. C5 groups ciphertext bits into bytes and uses train-only Markov next-byte entropy to create dynamic patches.
Setup
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
wandb login # for training-run logging
huggingface-cli login # for checkpoint uploads
Place the dataset files in the repo root (they are git-ignored and never committed):
ANLP-A1/
├── brown_cipher.txt
├── brown_plain.txt
└── ...
(Or edit cipher_path / plain_path in the configs.)
Running
Train one configuration:
python -m src.train --config configs/c1_base.yaml
Train all five ablation configs:
bash scripts/run_all.sh
Ablation configurations
| Config | Change from base | File |
|---|---|---|
| C1 | none (base) | configs/c1_base.yaml |
| C2 | RoPE positional encoding | configs/c2_rope.yaml |
| C3 | Grouped-Query Attention | configs/c3_gqa.yaml |
| C4 | RMSNorm | configs/c4_rmsnorm.yaml |
| C5 | BLT (token-free) | configs/c5_blt.yaml |
All other hyperparameters are identical across configs.
Repository layout
src/
├── models/
│ ├── attention.py # Scaled dot-product attention, MHA, GQA
│ ├── positional.py # Sinusoidal PE, RoPE
│ ├── norm.py # LayerNorm, RMSNorm (from scratch)
│ ├── transformer.py # Encoder/decoder layers, Seq2SeqTransformer
│ └── blt.py # Local encoder/decoder + BLT model (C5)
├── dataset.py # Tokenized (C1-C4) and token-free (C5) loaders
├── train.py # Config-driven training loop with WandB
└── utils.py # Greedy decoding, metrics, plots
configs/ # One YAML per ablation config
outputs/ # Plots, logs, metrics (checkpoints git-ignored)
scripts/run_all.sh # Run C1-C5 sequentially
Links
WandB project: https://wandb.ai/yajatlakhanpal31-iiit-hyderabad/anlp-a1/workspace?nw=nwuseryajatlakhanpal31
Hugging Face checkpoints: https://huggingface.co/Yajat31/anlp-a1-checkpoints
Evaluation
All reported numbers use greedy decoding. Metrics: bit-level accuracy,
sequence accuracy, Levenshtein distance, and BLEU/ROUGE (tokenized
models only). See src/utils.py.
Final test BLEU: C1 84.93, C2 98.68, C3 63.66, and C4 83.27. C5 reaches 0.726 bit accuracy while using the lowest peak GPU memory (1,333 MB).