ANLP Assignment 1: Configuration C2

positional encoding: sinusoidal -> RoPE

Encoder-decoder Transformer built from basic PyTorch operations, with attention written out by hand rather than taken from PyTorch's built-in transformer modules. Trained to decrypt binary cipher text into English plaintext.

One of five configurations in a controlled ablation. Each of C2-C5 changes exactly one component from the C1 base; every other hyperparameter is identical across all five runs.

This configuration

Axis Setting
Positional encoding rope
Attention mha
Normalization layernorm
Tokenization subword

Shared hyperparameters

d_model 256 | 4 encoder layers | 4 decoder layers
8 heads | d_ff 1024 | dropout 0.1 | pre-norm
AdamW lr 0.0003 | warmup 1000 | 15000 steps | batch 64 | seed 42
fp32 (no AMP)

Results

Metric Value
Bit-level accuracy 0.90806
Sequence accuracy 0.3487
Levenshtein distance 2.69
BLEU 78.59
ROUGE-L 0.8924
Parameters 10,507,264
Peak GPU memory (MB) 1351.4
Seconds per training step 0.0687

Usage

Checkpoints are plain PyTorch state dicts. See the accompanying code repository for the model definition; torch.load(...)["model"] loads into build_model(get_config("c2"), ...).

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support