Cipher Decryption with Transformer, Mamba, DeltaNet and a Mamba–DeltaNet Hybrid
Checkpoints for Part A of the ANLP (Monsoon 2026) Assignment 3. The task is to learn a character-level cipher from aligned plaintext/ciphertext pairs and decrypt previously unseen ciphertext.
Four architectures are trained under one identical protocol so that only the sequence mixer differs.
Each checkpoint is a self-contained torch.save dictionary holding the model weights, the training
configuration, and the character vocabularies, so no external config file is needed to reload it.
Data
Character level, built from the released train split.
- The ciphertext alphabet contains 18 punctuation symbols that are inserted noise. They are removed before modelling, which makes ciphertext and plaintext exactly one symbol per character (verified: equal length for 100% of pairs, mean length ratio 1.0000 instead of 1.1311).
- Source and target alphabets are then 63 and 27 symbols respectively, for a vocabulary of 94 tokens including padding, separator, end and unknown.
- Documents are split into aligned patches of 64 characters. Each patch carries a 4-character ciphertext warm-up so that the model sees the preceding symbols.
- Training uses a shuffled 100,000-document sample (~185 MB of JSONL), 10,000 steps at batch 16.
Model
The formulation is a per-position transducer, not an autoregressive decoder. Because the punctuation-stripped ciphertext is exactly aligned with the plaintext, the model reads a ciphertext patch and emits a 27-way distribution at every position in a single forward pass:
input : c_{i-4} … c_i (causal mixer over the patch)
output: p_i (27-way classification at the same position)
This removes the source–target alignment that an autoregressive decoder has to learn separately. It also makes inference a single pass rather than one step per output character.
| Component | Value |
|---|---|
d_model |
256 |
| Layers | 6 |
| Heads | 4 (head dim 64) |
| FFN width | 1024 |
| Dropout | 0.1 |
| Patch / warm-up | 64 / 4 characters |
| Steps / batch | 10,000 / 16 |
| Optimiser | AdamW, cosine schedule, 500-step warmup, grad clip 1.0 |
| Precision | fp16 autocast; scans and triangular solves in fp32 |
Mamba uses d_state=8, expand=4, kernel-4 causal convolution and a fused Triton selective scan.
DeltaNet uses chunk size 64 with a triangular solve. The hybrid alternates one Mamba and one DeltaNet
layer, four of each. Mamba blocks carry no separate feed-forward network, matching the paper; the other
mixers include one.
Results
Evaluated on 100 short-to-medium test documents with full-document decoding (up to 16 patches per document). Metrics are character-level; precision, recall and F1 are macro-averaged over characters.
| Model | Char acc | Word acc | Precision | Recall | F1 | Levenshtein | Params | FLOPs |
|---|---|---|---|---|---|---|---|---|
| Transformer | 8.97% | 1.0% | 54.2% | 2.14% | 1.57% | 738.0 | 4.92M | 1.37 G |
| DeltaNet | 9.15% | 6.0% | 67.5% | 2.31% | 1.91% | 736.5 | 4.82M | 1.52 G |
| Mamba | 9.21% | 9.0% | 71.9% | 2.38% | 2.04% | 736.0 | 5.17M | 2.81 G |
| Hybrid | 9.26% | 12.0% | 69.3% | 2.43% | 2.14% | 735.5 | 4.99M | 2.16 G |
Validation loss at the end of training is 2.848–2.855 for all four models, against 3.296 for a uniform distribution over the 27 plaintext symbols.
Interpretation
All four architectures land within 0.3 accuracy points of each other. The hybrid is best on five of the seven metrics and Mamba has the highest precision, but the ordering is within noise at this scale. The consistent finding across every model is the precision–recall split: precision of 54–72% with recall of only ~2%.
That pattern is a property of the cipher, not of the architectures. The ciphertext is a deterministic function of the plaintext prefix, but only the first few positions carry learnable signal: the best-guess accuracy at positions 0–3 is about 0.23 versus 0.037 chance, and it collapses to chance by position 5. Only 5.3% of validation contexts occur in training, so beyond the short prefixes the model is guessing. The models therefore learn a narrow, confident subset of the mapping and abstain elsewhere, which is exactly what the precision/recall split shows.
Decoder versus transducer
An earlier autoregressive decoder formulation, otherwise identical, reached 4.45% character accuracy with validation loss 1.43. The higher transducer loss is not a regression: the decoder conditions on the plaintext prefix it has already produced and therefore learns plaintext language modelling, which lowers teacher-forced loss without improving decryption. The transducer has no such shortcut. Its accuracy is 2.1× higher, and it evaluates roughly 50× faster.
Checkpoint format
import torch
payload = torch.load("transformer.pt", map_location="cpu", weights_only=False)
payload["model"] # state_dict
payload["config"] # training / architecture configuration
payload["source_chars"] # ciphertext vocabulary
payload["target_chars"] # plaintext vocabulary
payload["val_loss"] # best validation loss
Load with weights_only=False; the payload contains a dictionary and two string lists. The inference
code in the accompanying repository rebuilds the architecture from payload["config"] and the
vocabularies from the character lists.
Intended use and limitations
These weights are an academic artefact for comparing sequence-mixing architectures on a character-level transform. They are not suitable for securing or attacking real cryptographic systems.
The reported accuracy is low by design of the dataset, not because of an implementation failure: the mapping is deterministic but effectively unlearnable beyond the first few characters of each document, and no model, training budget or corpus size tested changed that.