neemon commited on
Commit
dcdb40b
·
verified ·
1 Parent(s): 3d980d0

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +54 -0
README.md ADDED
@@ -0,0 +1,54 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: mit
3
+ tags:
4
+ - transformer
5
+ - from-scratch
6
+ - ablation
7
+ - cipher-decryption
8
+ ---
9
+
10
+ # ANLP Assignment 1 — Configuration C2
11
+
12
+ positional encoding: sinusoidal -> RoPE
13
+
14
+ Encoder-decoder Transformer built from basic PyTorch operations (no `nn.Transformer` or
15
+ `nn.MultiheadAttention`), trained to decrypt binary cipher text into English plaintext.
16
+
17
+ One of five configurations in a controlled ablation. Each of C2-C5 changes exactly one
18
+ component from the C1 base; every other hyperparameter is identical across all five runs.
19
+
20
+ ## This configuration
21
+
22
+ | Axis | Setting |
23
+ |---|---|
24
+ | Positional encoding | rope |
25
+ | Attention | mha |
26
+ | Normalization | layernorm |
27
+ | Tokenization | subword |
28
+
29
+ ## Shared hyperparameters
30
+
31
+ ```
32
+ d_model 256 | 4 encoder layers | 4 decoder layers
33
+ 8 heads | d_ff 1024 | dropout 0.1 | pre-norm
34
+ AdamW lr 0.0003 | warmup 1000 | 15000 steps | batch 64 | seed 42
35
+ fp32 (no AMP)
36
+ ```
37
+
38
+ ## Results
39
+
40
+ | Metric | Value |
41
+ |---|---|
42
+ | Bit-level accuracy | 0.9918 |
43
+ | Sequence accuracy | 0.9024 |
44
+ | Levenshtein distance | 0.16 |
45
+ | BLEU | 97.98 |
46
+ | ROUGE-L | 0.9886 |
47
+ | Parameters | 9,524,992 |
48
+ | Peak GPU memory (MB) | 1277.4 |
49
+ | Seconds per training step | 0.0752 |
50
+
51
+ ## Usage
52
+
53
+ Checkpoints are plain PyTorch state dicts. See the accompanying code repository for the
54
+ model definition; `torch.load(...)["model"]` loads into `build_model(get_config("c2"), ...)`.