Create README.md
Browse files
README.md
ADDED
|
@@ -0,0 +1,85 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: mit
|
| 3 |
+
language:
|
| 4 |
+
- en
|
| 5 |
+
---
|
| 6 |
+
|
| 7 |
+
<p align="center"><b><h1>Transformers from Scratch</h1></b> <br> Advanced NLP - Assignment 1 <br></p>
|
| 8 |
+
|
| 9 |
+
HuggingFace repository for Assignment 1 of ANLP 2026 course offered at IIIT Hyderabad in Monsoon 2026.
|
| 10 |
+
|
| 11 |
+
By: *Abdul Kadir Nalawala* - 2026701019 - abdulkadir[dot].nalawala[at]research[dot]iiit[dot]ac[dot]in
|
| 12 |
+
|
| 13 |
+
|
| 14 |
+
### Training hyper-parameters:
|
| 15 |
+
|
| 16 |
+
| Hyper-parameter | Value |
|
| 17 |
+
| ------------------------- | -------------: |
|
| 18 |
+
| `seed` | 6969 |
|
| 19 |
+
| `batch_size` | 32 |
|
| 20 |
+
| `num_epochs` | 400 |
|
| 21 |
+
| `lr` | 0.0005 |
|
| 22 |
+
| `min_lr_ratio` | 0.1 |
|
| 23 |
+
| `warmup_ratio` | 0.1 |
|
| 24 |
+
| `weight_decay` | 0.01 |
|
| 25 |
+
| `dropout` | 0.1 |
|
| 26 |
+
| `early_stopping_patience` | 5 |
|
| 27 |
+
| `plateau_ratio` | 0.45 |
|
| 28 |
+
| `label_smoothing` | 0.2 |
|
| 29 |
+
| `train_ratio` | 0.8 |
|
| 30 |
+
| `test_ratio` | 0.15 |
|
| 31 |
+
| `validation_ratio` | 0.05 |
|
| 32 |
+
| `max_validation_tokens` | 512 |
|
| 33 |
+
| `validation_frequency` | 200 |
|
| 34 |
+
| `validation_samples` | 1 |
|
| 35 |
+
| `tgt_lang` | `en` |
|
| 36 |
+
| `src_lang` | `bin` |
|
| 37 |
+
| `seq_len` | 512 |
|
| 38 |
+
| `src_vocab_size` | 4,096 |
|
| 39 |
+
| `tgt_vocab_size` | 1,024 |
|
| 40 |
+
| `tokenization` | `bpe` |
|
| 41 |
+
| `d_model` | 256 |
|
| 42 |
+
| `tokenizer_path` | `./tokenizer` |
|
| 43 |
+
| `ff` | 1,024 |
|
| 44 |
+
| `heads` | 4 |
|
| 45 |
+
| `kv_heads` | 2 |
|
| 46 |
+
| `layers` | 4 |
|
| 47 |
+
| `attention` | `MHA`/`GQA`|
|
| 48 |
+
| `Norm` | `layernorm`/`rmsnorm` |
|
| 49 |
+
| `pos_encoding` | `rope`/`sinusoidal` |
|
| 50 |
+
| `model_filename` | `./weights/{config}}` |
|
| 51 |
+
| `model_folder` | `weights` |
|
| 52 |
+
|
| 53 |
+
|
| 54 |
+
### Results
|
| 55 |
+
| Configuration | Final Training Loss | Final Validation Loss | Epochs Trained |
|
| 56 |
+
| ------------- | ------------------: | --------------------: | -------------: |
|
| 57 |
+
| C1 (Base) | 1.287 | 1.013 | 294 |
|
| 58 |
+
| C2 (RoPE) | **1.179** | **0.605** | 156 |
|
| 59 |
+
| C3 (GQA) | 1.311 | 1.191 | 323 |
|
| 60 |
+
| C4 (RMSNorm) | 1.241 | 1.161 | 313 |
|
| 61 |
+
|
| 62 |
+
| Configuration | Bit Acc. | Seq. Acc. | BLEU (%) | ROUGE-1 | ROUGE-2 | ROUGE-L | ROUGE-Lsum |
|
| 63 |
+
| ------------- | --------: | --------: | ---------: | --------: | --------: | --------: | ---------: |
|
| 64 |
+
| C1 (Base) | 0.658 | 0.005 | 53.172 | 0.754 | 0.605 | 0.748 | 0.748 |
|
| 65 |
+
| C2 (RoPE) | **0.704** | **0.023** | **77.086** | **0.885** | **0.820** | **0.882** | **0.881** |
|
| 66 |
+
| C3 (GQA) | 0.654 | 0.005 | 49.969 | 0.737 | 0.577 | 0.730 | 0.730 |
|
| 67 |
+
| C4 (RMSNorm) | 0.659 | 0.007 | 53.512 | 0.756 | 0.607 | 0.749 | 0.749 |
|
| 68 |
+
|
| 69 |
+
|
| 70 |
+
### Repository structure:
|
| 71 |
+
|
| 72 |
+
```markdown
|
| 73 |
+
ANLP-Assignment1/
|
| 74 |
+
├── tokenizers/
|
| 75 |
+
│ ├── bin_4096.json ## source tokenizer - binary
|
| 76 |
+
│ └── en_1024.json ## target tokenizer - english
|
| 77 |
+
│
|
| 78 |
+
└── weights/ # best checkpoints for each config
|
| 79 |
+
├── C1_best.pt
|
| 80 |
+
├── C2_best.pt
|
| 81 |
+
├── C3_best.pt
|
| 82 |
+
└── C4_best.pt
|
| 83 |
+
```
|
| 84 |
+
|
| 85 |
+
### [WandB Report](https://api.wandb.ai/links/abdulper/1lljz0qj)
|