File size: 3,652 Bytes
6889877 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 | ---
license: mit
language:
- en
---
<p align="center"><b><h1>Transformers from Scratch</h1></b> <br> Advanced NLP - Assignment 1 <br></p>
HuggingFace repository for Assignment 1 of ANLP 2026 course offered at IIIT Hyderabad in Monsoon 2026.
By: *Abdul Kadir Nalawala* - 2026701019 - abdulkadir[dot].nalawala[at]research[dot]iiit[dot]ac[dot]in
### Training hyper-parameters:
| Hyper-parameter | Value |
| ------------------------- | -------------: |
| `seed` | 6969 |
| `batch_size` | 32 |
| `num_epochs` | 400 |
| `lr` | 0.0005 |
| `min_lr_ratio` | 0.1 |
| `warmup_ratio` | 0.1 |
| `weight_decay` | 0.01 |
| `dropout` | 0.1 |
| `early_stopping_patience` | 5 |
| `plateau_ratio` | 0.45 |
| `label_smoothing` | 0.2 |
| `train_ratio` | 0.8 |
| `test_ratio` | 0.15 |
| `validation_ratio` | 0.05 |
| `max_validation_tokens` | 512 |
| `validation_frequency` | 200 |
| `validation_samples` | 1 |
| `tgt_lang` | `en` |
| `src_lang` | `bin` |
| `seq_len` | 512 |
| `src_vocab_size` | 4,096 |
| `tgt_vocab_size` | 1,024 |
| `tokenization` | `bpe` |
| `d_model` | 256 |
| `tokenizer_path` | `./tokenizer` |
| `ff` | 1,024 |
| `heads` | 4 |
| `kv_heads` | 2 |
| `layers` | 4 |
| `attention` | `MHA`/`GQA`|
| `Norm` | `layernorm`/`rmsnorm` |
| `pos_encoding` | `rope`/`sinusoidal` |
| `model_filename` | `./weights/{config}}` |
| `model_folder` | `weights` |
### Results
| Configuration | Final Training Loss | Final Validation Loss | Epochs Trained |
| ------------- | ------------------: | --------------------: | -------------: |
| C1 (Base) | 1.287 | 1.013 | 294 |
| C2 (RoPE) | **1.179** | **0.605** | 156 |
| C3 (GQA) | 1.311 | 1.191 | 323 |
| C4 (RMSNorm) | 1.241 | 1.161 | 313 |
| Configuration | Bit Acc. | Seq. Acc. | BLEU (%) | ROUGE-1 | ROUGE-2 | ROUGE-L | ROUGE-Lsum |
| ------------- | --------: | --------: | ---------: | --------: | --------: | --------: | ---------: |
| C1 (Base) | 0.658 | 0.005 | 53.172 | 0.754 | 0.605 | 0.748 | 0.748 |
| C2 (RoPE) | **0.704** | **0.023** | **77.086** | **0.885** | **0.820** | **0.882** | **0.881** |
| C3 (GQA) | 0.654 | 0.005 | 49.969 | 0.737 | 0.577 | 0.730 | 0.730 |
| C4 (RMSNorm) | 0.659 | 0.007 | 53.512 | 0.756 | 0.607 | 0.749 | 0.749 |
### Repository structure:
```markdown
ANLP-Assignment1/
├── tokenizers/
│ ├── bin_4096.json ## source tokenizer - binary
│ └── en_1024.json ## target tokenizer - english
│
└── weights/ # best checkpoints for each config
├── C1_best.pt
├── C2_best.pt
├── C3_best.pt
└── C4_best.pt
```
### [WandB Report](https://api.wandb.ai/links/abdulper/1lljz0qj) |