---
license: mit
language:
- en
---

<p align="center"><b><h1>Transformers from Scratch</h1></b> <br> Advanced NLP - Assignment 1 <br></p>

HuggingFace repository for Assignment 1 of ANLP 2026 course offered at IIIT Hyderabad in Monsoon 2026.

By: *Abdul Kadir Nalawala* - 2026701019 - abdulkadir[dot].nalawala[at]research[dot]iiit[dot]ac[dot]in


### Training hyper-parameters:

| Hyper-parameter           |          Value |
| ------------------------- | -------------: |
| `seed`                    |          6969 |
| `batch_size`              |             32 |
| `num_epochs`              |            400 |
| `lr`                      |         0.0005 |
| `min_lr_ratio`            |            0.1 |
| `warmup_ratio`            |            0.1 |
| `weight_decay`            |           0.01 |
| `dropout`                 |            0.1 |
| `early_stopping_patience` |              5 |
| `plateau_ratio`           |           0.45 |
| `label_smoothing`         |            0.2 |
| `train_ratio`             |            0.8 |
| `test_ratio`              |           0.15 |
| `validation_ratio`        |           0.05 |
| `max_validation_tokens`   |            512 |
| `validation_frequency`    |            200 |
| `validation_samples`      |              1 |
| `tgt_lang`                |           `en` |
| `src_lang`                |          `bin` |
| `seq_len`                 |            512 |
| `src_vocab_size`          |          4,096 |
| `tgt_vocab_size`          |          1,024 |
| `tokenization`            |          `bpe` |
| `d_model`                 |            256 |
| `tokenizer_path`          |  `./tokenizer` |
| `ff`                      |          1,024 |
| `heads`                   |              4 |
| `kv_heads`                |              2 |
| `layers`                  |              4 |
| `attention`               |     `MHA`/`GQA`|
| `Norm`                    |    `layernorm`/`rmsnorm` |
| `pos_encoding`            |         `rope`/`sinusoidal` |
| `model_filename`          | `./weights/{config}}` |
| `model_folder`            |      `weights` |


### Results
| Configuration | Final Training Loss | Final Validation Loss | Epochs Trained |
| ------------- | ------------------: | --------------------: | -------------: |
| C1 (Base)     |               1.287 |                 1.013 |            294 |
| C2 (RoPE)     |           **1.179** |             **0.605** |            156 |
| C3 (GQA)      |               1.311 |                 1.191 |            323 |
| C4 (RMSNorm)  |               1.241 |                 1.161 |            313 |

| Configuration |  Bit Acc. | Seq. Acc. |   BLEU (%) |   ROUGE-1 |   ROUGE-2 |   ROUGE-L | ROUGE-Lsum |
| ------------- | --------: | --------: | ---------: | --------: | --------: | --------: | ---------: |
| C1 (Base)     |     0.658 |     0.005 |     53.172 |     0.754 |     0.605 |     0.748 |      0.748 |
| C2 (RoPE)     | **0.704** | **0.023** | **77.086** | **0.885** | **0.820** | **0.882** |  **0.881** |
| C3 (GQA)      |     0.654 |     0.005 |     49.969 |     0.737 |     0.577 |     0.730 |      0.730 |
| C4 (RMSNorm)  |     0.659 |     0.007 |     53.512 |     0.756 |     0.607 |     0.749 |      0.749 |


### Repository structure:

```markdown
ANLP-Assignment1/
├── tokenizers/
│   ├── bin_4096.json ## source tokenizer - binary
│   └── en_1024.json ## target tokenizer - english
│
└── weights/ # best checkpoints for each config
    ├── C1_best.pt
    ├── C2_best.pt
    ├── C3_best.pt
    └── C4_best.pt
```

### [WandB Report](https://api.wandb.ai/links/abdulper/1lljz0qj)