---
license: "mit"
---

# Advanced NLP, Assignment 1

```yaml
name: Vedant Kulkarni
roll: 2024101140
```

## Links

- [WandB](https://wandb.ai/gamemaker1-models/anlp-assignment-1)
- [HuggingFace](https://huggingface.co/gamemaker1/anlp-assignment-1)

## Setup

Python 3.13 and [uv](https://docs.astral.sh/uv/).

```bash
uv sync --extra cu126   # or --extra cpu on a machine without a gpu
```

The two extras select different PyTorch wheels and conflict with each
other, so install exactly one.

Download `brown_cipher.txt` and `brown_plain.txt` from the dataset link
in the assignment and put them in `data/`. Training reads those two
paths directly.

## Running

One YAML file under `cfg/` describes each configuration. To train a
model with a given configuration, run the training script.

```bash
uv run python src/train.py cfg/C1/byte-offset.yaml       # base
uv run python src/train.py cfg/C2/byte-offset.yaml       # rope
uv run python src/train.py cfg/C3/byte-offset.yaml       # gqa
uv run python src/train.py cfg/C4/byte-offset.yaml       # rmsnorm
uv run python src/train.py cfg/C5/entropy-patcher.yaml   # blt
```

Training writes its checkpoint and tokenizers to `outputs/<config>/`,
logs to WandB, and scores its best checkpoint on the test split with
greedy decoding when it finishes.

To score a saved checkpoint again, download the weights, then run
the evaluation script.

```bash
uv run hf download gamemaker1/anlp-assignment-1 --local-dir outputs/
uv run python src/evaluate.py cfg/C5/entropy-patcher.yaml --samples 5
```

`--limit N` scores only the first N test lines.