File size: 1,589 Bytes
a2d4742 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 | ---
license: "mit"
---
# Advanced NLP, Assignment 1
```yaml
name: Vedant Kulkarni
roll: 2024101140
```
## Links
- [WandB](https://wandb.ai/gamemaker1-models/anlp-assignment-1)
- [HuggingFace](https://huggingface.co/gamemaker1/anlp-assignment-1)
## Setup
Python 3.13 and [uv](https://docs.astral.sh/uv/).
```bash
uv sync --extra cu126 # or --extra cpu on a machine without a gpu
```
The two extras select different PyTorch wheels and conflict with each
other, so install exactly one.
Download `brown_cipher.txt` and `brown_plain.txt` from the dataset link
in the assignment and put them in `data/`. Training reads those two
paths directly.
## Running
One YAML file under `cfg/` describes each configuration. To train a
model with a given configuration, run the training script.
```bash
uv run python src/train.py cfg/C1/byte-offset.yaml # base
uv run python src/train.py cfg/C2/byte-offset.yaml # rope
uv run python src/train.py cfg/C3/byte-offset.yaml # gqa
uv run python src/train.py cfg/C4/byte-offset.yaml # rmsnorm
uv run python src/train.py cfg/C5/entropy-patcher.yaml # blt
```
Training writes its checkpoint and tokenizers to `outputs/<config>/`,
logs to WandB, and scores its best checkpoint on the test split with
greedy decoding when it finishes.
To score a saved checkpoint again, download the weights, then run
the evaluation script.
```bash
uv run hf download gamemaker1/anlp-assignment-1 --local-dir outputs/
uv run python src/evaluate.py cfg/C5/entropy-patcher.yaml --samples 5
```
`--limit N` scores only the first N test lines.
|