| --- |
| license: "mit" |
| --- |
| |
| # Advanced NLP, Assignment 1 |
|
|
| ```yaml |
| name: Vedant Kulkarni |
| roll: 2024101140 |
| ``` |
|
|
| ## Links |
|
|
| - [WandB](https://wandb.ai/gamemaker1-models/anlp-assignment-1) |
| - [HuggingFace](https://huggingface.co/gamemaker1/anlp-assignment-1) |
|
|
| ## Setup |
|
|
| Python 3.13 and [uv](https://docs.astral.sh/uv/). |
|
|
| ```bash |
| uv sync --extra cu126 # or --extra cpu on a machine without a gpu |
| ``` |
|
|
| The two extras select different PyTorch wheels and conflict with each |
| other, so install exactly one. |
|
|
| Download `brown_cipher.txt` and `brown_plain.txt` from the dataset link |
| in the assignment and put them in `data/`. Training reads those two |
| paths directly. |
|
|
| ## Running |
|
|
| One YAML file under `cfg/` describes each configuration. To train a |
| model with a given configuration, run the training script. |
|
|
| ```bash |
| uv run python src/train.py cfg/C1/byte-offset.yaml # base |
| uv run python src/train.py cfg/C2/byte-offset.yaml # rope |
| uv run python src/train.py cfg/C3/byte-offset.yaml # gqa |
| uv run python src/train.py cfg/C4/byte-offset.yaml # rmsnorm |
| uv run python src/train.py cfg/C5/entropy-patcher.yaml # blt |
| ``` |
|
|
| Training writes its checkpoint and tokenizers to `outputs/<config>/`, |
| logs to WandB, and scores its best checkpoint on the test split with |
| greedy decoding when it finishes. |
|
|
| To score a saved checkpoint again, download the weights, then run |
| the evaluation script. |
|
|
| ```bash |
| uv run hf download gamemaker1/anlp-assignment-1 --local-dir outputs/ |
| uv run python src/evaluate.py cfg/C5/entropy-patcher.yaml --samples 5 |
| ``` |
|
|
| `--limit N` scores only the first N test lines. |
|
|
