Neural Networks Coursework (HW3) - Transformer from Scratch on Shakespeare

Implementation of a decoder-only autoregressive Transformer language model trained from scratch on the complete Shakespeare dialogue corpus (105,152 clean dialogue lines across 36 plays).

GitHub Repository: Andrefty/NN_HW3
Author: Fărcășanu Tudor Andrei

1. Project Overview & Submission Timeline

This repository preserves the complete experimental history and trained model weights for HW3, including the progression across deadline phases:

Timeline & Progression

Pre-Deadline Initial Submission (pre_deadline_initial/ - from hw3_final_code/):
- Standard 80/10/10 split across the aggregated corpus.
- Character models converged well (PPL ~3.46), but subword models suffered from vocabulary mismatch across disparate plays (PPL 75–84).
Post-Extension Final Submission ("Approach 2" - approach2_stratified_final/ - from hw3_approach2_code/):
- Following a deadline extension, introduced Stratified by Play Splitting: each of the 36 Shakespeare plays was split 80/10/10 independently and then merged.
- Slashed subword test perplexity from 75 down to **36.0 PPL**, while character models improved to 3.22 PPL.
- This is the definitive version submitted and reported in the official 10-page report (hw3_Farcasanu-Tudor_Andrei_report.pdf).
Post-Deadline Exploration (run8_sliding_window/ - from hw3_final_code_r8/):
- Explored sliding-window tokenization prior to shuffle/split, reaching character PPL of 3.11.
Historical Iterations (historical_runs/):
- Preserves earlier training run checkpoints and Slurm logs (Runs 04, 05, 06, 07).

2. Model Architecture & Configurations

Decoder-only Transformer with causal self-attention, learned positional embeddings, pre-LayerNorm, and GELU feed-forward networks (intermediate dim = 4× hidden dim).

Parameter	Small Config	Large Config
Hidden Dimension ($d_{model}$)	384	512
Layers	6	8
Attention Heads	6	8
FFN Dimension	1536	2048
Max Context Length	1024 (char) / 900 (subword)	1024 (char) / 900 (subword)
Optimizer	AdamW ($eta_1=0.9, eta_2=0.95$, weight decay 0.1)	AdamW ($eta_1=0.9, eta_2=0.95$, weight decay 0.1)
LR Schedule	Cosine annealing with warmup	Cosine annealing with warmup

Tokenization Schemes

Character Tokenizer: Custom char-level vocabulary of 81 unique characters.
Subword Tokenizer: Llama-2 BPE tokenizer (32,004 vocabulary size).

3. Evaluation Results (Official Final Report - Approach 2)

Test Perplexity

Model	Test Loss	Test Perplexity	Best Epoch	Checkpoint Path
`small_char`	1.1701	3.2220	61	`approach2_stratified_final/small_char/best.pt`
`large_char`	1.1839	3.2669	32	`approach2_stratified_final/large_char/best.pt`
`small_subword`	3.5839	36.0073	287	`approach2_stratified_final/small_subword/best.pt`
`large_subword`	3.6052	36.7890	199	`approach2_stratified_final/large_subword/best.pt`

Generation Quality (BLEU & ROUGE on 20 Test Samples, Top-k=10, T=1.0)

Model	BLEU-1	BLEU-2	ROUGE-1	ROUGE-2	ROUGE-L
`small_char`	0.1104	0.0056	0.1117	0.0056	0.0800
`large_char`	0.1147	0.0043	0.1139	0.0039	0.0777
`small_subword`	0.1440	0.0068	0.1198	0.0053	0.0878
`large_subword`	0.1494	0.0026	0.1291	0.0023	0.1023

4. Qualitative Dialogue Generation Samples

Prompt 1: `TITUS ANDRONICUS: O gracious emperor! O gentle Aar`

Expected: on! Did ever raven sing so like a lark, That gives sweet tidings of the sun's uprise?...
small_char: onia! TAMORA: If there all the water of Rome are cry, LUCIUS: Still be another in your push. I prithee, Revenge, I warrant...
large_char: on, bring your worships, AARON: Ay, it is my life. MARCUS ANDRONICUS: Then, for thou hast but by this same to lend: TITUS ANDRONICUS: By heaven...
small_subword: , and, TAMORA: And I am, to see the emperor's name, TAMORA: And I have done in heaven to the Goth LAVINIA: Aaron of the earth's heart...
large_subword: AETRIUS: And that she hath been, QUINTUS: Thou hastst not be thy brother's life, TITUS ANDRONICUS: That I do thee to the Capitol? AARON: I have no more...

BONUS Generation: Romeo in Hamlet (`large_char`, Top-k=10, T=0.8)

HAMLET: To be, or not to be, that is the question.
ROMEO: That, sir, help me, she shall leave thee at the wind.
FRIAR LAURENCE: And be sure how do you not this? What is't thy heart?
BENVOLIO: And they were not to the promises of thine eye,
FRIAR LAURENCE: Be patient all the wretched bands of her.

5. Repository File Structure

approach2_stratified_final/: The 4 official reported models (best.pt, latest.pt), evaluation metrics JSONs, and training curve plots.
pre_deadline_initial/: Checkpoints, metrics, and plots from the initial pre-extension submission.
run8_sliding_window/: Post-deadline Run 8 exploration checkpoints and metrics.
historical_runs/: Archive tarballs for Run 04, Run 05, Run 06, and uncompressed Run 07 checkpoints.

Downloads last month: -; Downloads are not tracked for this model. How to track