Hangman CANINE Multitask Model
This is a Google CANINE based character model fine-tuned for English movie-title Hangman.
The model predicts the next useful letter from a partial board, such as:
QUIZ SHOW -> Q___ S___
It uses two prediction heads:
- MLM head: predicts hidden characters at each blank position.
- Letter head: predicts which alphabet letters are still present anywhere in the title.
In gameplay, the MLM head performed best, so the default inference setting is:
letter_weight = 0.0
mlm_weight = 1.0
Model Details
| Item | Value |
|---|---|
| Architecture | Custom CANINE multitask model |
| Base encoder | Google CANINE |
| Input | CANINE tokenizer output, attention mask, missed-letter vector |
| Output | Per-position MLM logits and 26-way letter logits |
| Task | Hangman next-letter prediction |
| Training data | English movie titles |
| Dataset | anilsathyan7/hangman-movie-titles |
This checkpoint uses custom model classes and gameplay code from the project repository. It is not directly loadable with AutoModel alone.
Training
The model was trained with Hugging Face Trainer on dynamically generated Hangman game states.
Main setup:
- Fixed train/validation/test split
- Dynamic board masking
- Missed-letter input
- 30% late-game sampling
- Full CANINE fine-tuning
- Validation loss used for best checkpoint selection
- Weights & Biases used for logging
Evaluation
Gameplay evaluation used the held-out test split from the Hangman movie-title dataset.
| Index Fallback | Win Rate | Avg Fails | Avg Guesses | Avg Score |
|---|---|---|---|---|
| No | 0.9189 | 1.6266 | 5.9201 | 0.7829 |
| Yes | 0.9754 | 1.0419 | 5.3435 | 0.8551 |
Best validation checkpoint metrics:
| Metric | Value |
|---|---|
| Eval loss | 0.9735 |
| MLM masked accuracy | 0.7342 |
| Letter top-1 accuracy | 0.9244 |
Usage
This model needs the project code to run inference or evaluation.
Clone the project repository, place this checkpoint as the model directory, and use the project scripts described in the project README.
Github: https://github.com/anilsathyan7/hangman-ai
Limitations
- Supports English alphabetic movie titles only.
- Numbers and special characters are removed from the training data.
- Very noisy or invented title words are difficult to predict reliably.
- CANINE reached stronger validation metrics, but SlimBERT performed better in full gameplay evaluation.