prefix-40000: the checkpoint the daemon loads, its vocabulary, training figures and model card
9d0f6fd verified | base_model: distilbert-base-uncased | |
| datasets: | |
| - Salesforce/wikitext | |
| library_name: pytorch | |
| tags: | |
| - chorded-typing | |
| - word-prediction | |
| - keyboard | |
| # stenographer: prefix-40000 | |
| The word model for [stenographer](https://github.com/maximus-powers/stenographer). You type a word by pressing its opening letters at once, like a stenographer. This model works out which word you meant from those letters and the sentence so far. | |
| This is the checkpoint the daemon loads by default (`scripts/stenod.py`, `--checkpoint runs/prefix-40000/checkpoint.pt`). | |
| ## What it is | |
| - DistilBERT (`distilbert-base-uncased`) encodes the sentence so far, with the word being typed masked. | |
| - A linear head scores the vocabulary. | |
| - At typing time, candidates are limited to the words that begin with the pressed letters. | |
| ## Training | |
| - Data: `Salesforce/wikitext`, config `wikitext-103-raw-v1`, as written by `scripts/prepare_corpus.py`. | |
| - Run: 40,000 steps and 10.24 million (context, word) examples, about 7.3 hours on one GPU. The context was trimmed to what the daemon actually has at typing time (`./retrain`). | |
| - Held-out accuracy at the last step, measured on 3,000 word positions: | |
| - top 1: 85.7% | |
| - top 3: 94.8% | |
| - top 5: 96.8% | |
| `history.json` has the curve, and `summary.json` has the run's figures. | |
| ## Files | |
| - `checkpoint.pt`: the PyTorch checkpoint, loaded by `BertPredictor.load` in the repository. It is a pickle, so load it only from a source you trust. | |
| - `vocab/vocab.json` and `vocab/letters.npy`: the vocabulary the head scores, and the letters index the chords use. Pass this folder as `--vocab`. | |
| ## Use | |
| ```bash | |
| git clone https://github.com/maximus-powers/stenographer | |
| cd stenographer | |
| # put checkpoint.pt in runs/prefix-40000/ and the vocab folder in data/vocab/ | |
| ./steno | |
| ``` | |