---
base_model: distilbert-base-uncased
datasets:
- Salesforce/wikitext
library_name: pytorch
tags:
- chorded-typing
- word-prediction
- keyboard
---

# stenographer: prefix-40000

The word model for [stenographer](https://github.com/maximus-powers/stenographer). You type a word by pressing its opening letters at once, like a stenographer. This model works out which word you meant from those letters and the sentence so far.

This is the checkpoint the daemon loads by default (`scripts/stenod.py`, `--checkpoint runs/prefix-40000/checkpoint.pt`).

## What it is

- DistilBERT (`distilbert-base-uncased`) encodes the sentence so far, with the word being typed masked.
- A linear head scores the vocabulary.
- At typing time, candidates are limited to the words that begin with the pressed letters.

## Training

- Data: `Salesforce/wikitext`, config `wikitext-103-raw-v1`, as written by `scripts/prepare_corpus.py`.
- Run: 40,000 steps and 10.24 million (context, word) examples, about 7.3 hours on one GPU. The context was trimmed to what the daemon actually has at typing time (`./retrain`).
- Held-out accuracy at the last step, measured on 3,000 word positions:
  - top 1: 85.7%
  - top 3: 94.8%
  - top 5: 96.8%

  `history.json` has the curve, and `summary.json` has the run's figures.

## Files

- `checkpoint.pt`: the PyTorch checkpoint, loaded by `BertPredictor.load` in the repository. It is a pickle, so load it only from a source you trust.
- `vocab/vocab.json` and `vocab/letters.npy`: the vocabulary the head scores, and the letters index the chords use. Pass this folder as `--vocab`.

## Use

```bash
git clone https://github.com/maximus-powers/stenographer
cd stenographer
# put checkpoint.pt in runs/prefix-40000/ and the vocab folder in data/vocab/
./steno
```