Toney Soprano
The name is a pun on "tone": the model judges the tone of words, and it drives the tones of a keyboard.
A 23 MB cross-encoder that answers questions about words by scoring a list of options. Give it a state, a question and some options; it returns a probability for each. It does not generate text, and it spreads its probability out when a question has no good answer.
Built for the Semantic Keyboard, a browser instrument inspired by the Casio PT-1 whose keys play words instead of notes. It also works on its own.
| Size | 23 MB (int8 ONNX) |
| Input | [CLS] state question [SEP] option [SEP], one row per option, max 64 tokens |
| Output | one logit per option; divide by the question type's temperature, then softmax |
| Question types | choice (pick one), score (pick a level on a rubric), noul (yes/no) |
| Base | all-MiniLM-L6-v2 |
| Licence | CC BY-NC 4.0 (model); other files vary, see Files |
Quick start
Python
pip install onnxruntime tokenizers numpy
import json
import numpy as np
import onnxruntime as ort
from tokenizers import Tokenizer
tok = Tokenizer.from_file("tokenizer.json")
tok.enable_truncation(max_length=64)
tok.enable_padding()
sess = ort.InferenceSession("model.int8.onnx")
temps = json.load(open("temps.json"))["temps"]
def score(state, question, options, qtype="choice"):
left = f"{state} {question}".strip()
enc = tok.encode_batch([(left, o) for o in options])
feeds = {
"input_ids": np.array([e.ids for e in enc], dtype=np.int64),
"attention_mask": np.array([e.attention_mask for e in enc], dtype=np.int64),
"token_type_ids": np.array([e.type_ids for e in enc], dtype=np.int64),
}
z = sess.run(None, feeds)[0].reshape(-1) / temps[qtype]
p = np.exp(z - z.max())
return dict(zip(options, (p / p.sum()).round(3).tolist()))
score("Chain: ocean -> wave", "Which word should the next key play?", ["surf", "tax", "shell", "invoice"])
# {'surf': 0.963, 'tax': 0.0, 'shell': 0.035, 'invoice': 0.002}
score("", "What is the capital of France?", ["wave", "desk", "salt", "tax"]) # no good answer
# {'wave': 0.239, 'desk': 0.248, 'salt': 0.248, 'tax': 0.264}
Write the state the way the model was trained: Chain: ocean -> wave for a chain, Word: granite for a single
word. A bare word as the state gives poor answers.
JavaScript (browser)
onnxruntime-web runs the model; Transformers.js is used for the tokenizer only.
npm install onnxruntime-web @huggingface/transformers
import * as ort from "onnxruntime-web";
import { AutoTokenizer } from "@huggingface/transformers";
const REPO = "actuallywolverine/toney-soprano";
const BASE = `https://huggingface.co/${REPO}/resolve/main/`;
const tokenizer = await AutoTokenizer.from_pretrained(REPO);
const session = await ort.InferenceSession.create(BASE + "model.int8.onnx");
const { temps } = await (await fetch(BASE + "temps.json")).json();
async function score(state, question, options, type = "choice") {
const enc = tokenizer(options.map(() => `${state} ${question}`), {
text_pair: options, padding: true, truncation: true, max_length: 64,
});
const int64 = (t) => new ort.Tensor("int64", t.data, t.dims);
const out = await session.run({
input_ids: int64(enc.input_ids), attention_mask: int64(enc.attention_mask), token_type_ids: int64(enc.token_type_ids),
});
const logits = Array.from(out.logits.data, (x) => x / temps[type]);
const max = Math.max(...logits);
const exp = logits.map((x) => Math.exp(x - max)), sum = exp.reduce((a, b) => a + b);
return Object.fromEntries(options.map((o, i) => [o, +(exp[i] / sum).toFixed(3)]));
}
await score("Chain: ocean -> wave", "Which word should the next key play?", ["surf", "tax", "shell", "invoice"]);
// { surf: 0.963, tax: 0, shell: 0.035, invoice: 0.002 }
Probabilities can move by up to about 0.05 between runs while the ranking holds: the int8 model scales its numbers by the batch it is given, and the browser runtime rounds slightly differently from native onnxruntime.
Ratings only
word_ratings.json works alone as a lookup table: 6,132 words x 15 dimensions, integers 1-9.
ratings = json.load(open("word_ratings.json"))
dict(zip(ratings["dims"], ratings["words"]["ocean"])) # {'valence': 8, 'arousal': 5, ..., 'brightness': 6, ...}
Intended use
Word games (next word, odd one out), ranking alternatives by a quality such as brightness, mapping sliders, keys or colours to word qualities, and research on fast scoring that admits uncertainty.
Out of scope: decisions about people, factual question answering, text generation.
Background: System One
A "System One" model in Kahneman's sense: one fast pass that returns a gut ranking and how sure it is, with no reasoning steps and no generated text. The framing is loosely inspired by, and unrelated to, TypeSafe AI's System One Models.
Evaluation
Mean of 2 training runs, fp32, calibrated.
Held-out splits (chance is 25% for choice):
| Split | Choice | Yes/no |
|---|---|---|
| Unseen words | 86% | 93% |
| Unseen wordings | 76% | 89% |
| Unseen family ("which word is most exciting?") | 60% (ECE 0.14) | not tested |
Keyboard eval (177 hand-written questions the app asks, answers judged by hand). The baseline is the untuned
base model: cosine similarity between the question text and each option, highest wins
(tools/baseline_cosine.py).
| Category | Questions | This model | Untuned MiniLM |
|---|---|---|---|
| Next word | 20 | 100% | 90% |
| Next word, ambiguous senses | 5 | 5/5 | 4/5 |
| Chain coherence (yes/no) | 48 | 82% | 50% |
| Timbre (yes/no, wording never trained on) | 22 | 73% | 36% |
| New rating scales, within one level | 64 | 92% | 59% |
| Key mapping | 8 | 62% | 50% |
| No-answer, mean confidence (ideal 0.35) | 10 | 0.52 | no probabilities |
Easy next-word questions are nearly solved by the base model; the fine-tuning shows in the yes/no and rating questions, where the baseline is at or below chance.
In an app-realistic next-word test (29 related words as options), the top pick is linked in the LWOW association data 65% of the time (random 11%). The int8 model picks the same answer as fp32 on 93% of 10,100 held-out questions.
Limitations
- Abstention is partial: 0.52 mean confidence on no-answer questions against an ideal of 0.35. A high probability does not prove a good answer exists.
- New kinds of question are weaker: 60% on an unseen family. Plain wordings work much better than figurative ones.
- Key mapping is unreliable: 62% on 8 questions, barely above the untuned baseline.
- Quantization changes about 7% of answers relative to fp32.
- English only, single words and short chains, trained on an 8,000-word vocabulary.
- Synthetic data: associations and some ratings are LLM-generated, so the model inherits their biases.
Training
Fine-tuned from all-MiniLM-L6-v2 for 2 epochs (batch 32, learning rate 3e-5, AdamW, one-cycle) on an Apple M1. Data: about 138k generated questions over 8,000 words, built from LWOW associations (LLM-generated), human norms (Warriner et al. 2013, Brysbaert et al. 2014) and ratings generated with Claude Opus 5.5, with about 4% no-answer questions. One temperature per question type is fitted on held-out words.
Files
| File | Contents | Source | Licence |
|---|---|---|---|
model.int8.onnx |
the model | all-MiniLM-L6-v2, fine-tuned on LWOW, the human norms (not redistributed) and Claude ratings | CC BY-NC 4.0 |
temps.json |
per-type temperatures | this work | CC BY-NC 4.0 |
word_ratings.json |
6,132 words x 15 dimensions | generated with Claude (Anthropic) | CC BY-NC 4.0 |
word_vectors.* |
vectors for the same words | ConceptNet Numberbatch 17.06 (subset) | CC BY-SA 4.0 |
word_directions.* |
6,132 words x 5 directions (brightness, warmth, energy, weight, mood) | ratings + Numberbatch | CC BY-SA 4.0 |
tokenizer*.json |
WordPiece tokenizer | all-MiniLM-L6-v2 | Apache 2.0 |
The model is non-commercial because training used the Warriner norms (CC BY-NC-ND 3.0). Also used to build the
data: LWOW (CC BY 4.0), GloVe (PDDL), wordfreq (MIT / CC BY-SA 4.0), the LDNOOBW word list (CC BY 4.0). Full terms
are in LICENSE.
Links and citation
Demo: Semantic Keyboard. Playground:
Ask about words. Code: pt1-system-one and
semantic-keyboard on GitHub (to be made public).
@misc{irizarry2026toneysoprano,
author = {Irizarry, Justin},
title = {Toney Soprano: a small calibrated cross-encoder for word questions},
year = {2026},
note = {https://huggingface.co/actuallywolverine/toney-soprano}
}
Model tree for actuallywolverine/toney-soprano
Base model
nreimers/MiniLM-L6-H384-uncased