Toney Soprano

The name is a pun on "tone": the model judges the tone of words, and it drives the tones of a keyboard.

A 23 MB cross-encoder that answers questions about words by scoring a list of options. Give it a state, a question and some options; it returns a probability for each. It does not generate text, and it spreads its probability out when a question has no good answer.

Built for the Semantic Keyboard, a browser instrument inspired by the Casio PT-1 whose keys play words instead of notes. It also works on its own.

The Semantic Keyboard playing a ten-note tune: each key press shows a new word, chosen with this model, and a line chart traces the words' tone beside the melody

Size 23 MB (int8 ONNX)
Input [CLS] state question [SEP] option [SEP], one row per option, max 64 tokens
Output one logit per option; divide by the question type's temperature, then softmax
Question types choice (pick one), score (pick a level on a rubric), noul (yes/no)
Base all-MiniLM-L6-v2
Licence CC BY-NC 4.0 (model); other files vary, see Files

Quick start

Python

pip install onnxruntime tokenizers numpy
import json
import numpy as np
import onnxruntime as ort
from tokenizers import Tokenizer

tok = Tokenizer.from_file("tokenizer.json")
tok.enable_truncation(max_length=64)
tok.enable_padding()
sess = ort.InferenceSession("model.int8.onnx")
temps = json.load(open("temps.json"))["temps"]

def score(state, question, options, qtype="choice"):
    left = f"{state} {question}".strip()
    enc = tok.encode_batch([(left, o) for o in options])
    feeds = {
        "input_ids": np.array([e.ids for e in enc], dtype=np.int64),
        "attention_mask": np.array([e.attention_mask for e in enc], dtype=np.int64),
        "token_type_ids": np.array([e.type_ids for e in enc], dtype=np.int64),
    }
    z = sess.run(None, feeds)[0].reshape(-1) / temps[qtype]
    p = np.exp(z - z.max())
    return dict(zip(options, (p / p.sum()).round(3).tolist()))

score("Chain: ocean -> wave", "Which word should the next key play?", ["surf", "tax", "shell", "invoice"])
# {'surf': 0.963, 'tax': 0.0, 'shell': 0.035, 'invoice': 0.002}

score("", "What is the capital of France?", ["wave", "desk", "salt", "tax"])   # no good answer
# {'wave': 0.239, 'desk': 0.248, 'salt': 0.248, 'tax': 0.264}

Write the state the way the model was trained: Chain: ocean -> wave for a chain, Word: granite for a single word. A bare word as the state gives poor answers.

JavaScript (browser)

onnxruntime-web runs the model; Transformers.js is used for the tokenizer only.

npm install onnxruntime-web @huggingface/transformers
import * as ort from "onnxruntime-web";
import { AutoTokenizer } from "@huggingface/transformers";

const REPO = "actuallywolverine/toney-soprano";
const BASE = `https://huggingface.co/${REPO}/resolve/main/`;
const tokenizer = await AutoTokenizer.from_pretrained(REPO);
const session = await ort.InferenceSession.create(BASE + "model.int8.onnx");
const { temps } = await (await fetch(BASE + "temps.json")).json();

async function score(state, question, options, type = "choice") {
  const enc = tokenizer(options.map(() => `${state} ${question}`), {
    text_pair: options, padding: true, truncation: true, max_length: 64,
  });
  const int64 = (t) => new ort.Tensor("int64", t.data, t.dims);
  const out = await session.run({
    input_ids: int64(enc.input_ids), attention_mask: int64(enc.attention_mask), token_type_ids: int64(enc.token_type_ids),
  });
  const logits = Array.from(out.logits.data, (x) => x / temps[type]);
  const max = Math.max(...logits);
  const exp = logits.map((x) => Math.exp(x - max)), sum = exp.reduce((a, b) => a + b);
  return Object.fromEntries(options.map((o, i) => [o, +(exp[i] / sum).toFixed(3)]));
}

await score("Chain: ocean -> wave", "Which word should the next key play?", ["surf", "tax", "shell", "invoice"]);
// { surf: 0.963, tax: 0, shell: 0.035, invoice: 0.002 }

Probabilities can move by up to about 0.05 between runs while the ranking holds: the int8 model scales its numbers by the batch it is given, and the browser runtime rounds slightly differently from native onnxruntime.

Ratings only

word_ratings.json works alone as a lookup table: 6,132 words x 15 dimensions, integers 1-9.

ratings = json.load(open("word_ratings.json"))
dict(zip(ratings["dims"], ratings["words"]["ocean"]))   # {'valence': 8, 'arousal': 5, ..., 'brightness': 6, ...}

Intended use

Word games (next word, odd one out), ranking alternatives by a quality such as brightness, mapping sliders, keys or colours to word qualities, and research on fast scoring that admits uncertainty.

Out of scope: decisions about people, factual question answering, text generation.

Background: System One

A "System One" model in Kahneman's sense: one fast pass that returns a gut ranking and how sure it is, with no reasoning steps and no generated text. The framing is loosely inspired by, and unrelated to, TypeSafe AI's System One Models.

Evaluation

Mean of 2 training runs, fp32, calibrated.

Held-out splits (chance is 25% for choice):

Split Choice Yes/no
Unseen words 86% 93%
Unseen wordings 76% 89%
Unseen family ("which word is most exciting?") 60% (ECE 0.14) not tested

Keyboard eval (177 hand-written questions the app asks, answers judged by hand). The baseline is the untuned base model: cosine similarity between the question text and each option, highest wins (tools/baseline_cosine.py).

Category Questions This model Untuned MiniLM
Next word 20 100% 90%
Next word, ambiguous senses 5 5/5 4/5
Chain coherence (yes/no) 48 82% 50%
Timbre (yes/no, wording never trained on) 22 73% 36%
New rating scales, within one level 64 92% 59%
Key mapping 8 62% 50%
No-answer, mean confidence (ideal 0.35) 10 0.52 no probabilities

Easy next-word questions are nearly solved by the base model; the fine-tuning shows in the yes/no and rating questions, where the baseline is at or below chance.

In an app-realistic next-word test (29 related words as options), the top pick is linked in the LWOW association data 65% of the time (random 11%). The int8 model picks the same answer as fp32 on 93% of 10,100 held-out questions.

Limitations

  • Abstention is partial: 0.52 mean confidence on no-answer questions against an ideal of 0.35. A high probability does not prove a good answer exists.
  • New kinds of question are weaker: 60% on an unseen family. Plain wordings work much better than figurative ones.
  • Key mapping is unreliable: 62% on 8 questions, barely above the untuned baseline.
  • Quantization changes about 7% of answers relative to fp32.
  • English only, single words and short chains, trained on an 8,000-word vocabulary.
  • Synthetic data: associations and some ratings are LLM-generated, so the model inherits their biases.

Training

Fine-tuned from all-MiniLM-L6-v2 for 2 epochs (batch 32, learning rate 3e-5, AdamW, one-cycle) on an Apple M1. Data: about 138k generated questions over 8,000 words, built from LWOW associations (LLM-generated), human norms (Warriner et al. 2013, Brysbaert et al. 2014) and ratings generated with Claude Opus 5.5, with about 4% no-answer questions. One temperature per question type is fitted on held-out words.

Files

File Contents Source Licence
model.int8.onnx the model all-MiniLM-L6-v2, fine-tuned on LWOW, the human norms (not redistributed) and Claude ratings CC BY-NC 4.0
temps.json per-type temperatures this work CC BY-NC 4.0
word_ratings.json 6,132 words x 15 dimensions generated with Claude (Anthropic) CC BY-NC 4.0
word_vectors.* vectors for the same words ConceptNet Numberbatch 17.06 (subset) CC BY-SA 4.0
word_directions.* 6,132 words x 5 directions (brightness, warmth, energy, weight, mood) ratings + Numberbatch CC BY-SA 4.0
tokenizer*.json WordPiece tokenizer all-MiniLM-L6-v2 Apache 2.0

The model is non-commercial because training used the Warriner norms (CC BY-NC-ND 3.0). Also used to build the data: LWOW (CC BY 4.0), GloVe (PDDL), wordfreq (MIT / CC BY-SA 4.0), the LDNOOBW word list (CC BY 4.0). Full terms are in LICENSE.

Links and citation

Demo: Semantic Keyboard. Playground: Ask about words. Code: pt1-system-one and semantic-keyboard on GitHub (to be made public).

@misc{irizarry2026toneysoprano,
  author = {Irizarry, Justin},
  title  = {Toney Soprano: a small calibrated cross-encoder for word questions},
  year   = {2026},
  note   = {https://huggingface.co/actuallywolverine/toney-soprano}
}
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for actuallywolverine/toney-soprano

Space using actuallywolverine/toney-soprano 1