actuallywolverine/tony-soprano

Tony Soprano (working name)

A small, calibrated scorer for questions about words: which word goes with another, which continues a chain, which is the odd one out, how bright, warm or calm a word feels. Given a state, a question and a list of options, it scores every option in one parallel pass and returns calibrated probabilities: when it is unsure, it says so. No text generation. It is small enough (23 MB) to run in a web page.

It was built for the Semantic Keyboard, a browser instrument inspired by the Casio PT-1 whose keys play words instead of notes. The music lives in the app (how far you move between keys sets how related the next word is, and higher keys pick brighter words); the model only answers the word questions, so it can be used without the keyboard.

Try it: play the Semantic Keyboard at semantic-keyboard.pt1-lab.workers.dev. It runs this model in your browser.

state:    Chain: ocean -> wave
question: Which word should the next key play?
options:  surf, foam, tide, shell, harbor       ->  tide 0.55, surf 0.39, harbor 0.03, shell 0.02, foam 0.01

• 23 MB (int8 ONNX), runs entirely in the browser with onnxruntime-web (WebAssembly); nothing leaves the player's device.
• Speed: ~55 ms to score 29 options on one CPU thread with native onnxruntime (Apple M1). In the browser (WebAssembly, one thread, headless Chrome on the same M1) the app's full think-ahead step -- finding 32 candidates and scoring them -- takes ~0.4-0.5 s; the app does it right after each note, so key presses feel instant.
• Question types: choice (pick one), score (pick a level on a rubric), noul (calibrated yes/no).
• Honest when unsure: trained on questions with no right answer, so off-topic questions get spread-out probability.

Shipped file: v5, quantized to int8. Evaluation tables below are the mean of 2 training runs (fp32, before quantization) unless marked otherwise.

Usage

Python

pip install onnxruntime tokenizers numpy

import json, numpy as np, onnxruntime as ort
from tokenizers import Tokenizer

tok = Tokenizer.from_file("tokenizer.json")
tok.enable_padding(); tok.enable_truncation(64)
sess = ort.InferenceSession("model.int8.onnx")
temps = json.load(open("temps.json"))["temps"]

def ask(state, question, options, qtype="choice"):
    enc = tok.encode_batch([(f"{state} {question}", o) for o in options])
    feeds = {name: np.array([getattr(e, attr) for e in enc], dtype=np.int64)
             for name, attr in [("input_ids", "ids"), ("attention_mask", "attention_mask"), ("token_type_ids", "type_ids")]}
    logits = sess.run(None, feeds)[0] / temps[qtype]
    p = np.exp(logits - logits.max()); p /= p.sum()
    return dict(zip(options, p.astype(float).round(3).tolist()))

ask("Chain: ocean -> wave", "Which word should the next key play?", ["surf", "tax", "shell", "invoice"])
# {'surf': 0.963, 'tax': 0.0, 'shell': 0.035, 'invoice': 0.002}

# the word ratings need no model: one row of 15 integers (1-9) per word, in the order of "dims"
ratings = json.load(open("word_ratings.json"))
dict(zip(ratings["dims"], ratings["words"]["ocean"]))   # {'valence': 8, 'arousal': 5, ..., 'brightness': 6, ...}

JavaScript (browser)

The same steps with onnxruntime-web for the model and Transformers.js for the tokenizer only.

npm install onnxruntime-web @huggingface/transformers

import * as ort from "onnxruntime-web";
import { AutoTokenizer } from "@huggingface/transformers";

const REPO = "actuallywolverine/tony-soprano";
const BASE = `https://huggingface.co/${REPO}/resolve/main/`;
const tokenizer = await AutoTokenizer.from_pretrained(REPO);
const session = await ort.InferenceSession.create(BASE + "model.int8.onnx");
const { temps } = await (await fetch(BASE + "temps.json")).json();

async function ask(state, question, options, type = "choice") {
  const enc = tokenizer(options.map(() => `${state} ${question}`), {
    text_pair: options, padding: true, truncation: true, max_length: 64,
  });
  const int64 = (t) => new ort.Tensor("int64", t.data, t.dims);
  const out = await session.run({
    input_ids: int64(enc.input_ids), attention_mask: int64(enc.attention_mask), token_type_ids: int64(enc.token_type_ids),
  });
  const logits = Array.from(out.logits.data, (x) => x / temps[type]);
  const max = Math.max(...logits);
  const exp = logits.map((x) => Math.exp(x - max)), sum = exp.reduce((a, b) => a + b);
  return Object.fromEntries(options.map((o, i) => [o, +(exp[i] / sum).toFixed(3)]));
}

await ask("Chain: ocean -> wave", "Which word should the next key play?", ["surf", "tax", "shell", "invoice"]);
// { surf: 0.963, tax: 0, shell: 0.035, invoice: 0.002 }

The Semantic Keyboard runs these steps in a Web Worker with its own small WordPiece tokenizer (see the app's src/engine/).

Small differences between runs are expected. The int8 model scales its numbers by the whole batch it is given, so the same option can score a little differently depending on which other options are scored with it, and the browser runtime rounds slightly differently from native onnxruntime. In testing, probabilities moved by up to ~0.05 while the ranking held.

Intended use

Scoring a small set of candidate words (or short options) against a question about words: which word continues a chain, whether a chain is still coherent, how a word rates on a quality such as brightness or tone. For example:

• Word games: the next word in a chain, the odd one out, "which of these is warmest?"
• Writing helpers: a darker or brighter alternative, synonyms ranked by a quality.
• Instruments and visuals: map a slider, key or colour to word qualities, as the Semantic Keyboard does.
• Research: a small example of a calibrated "System One" scorer (fast, intuitive judgement, in Kahneman's sense) that runs in the browser and is honest when a question has no right answer.

The word ratings and the 5 word directions shipped alongside it (see Files) can also be used on their own, without the model.

Out of scope: decisions about people (hiring, moderation, profiling or anything similar), factual question answering, and text generation. It knows word associations and word qualities, not facts about the world.

How it works

Every option is encoded together with the state and question by a MiniLM-L6 cross-encoder ([CLS] state question [SEP] option [SEP]), mean-pooled, and a small head gives one logit per option. A softmax over a question's options, after dividing by a per-type temperature (temps.json), gives the answer distribution.

In the keyboard app the model answers the contextual questions (which word continues the chain, is the chain still coherent). Fixed facts about single words (pitch, brightness, loudness, mood...) come from the ratings table shipped alongside it (word_ratings.json), which is both more accurate and instant.

The keyboard's "What?" panel after one key press: the app gathers candidate words, this model scores them (the percentages), and the app weighs those scores against the melody before picking a word.

Files

file	what	licence
model.int8.onnx	the model; inputs input_ids, attention_mask, token_type_ids (one row per option), output one logit per row	CC BY-NC 4.0
tokenizer.json, tokenizer_config.json	the base model's WordPiece tokenizer; encode (state + " " + question, option) as a text pair, max 64 tokens	Apache 2.0
temps.json	per-type temperatures: divide logits by temps[type] before the softmax	CC BY-NC 4.0
word_ratings.json	the 6,132 playable words x 15 dimensions (valence, arousal, dominance, concreteness, brightness, loudness, warmth, speed, weight, hardness, age, dreaminess, pitch, evocative, suitable), integers 1-9, rated by Claude (see Training)	CC BY-NC 4.0
word_vectors.bin, word_vectors.json	ConceptNet Numberbatch 17.06 vectors for the same words (subset, unit length, float16): the app's candidates for the next key are the last word's nearest neighbours	CC BY-SA 4.0
word_directions.bin, word_directions.json	each playable word's position along 5 named directions (brightness, warmth, energy = arousal, weight, mood = valence), float32, 6,132 x 5, same word order as word_vectors.json. Derived from the Claude ratings and the Numberbatch vectors: ridge regression of the ratings on the vectors (fit r 0.80-0.90), see tools/export_directions.py. The app describes a step between two words as its move along these directions, so a repeated melody can replay the same moves	CC BY-SA 4.0

See LICENSE for details.

The word directions at work: the keyboard places each word along brightness and energy (from word_directions.bin). Play a melody again from higher keys and the new words repeat the same moves, drawn as matching lines.

Training

Base: sentence-transformers/all-MiniLM-L6-v2 (22.7M parameters), fine-tuned 2 epochs, batch 32, lr 3e-5, AdamW with a one-cycle schedule, on an Apple M1 GPU (~50 min). Score questions use ordinal label smoothing; questions with several good answers use graded targets; questions with no right answer use a uniform target. One temperature per question type is fitted on held-out words.

Data: ~138k generated questions with checkable answers, across 40+ families, over an 8,000-word training vocabulary (the most frequent English words with a GloVe vector and a Brysbaert entry, profanity removed):

• Associations from LWOW (LLM World of Words): free associations generated by LLMs -- we used its Claude Haiku and Mistral sets. Families: strongest associate, distant (two-hop) associate, chain continuation, odd one out, are-these-linked, and next word in the app's own format (16-29 related candidates, target weighted by association strength).
• Word qualities as rubric scores, comparisons (most / least) and yes/no: valence, arousal and dominance from human norms (Warriner et al. 2013), concreteness from human norms (Brysbaert et al. 2014), and brightness, loudness, warmth, speed, weight and dreaminess from ratings generated with Claude (Opus 5.5).
• No-answer questions (~4%): off-topic questions, unknowable yes/no questions, and questions missing the context they need. Target: spread evenly.
• ~35 hand-written wordings per original family and ~8 patterned ones per newer family; 4 (original) or 1 (newer) held out per family for testing.

The Claude ratings. Claude Opus 5.5 (Batch API) rated the 8,000-word training vocabulary on 12 qualities, after a 500-word pilot that checked it against human norms (r = 0.90 for valence, 0.91 for concreteness, 0.69 for arousal, 0.51 for dominance; rating the same words twice agreed at 0.95-0.98). To build the playable vocabulary it then rated ~18,500 candidate words (the training vocabulary plus 12,718 more of the 60,000 most frequent) for how evocative and how suitable for a general audience each is, plus pitch, and gave the new keepers full ratings: about 21,000 distinct words in all, for about $20. word_ratings.json keeps the 6,132 playable words (1,883 from the training vocabulary, 4,249 added) with all 15 dimensions.

Evaluation

Held-out splits (calibrated, mean of 2 seeds):

split	what's new	choice	yes/no
unseen words	words never trained on	86%	93%
unseen wordings	4 held-out phrasings per family	76%	89%
unseen family	"which word is most exciting?" -- never trained on	60% (chance 25%), calibration error (ECE) 0.14

Keyboard eval (eval/keyboard_eval.jsonl: 177 hand-written questions the app will actually ask, answers judged by hand; mean of 2 seeds):

category	questions	score
next word	20	100%
next word, ambiguous senses ("key -> lock -> ?" with piano distractors)	5	5/5
chain coherence	48	82%
timbre ("would this word suit a low, dark note?") -- wording never trained on	22	73%
new rating scales, within one level	64	92%
key mapping ("which word suits the highest key?")	8	62%
questions with no right answer: mean confidence (lower is more honest)	10	0.52

For the no-answer questions, 0.35 would be perfectly honest: it's the confidence an even spread gets on those questions (6 with four options, 0.25 each; 4 yes/no, 0.5 each). Before the no-answer training it was 0.70.

App-realistic next word (tools/eval_app.py, mean of 2 seeds): given a chain ending in a held-out word and that word's 29 nearest playable neighbours as options, the model's top pick is a word the LLM free-association data (LWOW) links to it 65% of the time (random pick: 11%; an earlier version: 54%), and one is in its top 3 83% of the time. Training on this exact format cost some chain coherence (89% -> 82%) and timbre wording (82% -> 73%); in the app, pitch and brightness come from the ratings table instead, and coherence can be cross-checked geometrically.

int8: per-channel dynamic quantization keeps ~93% of answers identical to fp32 over 10k questions (measured on an earlier version) and stays within ~1-3 points of fp32 on every split.

Limitations

• English only; tuned to single words and short chains, not sentences.
• Associations come from LLM-generated free associations, and word qualities from human norms and Claude ratings; all carry their sources' biases (arousal is noisy even among humans; Claude's dominance ratings agree weakly with human ones, r = 0.51).
• Unfamiliar or figurative question wordings ("most worked up", "puts you in the driver's seat") are much weaker than plain ones; ask questions in plain words.
• Probabilities are calibrated on familiar question types; on new kinds of questions the model is more confident than it should be, though far less than before the no-answer training.

Links

• Demo: the Semantic Keyboard
• Training code, evaluations and tools: https://github.com/justinleeirizarry/pt1-system-one (to be made public)
• App: https://github.com/justinleeirizarry/semantic-keyboard (to be made public)

Citation

@misc{irizarry2026modelname,
  title  = {[Model name TBD]: a small System One scorer for a word keyboard},
  author = {Irizarry, Justin},
  year   = {2026},
  note   = {https://huggingface.co/actuallywolverine/tony-soprano}
}

Licence and credits

Licensed per file (see the Files table and LICENSE): the model, temperatures and ratings under CC BY-NC 4.0 (non-commercial, because training used the Warriner norms); the word vectors and the word directions derived from them under CC BY-SA 4.0, as required by ConceptNet Numberbatch; the tokenizer under Apache 2.0 from the base model.

word_ratings.json contains only ratings generated with Claude. The human norms used in training (Warriner et al. 2013, Brysbaert et al. 2014) are not included in this repository.

Built with:

• Warriner, Kuperman & Brysbaert (2013), valence/arousal/dominance norms -- CC BY-NC-ND 3.0 (training only; not redistributed)
• Brysbaert, Warriner & Kuperman (2014), concreteness norms -- released freely for research; no formal licence stated (training only; not redistributed)
• LWOW (LLM World of Words) free associations -- CC BY 4.0
• ConceptNet Numberbatch 17.06 (Speer, Chin & Havasi) -- CC BY-SA 4.0 (shipped vectors, subset and converted)
• GloVe (Pennington, Socher & Manning 2014) -- PDDL (used to build the training data; not shipped)
• wordfreq (Speer) -- MIT / CC BY-SA 4.0 (vocabulary)
• LDNOOBW word list -- CC BY 4.0 (profanity filter)
• all-MiniLM-L6-v2 (sentence-transformers) -- Apache 2.0 (base model and tokenizer)
• Word ratings generated with Claude (Anthropic)