MakeItMore commited on
Commit
de6c56d
·
verified ·
1 Parent(s): da8de96

SpellSpeak Tone rc3

Browse files
Files changed (7) hide show
  1. MANIFEST.json +12 -12
  2. README.md +70 -77
  3. config.json +16 -16
  4. example.py +2 -2
  5. model.onnx +1 -1
  6. model.opt.onnx +1 -1
  7. model_fp16.onnx +1 -1
MANIFEST.json CHANGED
@@ -1,33 +1,33 @@
1
  {
2
  "name": "SpellSpeak Tone",
3
- "release": "rc2.1",
4
- "package": "spellspeak-tone-rc2.2",
5
- "created": "2026-10-09T22:16:39Z",
6
  "contract": "line-tags-0.2",
7
  "files": {
8
  "README.md": {
9
- "bytes": 14297,
10
- "sha256": "3013c4ff392150819362ac5c7735f894aff751aa3aa0b6f334948d3ecab40a3b"
11
  },
12
  "config.json": {
13
- "bytes": 1515,
14
- "sha256": "531d5558cd620780ac3fe02d2acf2aad960e768619440d48d8c0abf4b32fa182"
15
  },
16
  "example.py": {
17
- "bytes": 2930,
18
- "sha256": "91087ec1faa9557ccae9cf31d4550a4a983cb3bb334cafc884c411aa93f23f77"
19
  },
20
  "model.onnx": {
21
  "bytes": 111170339,
22
- "sha256": "ee4d0f23840e74d4b529de7ce3f5f057a2202fc95cb1166a0392c833919a3e80"
23
  },
24
  "model.opt.onnx": {
25
  "bytes": 111132591,
26
- "sha256": "d6809b08375415f0cd2bbc0282d9d3c8d8282b010001ee52559f7fae5b857e60"
27
  },
28
  "model_fp16.onnx": {
29
  "bytes": 55694715,
30
- "sha256": "08201561726a6e80cf454ab57e786b268ee428d152dc9299f80941e541ad11d7"
31
  },
32
  "requirements.txt": {
33
  "bytes": 287,
 
1
  {
2
  "name": "SpellSpeak Tone",
3
+ "release": "rc3",
4
+ "package": "spellspeak-tone-rc3.1",
5
+ "created": "2026-10-13T18:24:17Z",
6
  "contract": "line-tags-0.2",
7
  "files": {
8
  "README.md": {
9
+ "bytes": 13322,
10
+ "sha256": "02f6f2312c332c2e43995d0da70b04452b2c03e47ee516ff45066b20957ba793"
11
  },
12
  "config.json": {
13
+ "bytes": 1512,
14
+ "sha256": "a951d88defbccbe43fa103e8a24e9b233ea6cfe2cda3fd8fec574f67510f7ca8"
15
  },
16
  "example.py": {
17
+ "bytes": 2926,
18
+ "sha256": "efd2134c6eb28ac84bab1c9ad932bb61d8de8bc123e4a372ffd98629abdaad97"
19
  },
20
  "model.onnx": {
21
  "bytes": 111170339,
22
+ "sha256": "dc5aa341edd3e21f83f782900b7a9737f88e671fb2a9ce44b190236f65c4a199"
23
  },
24
  "model.opt.onnx": {
25
  "bytes": 111132591,
26
+ "sha256": "f53f61ddc0823ff6ffbecd5cdeac49888e08b5156adcebf7ce5ae6e69f3ed425"
27
  },
28
  "model_fp16.onnx": {
29
  "bytes": 55694715,
30
+ "sha256": "59754462606ddc50086c4cbba14b41bf336d504047f4160792f747cab257355b"
31
  },
32
  "requirements.txt": {
33
  "bytes": 287,
README.md CHANGED
@@ -20,29 +20,28 @@ tags:
20
  - dialogue
21
  ---
22
 
23
- # SpellSpeak Tone · RC2.1
24
 
25
  When several AI characters share one conversation, as in a game, an interactive story or any other multi-character voice experience, each of them needs a structured reading of every line: how the speaker sounds, what the line does to each person present, and whether it leaves a question open for someone else to answer. The language model voicing a character could produce that reading, but only through extra output tokens, training or tool calls. That adds latency to every turn, takes capacity from the dialogue, risks malformed output, and repeats the work for each character in the scene.
26
 
27
- SpellSpeak Tone treats the reading as multi-task classification by a separate small model. One pass over a line predicts the speaker's emotion, the social act directed at each person present with a hostility probability per person, and, new in rc2, the move the line makes in the conversation: does it ask, answer, pass on a rumour, say it does not know, hold back, or close the exchange. The application uses the tags for faces, feelings and who speaks next. Its constraints are millisecond inference on a CPU, robustness to unstructured user input, and rare false hostility.
28
 
29
  [Examples and source](https://github.com/spellspeak/models/tree/main/tone)
30
 
31
- ## What is new in RC2.1
32
 
33
- - **A 16-bit file at half the size, with the same answers.** `model_fp16.onnx` is rc2's model with its weights stored in 16 bits: 55.7 MB against 111.1 MB. On 5,204 test lines, read in all four columns with and without the optional inputs, at most one line in a column gets a different tag from full precision. No scorecard measure changes beyond noise.
34
- - **The same speed, and no memory saved.** It computes in 32 bits. p95 per line for 8 people on 4 threads of an Apple M1 Max: 10.0 ms against 9.9 ms for full precision, timed side by side (1 thread: 29.4 against 29.5 ms). Resident memory: 222 MB against 201 MB. What it saves is the download and the disk.
35
- - **Full precision is unchanged and stays the default.** `model.opt.onnx`, `model.onnx`, `tokenizer.json` and `config.json` are rc2's, byte for byte. Ask for the 16-bit file with `precision="fp16"`, or `python example.py --fp16`.
36
-
37
- Full precision is the reference: every other number on this card is full precision's, and it is the file to keep for any further training.
38
 
39
  | | |
40
  |---|---|
41
- | **Released** | 2026-10-09 · rc2.1 (package `spellspeak-tone-rc2.2`) |
42
  | **What it does** | Reads one line of dialogue and tags the speaker's emotion, the social act toward each person present, and the line's move in the conversation |
43
- | **What changed from rc1** | One new output per line, the move, from two extra encoder layers and a small head beside rc1's unchanged encoder, in the same pass. Every rc1 output (emotion, acts, intensities, backchannel, hostility, confidences) is rc1's, checked on every one of 3,754 test lines |
44
- | **Size** | 27.9M parameters (rc1's 23.6M plus 4.3M for the move) · 111.1 MB ONNX in full precision (fp32), 55.7 MB with the weights in 16 bits (`model_fp16.onnx`) |
45
- | **Speed** | p95 per line on 4 CPU threads (Apple M1 Max): 7.0 ms for 1 person, 7.9 for 4, 9.7 for 8 (rc1: 5.5, 6.6, 7.3). On 1 thread: 19 to 28 ms. The 16-bit file takes the same time |
46
  | **Runtime** | ONNX Runtime (1.24 or later) and tokenizers, on CPU. A Python reference runtime ships with it |
47
  | **Files** | `model.opt.onnx` (the graph to run), `model.onnx` (the same graph unoptimized), `model_fp16.onnx` (the graph to run with its weights in 16 bits), `tokenizer.json` and `config.json`, with the reference runtime (`runtime/`), `example.py` and `requirements.txt`. `MANIFEST.json` lists every file with its size and sha256 |
48
  | **Licence** | Apache-2.0 |
@@ -51,7 +50,7 @@ Full precision is the reference: every other number on this card is full precisi
51
 
52
  ```bash
53
  pip install -U huggingface_hub
54
- hf download spellspeak/tone --revision rc2.1 --local-dir spellspeak-tone
55
  cd spellspeak-tone
56
  pip install -r requirements.txt
57
  python example.py
@@ -61,34 +60,30 @@ The example puts three characters and the player in a room. The player asks ever
61
 
62
  ```
63
  player to everyone: 'Who knows where the mill key is?'
64
- move asks (0.96) · emotion neutral · acts: none
65
  Mara to player: "It's under the counter, love."
66
- move answers (0.91) · emotion neutral · acts: none
67
  Tomas to player: 'I heard the miller had it last.'
68
  move hedges (0.98) · emotion neutral · acts: none
69
  Wren to player: 'Why do you want to know?'
70
- move withholds (0.52) · emotion neutral · acts: none
71
  Tomas to player: 'No idea, sorry.'
72
  move doesnt_know (0.99) · emotion neutral · acts: player apology
73
  player to Mara: "thanks, that's all i needed"
74
- move closes (0.99) · emotion happy · acts: Mara thanks
75
  player to Tomas: "Touch my horse again and you'll regret it."
76
- move asks (0.59) · emotion angry · acts: Tomas threat
77
  ```
78
 
79
- A group conversation can play Mara's answer first, add Tomas's rumour as a second voice, and drop the dodge and the "no idea".
80
-
81
- The small download leaves out the two full-precision graphs, about 56 MB in all, and runs the 16-bit file:
82
 
83
  ```bash
84
- hf download spellspeak/tone --revision rc2.1 --local-dir spellspeak-tone-small --exclude model.onnx --exclude model.opt.onnx
85
  cd spellspeak-tone-small
86
  pip install -r requirements.txt
87
  python example.py --fp16
88
  ```
89
 
90
- It prints exactly the same output.
91
-
92
  In your own code, with the downloaded folder as the working directory:
93
 
94
  ```python
@@ -100,49 +95,49 @@ from harness.expression.classifier import LineClassifier
100
 
101
  tone = LineClassifier(".") # full precision (model.opt.onnx), the default
102
  # tone = LineClassifier(".", precision="fp16") # the 16-bit file (model_fp16.onnx), for the small download
103
- tags = tone.tag(LineInput(speaker="Wren", speaker_kind="npc", to="Tomas", targets=["Tomas", "Mara", "player"],
104
- text="Aye, I saw the whole thing.", previous="Tomas: Wren, you were there, weren't you?"))
105
- print(tags.exchange) # label='answers' confidence=0.9413
106
- print(tags.emotion, tags.acts["Tomas"]) # neutral, act='none' ...
107
  ```
108
 
109
  ## Input and output
110
 
111
- - **Input:** the line, the line before it, who speaks (the player or a character), who it is said to (or the room), and everyone present (tested up to 8 people; 160 tokens at most). Who it is said to and the line before are optional, and help: see the table below.
112
- - **Per line:** an emotion (neutral, happy, amused, angry, sad, afraid, surprised, disgusted, contemptuous) with intensity (low, medium, high) and a confidence. Whether the line is only a listening sound such as "mm-hm" (backchannel). New in rc2: the **move**, with a confidence.
113
  - **Per person:** an act (insult, threat, accusation, tease, demand, dismiss, refusal, request, thanks, praise, apology, comfort, warning, none) with intensity and a confidence.
114
- - **The moves.** asks: wants a reply or an action. answers: gives what was asked for, or offers information. hedges: offers something second-hand or unsure. doesnt_know: says it does not know. withholds: declines, deflects or dodges. closes: winds the exchange down and wants no reply. other: anything else. After doesnt_know or withholds, a question is still open, so someone else may answer it.
115
- - **Hostile** means insult, threat, accusation, or demand at high intensity. A tag is hostile when the hostility score reaches 0.775.
116
  - **Confidence** is the chance the tag is right (calibrated). For a hostile tag it is the chance the line is hostile toward that person.
117
 
118
  ## Performance (held-out test sets, committee labels)
119
 
120
- The move:
121
 
122
- | Measure | Result |
123
- |---|---|
124
- | Move macro-F1, main test (824 lines) | 0.812 (95% interval 0.781 to 0.842) |
125
- | For scale: two frontier labellers agree with each other at | 0.845 on the same lines |
126
- | Answered or still open: lines labelled answers, doesnt_know or withholds (454) | 82.6% on the right side (78.8% to 85.8%). **The target was 90%, and it is not met.** 7.5% land on the wrong side |
127
- | The same, on characters' lines only (363) | 86.2% |
128
- | Asks or not (824 lines) | 95.3% (93.6% to 96.5%) |
129
- | Move macro-F1 on player lines (324), between characters (109), in settings never seen in training (293) | 0.737, 0.754, 0.712 |
130
- | Move confidence calibration error | 0.04 |
131
 
132
- The two frontier labellers themselves agree on "answered or still open" only about 89% of the time, so the 90% target sits near the labels' own ceiling. rc2 mixes answers with refusals most. A dodge often looks like an answer.
133
 
134
- The emotion and the acts, the same as rc1:
135
 
136
- | Measure | Result |
137
- |---|---|
138
- | Act macro-F1, main test (1,905 lines) | 0.678 (0.648 to 0.703) |
139
- | Emotion macro-F1, main test | 0.677 |
140
- | Natural player speech (900 lines): act F1, emotion F1 | 0.761, 0.711 |
141
- | Game settings never seen in training (900 lines): act F1 | 0.784 |
142
- | Harmless lines called hostile: prose-style, player speech | 1.1%, 1.7% |
143
- | Hostile lines caught: main test, player speech | 72%, 78% |
144
- | Calibration error: act tags, emotion | 0.08, 0.05 |
145
- | For scale: two frontier labellers agree with each other at | 0.766 act F1, 0.709 emotion F1 |
 
 
 
 
 
146
 
147
  ### With and without the optional inputs
148
 
@@ -150,16 +145,16 @@ The same test lines, scored again without who a line is said to, without the lin
150
 
151
  | Measure | As given | Without `to` | Without the line before | Without either |
152
  |---|---|---|---|---|
153
- | Act macro-F1, main test | 0.678 | 0.571 | 0.663 | 0.553 |
154
- | Emotion macro-F1, main test | 0.677 | 0.676 | 0.670 | 0.666 |
155
- | Harmless lines called hostile, prose | 1.1% | 0.7% | 0.9% | 0.8% |
156
- | Hostile lines caught, main test | 72% | 70% | 73% | 70% |
157
- | Move macro-F1 | 0.812 | 0.810 | 0.740 | 0.707 |
158
- | Answered or still open | 82.6% | 81.7% | 67.2% | 63.7% |
159
- | Asks or not | 95.3% | 94.5% | 92.0% | 89.2% |
160
 
161
- - **Send who a line is said to when you know it.** Without it, a "you" in a group is read as aimed at everyone. On those lines, harmless pairs called hostile go from 2.1% to 17.6%.
162
- - **Send the line before for the move.** A bare "Yes." is an answer only after a question. Without the line before, "answered or still open" drops 15 points.
163
 
164
  ### The 16-bit file
165
 
@@ -167,28 +162,26 @@ The same 5,204 test lines, tagged by `model_fp16.onnx` and by full precision, in
167
 
168
  | | As given | Without `to` | Without the line before | Without either |
169
  |---|---|---|---|---|
170
- | Lines with every tag the same as full precision | 100% | 100% | 99.98% (one move differs) | 99.98% (one act toward one person differs) |
171
 
172
- Rounded as the tables above show them, every measure is the same with the 16-bit file. Calibration error moves by 0.0012 at most.
173
 
174
  ## How it was made
175
 
176
- - **Model:** MiniLM-L6 (`sentence-transformers/all-MiniLM-L6-v2` at 1110a24, Apache-2.0) with six small heads, unchanged from rc1. For the move, trainable copies of its top two layers read its fourth layer's output, and a small head reads them. The encoder itself was frozen, so every rc1 output is unchanged.
177
- - **Distillation:** the emotion and the acts from three Ettin-1B teachers (`jhu-clsp/ettin-encoder-1b`, MIT) over 359,424 lines, plus 11,486 lines labelled directly. The move from three more Ettin-1B teachers, fine-tuned on the move labels, over 113,201 lines, plus 3,296 labelled training lines (noisy copies included).
178
- - **Labels:** the majority vote of a committee of three large language models. 4,992 lines were labelled for the move.
179
- - **Lines:** our own game dialogue, Project Gutenberg dialogue, GoEmotions (Apache-2.0), Civil Comments (CC0), and lines written for this project by Qwen3.8-27B. The last include 95,000 of natural player speech, 48,000 aimed at sarcasm, doubling down and implied threats, and 9,126 short exchanges for the rarer moves: rumours, dodges, "don't know", closings, and exchanges between characters.
180
- - **Calibration:** the hostility threshold and the act and emotion curves are rc1's. A curve for the move's confidence was fitted on the dev set.
181
- - **The 16-bit file:** full precision's graph with every weight tensor of 1,024 values or more stored in 16 bits and turned back into 32 bits by the graph itself. Inputs, outputs and arithmetic stay 32-bit, so the same runtime and the same calibration curves serve both files.
182
 
183
  ## Use and limits
184
 
185
  - **Use it** for character reactions and turn-taking in multi-character conversations: faces, feeling meters, moods, and who speaks next. **Do not use it** for moderation, safety decisions or judging real people.
186
  - **English only.** It reads one previous line.
187
- - **The move is read from the words.** It cannot know whether a character really knows something or is lying: "I don't know" is "doesn't know".
188
- - **It misses things.** It gets "answered or still open" wrong about one time in six. It misses about two in three sarcastic jabs and about one in four hostile lines.
189
- - **Player lines are harder** than characters' lines for the move (macro-F1 0.74 against 0.81), and new settings harder still (0.71).
190
- - **About 30% slower than rc1**, because two extra encoder layers run for the move.
191
- - **The 16-bit file saves download and disk, not time or memory.** It has been timed on CPU only.
192
  - **It has only been tested in game settings.** Its test lines are committee-labelled generated and game dialogue, not logs from real players or from other multi-character voice experiences.
193
 
194
- The numbers come from the training repository's reports: for the 16-bit file, `2026-10-09-linecls-precision.md` (at commit 9498f60); for the model, `2026-10-09-linecls-r8-exchange.md` (at commit a31743c) and, for the emotion and act heads unchanged since rc1, `2026-10-08-linecls-r6-sarcasm.md` and `2026-10-08-linecls-r7-calibration.md`. The full-precision files were packaged from git tag `spellspeak-tone-rc2` (commit 691b8e2), the 16-bit file was built from them by the code at commit 0c8ed10, and this release was packaged from git tag `spellspeak-tone-rc2.2`. That repository is private.
 
20
  - dialogue
21
  ---
22
 
23
+ # SpellSpeak Tone · RC3
24
 
25
  When several AI characters share one conversation, as in a game, an interactive story or any other multi-character voice experience, each of them needs a structured reading of every line: how the speaker sounds, what the line does to each person present, and whether it leaves a question open for someone else to answer. The language model voicing a character could produce that reading, but only through extra output tokens, training or tool calls. That adds latency to every turn, takes capacity from the dialogue, risks malformed output, and repeats the work for each character in the scene.
26
 
27
+ SpellSpeak Tone treats the reading as multi-task classification by a separate small model. One pass over a line predicts the speaker's emotion, the social act directed at each person present with a hostility probability per person, and the move the line makes in the conversation: does it ask, answer, pass on a rumour, say it does not know, hold back, or close the exchange. The application uses the tags for faces, feelings and who speaks next. Its constraints are millisecond inference on a CPU, robustness to unstructured user input, and rare false hostility.
28
 
29
  [Examples and source](https://github.com/spellspeak/models/tree/main/tone)
30
 
31
+ ## What is new in RC3
32
 
33
+ - **It reads sarcasm.** On a held-out test of game dialogue in settings it never saw, it catches 73% of the sarcastic lines that are hostile toward someone (rc2: 52%), and reads 6% of sarcastic lines at face value, as thanks or praise (rc2: 14%). "Sure you will", "what gave it away?" and "wow, so helpful" now read as the jabs they are. Sarcasm about the weather, about oneself or between friends, and sincere lines with the same words ("thanks a lot" after good news), are called hostile less often than before: 3.3% against 5.5%.
34
+ - **Insults and teases in a row.** A short follow-up such as "a disgrace" after an insult is read as the insult it continues, and a complaint about a thing followed by the same words is not. Scripted feeling-meter scenes replayed with the model's own tags: 26 of 27 (rc2: 24).
35
+ - **A 16-bit file, as in rc2.1.** `model_fp16.onnx` is half the size (55.7 MB) and gives the same tags on 99.96% or more of the test lines in every input setting.
36
+ - **What it costs.** Emotion macro-F1 on the main test is 0.649 against rc2's 0.677, most of it in rare emotions (disgust, fear). The move on lines between characters is weaker (0.633 against 0.754). Without who a line is said to, harmless player lines are called hostile more often (3.9% against 1.9%).
 
37
 
38
  | | |
39
  |---|---|
40
+ | **Released** | 2026-10-13 · rc3 (package `spellspeak-tone-rc3.1`) |
41
  | **What it does** | Reads one line of dialogue and tags the speaker's emotion, the social act toward each person present, and the line's move in the conversation |
42
+ | **What changed from rc2** | The encoder was trained again with sarcasm and its look-alikes, and the move head rebuilt on it. Same outputs, same size, same speed. The hostility threshold is 0.675 (rc2: 0.775), set so that harmless player lines are called hostile no more often than with rc2 |
43
+ | **Size** | 27.9M parameters · 111.1 MB ONNX in full precision (fp32), 55.7 MB with the weights in 16 bits (`model_fp16.onnx`) |
44
+ | **Speed** | p95 per line on 4 CPU threads (Apple M1 Max): 9.6 ms for a scene of 8 people, the same as rc2 timed side by side. On 1 thread: 28.6 ms. The 16-bit file takes the same time |
45
  | **Runtime** | ONNX Runtime (1.24 or later) and tokenizers, on CPU. A Python reference runtime ships with it |
46
  | **Files** | `model.opt.onnx` (the graph to run), `model.onnx` (the same graph unoptimized), `model_fp16.onnx` (the graph to run with its weights in 16 bits), `tokenizer.json` and `config.json`, with the reference runtime (`runtime/`), `example.py` and `requirements.txt`. `MANIFEST.json` lists every file with its size and sha256 |
47
  | **Licence** | Apache-2.0 |
 
50
 
51
  ```bash
52
  pip install -U huggingface_hub
53
+ hf download spellspeak/tone --revision rc3 --local-dir spellspeak-tone
54
  cd spellspeak-tone
55
  pip install -r requirements.txt
56
  python example.py
 
60
 
61
  ```
62
  player to everyone: 'Who knows where the mill key is?'
63
+ move asks (0.92) · emotion neutral · acts: none
64
  Mara to player: "It's under the counter, love."
65
+ move answers (0.92) · emotion neutral · acts: none
66
  Tomas to player: 'I heard the miller had it last.'
67
  move hedges (0.98) · emotion neutral · acts: none
68
  Wren to player: 'Why do you want to know?'
69
+ move withholds (0.61) · emotion neutral · acts: none
70
  Tomas to player: 'No idea, sorry.'
71
  move doesnt_know (0.99) · emotion neutral · acts: player apology
72
  player to Mara: "thanks, that's all i needed"
73
+ move closes (0.98) · emotion happy · acts: Mara thanks
74
  player to Tomas: "Touch my horse again and you'll regret it."
75
+ move other (0.71) · emotion angry · acts: Tomas threat
76
  ```
77
 
78
+ The small download leaves out the two full-precision graphs and runs the 16-bit file. It prints the same tags:
 
 
79
 
80
  ```bash
81
+ hf download spellspeak/tone --revision rc3 --local-dir spellspeak-tone-small --exclude model.onnx --exclude model.opt.onnx
82
  cd spellspeak-tone-small
83
  pip install -r requirements.txt
84
  python example.py --fp16
85
  ```
86
 
 
 
87
  In your own code, with the downloaded folder as the working directory:
88
 
89
  ```python
 
95
 
96
  tone = LineClassifier(".") # full precision (model.opt.onnx), the default
97
  # tone = LineClassifier(".", precision="fp16") # the 16-bit file (model_fp16.onnx), for the small download
98
+ tags = tone.tag(LineInput(speaker="player", speaker_kind="player", to="Mara", targets=["Mara", "Tomas"],
99
+ text="oh great, thanks a lot", previous="Mara: Sorry, we're out of rooms tonight."))
100
+ print(tags.emotion, tags.acts["Mara"]) # angry act='insult' ... after bad news; after good news the same words read as thanks
 
101
  ```
102
 
103
  ## Input and output
104
 
105
+ - **Input:** the line, the line before it, who speaks (the player or a character), who it is said to (or the room), and everyone present (tested up to 8 people; 160 tokens at most). Who it is said to and the line before are optional, and help: see the table below. Sarcasm is mostly read from the line before.
106
+ - **Per line:** an emotion (neutral, happy, amused, angry, sad, afraid, surprised, disgusted, contemptuous) with intensity (low, medium, high) and a confidence. Whether the line is only a listening sound such as "mm-hm" (backchannel). The **move**, with a confidence.
107
  - **Per person:** an act (insult, threat, accusation, tease, demand, dismiss, refusal, request, thanks, praise, apology, comfort, warning, none) with intensity and a confidence.
108
+ - **The moves.** asks: wants a reply or an action. answers: gives what was asked for, or offers information. hedges: offers something second-hand or unsure. doesnt_know: says it does not know. withholds: declines, deflects or dodges. closes: winds the exchange down and wants no reply. other: anything else.
109
+ - **Hostile** means insult, threat, accusation, or demand at high intensity. A tag is hostile when the hostility score reaches 0.675.
110
  - **Confidence** is the chance the tag is right (calibrated). For a hostile tag it is the chance the line is hostile toward that person.
111
 
112
  ## Performance (held-out test sets, committee labels)
113
 
114
+ Sarcasm, on 600 lines of game dialogue in six settings that no training line uses (397 sarcastic, 203 sincere look-alikes):
115
 
116
+ | Measure | rc2 | rc3 |
117
+ |---|---|---|
118
+ | Sarcastic lines hostile toward someone: caught (141 pairs) | 52% | 73% (21 points more; 95% interval 13 to 30) |
119
+ | Sarcastic lines read at face value (294) | 14% | 6% |
120
+ | Look-alikes and harmless sarcasm called hostile | 5.5% | 3.3% |
 
 
 
 
121
 
122
+ On sarcasm from sitcoms and Reddit, which is mostly not aimed at anyone present, rc3 is about level with rc2.
123
 
124
+ The emotion, the acts and the move:
125
 
126
+ | Measure | rc2 | rc3 |
127
+ |---|---|---|
128
+ | Act macro-F1, main test (1,905 lines) | 0.678 | 0.661 (0.630 to 0.689) |
129
+ | Emotion macro-F1, main test | 0.677 | 0.649 |
130
+ | Natural player speech (900 lines): act F1, emotion F1 | 0.761, 0.711 | 0.781, 0.687 |
131
+ | Game settings never seen in training (900 lines): act F1 | 0.784 | 0.771 |
132
+ | Harmless lines called hostile: prose-style, player speech | 1.1%, 1.7% | 1.4%, 2.3% |
133
+ | Hostile lines caught: main test, player speech | 72%, 78% | 76%, 84% |
134
+ | Move macro-F1 (824 lines) | 0.812 | 0.816 (0.785 to 0.845) |
135
+ | Answered or still open (454 lines) | 82.6% | 83.0% |
136
+ | Asks or not (824 lines) | 95.3% | 96.0% |
137
+ | Move macro-F1 on player lines, between characters | 0.737, 0.754 | 0.774, 0.633 |
138
+ | Calibration error: act tags, emotion, move | 0.08, 0.05, 0.04 | 0.07, 0.05, 0.05 |
139
+
140
+ For scale, two frontier labellers agree with each other at 0.766 act F1, 0.709 emotion F1 and 0.845 move F1.
141
 
142
  ### With and without the optional inputs
143
 
 
145
 
146
  | Measure | As given | Without `to` | Without the line before | Without either |
147
  |---|---|---|---|---|
148
+ | Act macro-F1, main test | 0.661 | 0.564 | 0.656 | 0.552 |
149
+ | Emotion macro-F1, main test | 0.649 | 0.649 | 0.640 | 0.636 |
150
+ | Harmless lines called hostile, prose | 1.4% | 1.5% | 1.3% | 1.8% |
151
+ | Harmless lines called hostile, player speech | 2.3% | 3.9% | 2.9% | 5.7% |
152
+ | Hostile lines caught, main test | 76% | 77% | 76% | 77% |
153
+ | Move macro-F1 | 0.816 | 0.805 | 0.722 | 0.683 |
154
+ | Answered or still open | 83.0% | 80.0% | 66.1% | 60.1% |
155
 
156
+ - **Send who a line is said to when you know it.** Without it, more harmless player lines are called hostile.
157
+ - **Send the line before.** Sarcasm and the move are both read from it.
158
 
159
  ### The 16-bit file
160
 
 
162
 
163
  | | As given | Without `to` | Without the line before | Without either |
164
  |---|---|---|---|---|
165
+ | Lines with every tag the same as full precision | 100% | 99.98% (one act toward one person differs) | 100% | 99.96% (two lines differ) |
166
 
167
+ Every measure in the tables above is the same with the 16-bit file. It computes in 32 bits, so it saves the download and the disk, not time or memory (224 MB resident against 205 MB).
168
 
169
  ## How it was made
170
 
171
+ - **Model:** MiniLM-L6 (`sentence-transformers/all-MiniLM-L6-v2` at 1110a24, Apache-2.0) with six small heads. For the move, trainable copies of its top two layers read its fourth layer's output, and a small head reads them.
172
+ - **Distillation:** the emotion and the acts from three Ettin-1B teachers (`jhu-clsp/ettin-encoder-1b`, MIT) over 374,534 lines, plus 20,286 lines labelled directly. The move from rc2's three move teachers over 113,201 lines, plus 3,296 labelled training lines.
173
+ - **Labels:** the majority vote of a committee of three large language models.
174
+ - **Lines:** our own game dialogue, Project Gutenberg dialogue, GoEmotions (Apache-2.0), Civil Comments (CC0), and lines written for this project by Qwen3.8-27B. New in rc3: 40,758 lines of sarcasm in twelve kinds with sincere look-alikes, and 8,284 of a speaker's two lines in a row. Sitcom and Reddit sarcasm was used only as a test.
175
+ - **Calibration:** the hostility threshold was matched on a player-speech dev set; the act, emotion and move curves were fitted on dev.
176
+ - **The 16-bit file:** full precision's graph with every weight tensor of 1,024 values or more stored in 16 bits and turned back into 32 bits by the graph itself, so the same runtime and calibration serve both files.
177
 
178
  ## Use and limits
179
 
180
  - **Use it** for character reactions and turn-taking in multi-character conversations: faces, feeling meters, moods, and who speaks next. **Do not use it** for moderation, safety decisions or judging real people.
181
  - **English only.** It reads one previous line.
182
+ - **It misses things.** It misses about one in four hostile sarcastic lines in game dialogue, and most sarcasm in sitcom and Reddit text. It gets "answered or still open" wrong about one time in six.
183
+ - **Rare emotions are weaker than in rc2**, disgust and fear most.
184
+ - **The move on lines between characters is weaker than in rc2** (0.633 against 0.754).
 
 
185
  - **It has only been tested in game settings.** Its test lines are committee-labelled generated and game dialogue, not logs from real players or from other multi-character voice experiences.
186
 
187
+ The numbers come from the training repository's report for this release, `2026-10-13-linecls-r9-sarcasm.md` (at commit ecaafed), with the open-data sarcasm test in `2026-10-13-linecls-r9-sarcasm-test.md`. The full-precision files were packaged from git tag `spellspeak-tone-rc3` (commit c85d633), and this release was packaged from git tag `spellspeak-tone-rc3.1`. That repository is private.
config.json CHANGED
@@ -1,11 +1,11 @@
1
  {
2
- "run_id": "linecls-r8-exchange-t3b2",
3
  "encoder": "minilm-l6",
4
  "repo": "sentence-transformers/all-MiniLM-L6-v2",
5
  "revision": "1110a243fdf4706b3f48f1d95db1a4f5529b4d41",
6
  "max_len": 160,
7
  "render": "linecls-input-0.1",
8
- "tau": 0.775,
9
  "acts": [
10
  "insult",
11
  "threat",
@@ -22,7 +22,7 @@
22
  "warning",
23
  "none"
24
  ],
25
- "tau_trained": 0.25,
26
  "tau_rule": "matched to r5's false hostility on player-speech dev (exp-235)",
27
  "emotions": [
28
  "neutral",
@@ -42,28 +42,28 @@
42
  ],
43
  "calibration": {
44
  "hostility": {
45
- "a": 1.115388,
46
- "b": -1.202297,
47
- "items": 2999
48
  },
49
  "act": {
50
- "a": 1.175358,
51
- "b": 1.357617,
52
- "items": 2628
53
  },
54
  "emotion": {
55
- "a": 1.295843,
56
- "b": 0.491525,
57
- "items": 1957
58
  },
59
  "exchange": {
60
- "a": 1.278981,
61
- "b": 0.456079,
62
  "items": 680
63
  }
64
  },
65
  "calibration_note": "Platt curves fitted on dev (exp-237); a confidence is the chance the tag is right; r8: the exchange curve fitted on the exchange dev lines (exp-247)",
66
- "base_run_id": "linecls-r2-student-minilm-l6-r6s1-seed1",
67
  "exchanges": [
68
  "asks",
69
  "answers",
@@ -77,7 +77,7 @@
77
  "variant": "t3-b2-soft",
78
  "kind": "attn4",
79
  "branch": 2,
80
- "seed": 2,
81
  "params": 4290823
82
  },
83
  "contract": "line-tags-0.2"
 
1
  {
2
+ "run_id": "linecls-r9-exchange",
3
  "encoder": "minilm-l6",
4
  "repo": "sentence-transformers/all-MiniLM-L6-v2",
5
  "revision": "1110a243fdf4706b3f48f1d95db1a4f5529b4d41",
6
  "max_len": 160,
7
  "render": "linecls-input-0.1",
8
+ "tau": 0.675,
9
  "acts": [
10
  "insult",
11
  "threat",
 
22
  "warning",
23
  "none"
24
  ],
25
+ "tau_trained": 0.325,
26
  "tau_rule": "matched to r5's false hostility on player-speech dev (exp-235)",
27
  "emotions": [
28
  "neutral",
 
42
  ],
43
  "calibration": {
44
  "hostility": {
45
+ "a": 1.043348,
46
+ "b": -1.115049,
47
+ "items": 4119
48
  },
49
  "act": {
50
+ "a": 1.033672,
51
+ "b": 1.253065,
52
+ "items": 3422
53
  },
54
  "emotion": {
55
+ "a": 1.136742,
56
+ "b": 0.192477,
57
+ "items": 2793
58
  },
59
  "exchange": {
60
+ "a": 1.063282,
61
+ "b": 0.528266,
62
  "items": 680
63
  }
64
  },
65
  "calibration_note": "Platt curves fitted on dev (exp-237); a confidence is the chance the tag is right; r8: the exchange curve fitted on the exchange dev lines (exp-247)",
66
+ "base_run_id": "linecls-r2-student-minilm-l6-r9bs1-seed1",
67
  "exchanges": [
68
  "asks",
69
  "answers",
 
77
  "variant": "t3-b2-soft",
78
  "kind": "attn4",
79
  "branch": 2,
80
+ "seed": 0,
81
  "params": 4290823
82
  },
83
  "contract": "line-tags-0.2"
example.py CHANGED
@@ -5,7 +5,7 @@
5
  python example.py --fp16 # the 16-bit file: model_fp16.onnx, half the size
6
 
7
  The model files are read from this folder. Without them, as in a git checkout, they are downloaded once from Hugging
8
- Face (spellspeak/tone, release rc2.1). With --fp16, only model_fp16.onnx, tokenizer.json and config.json are needed.
9
  """
10
  import pathlib
11
  import sys
@@ -23,7 +23,7 @@ model_dir = HERE
23
  if not all((HERE / f).exists() for f in MODEL_FILES):
24
  from huggingface_hub import snapshot_download
25
 
26
- model_dir = snapshot_download("spellspeak/tone", revision="rc2.1", allow_patterns=MODEL_FILES)
27
  tone = LineClassifier(model_dir, threads=4, precision=PRECISION)
28
 
29
  # Three characters and the player. Each line names who speaks, who it is to (None: the room) and the line before it.
 
5
  python example.py --fp16 # the 16-bit file: model_fp16.onnx, half the size
6
 
7
  The model files are read from this folder. Without them, as in a git checkout, they are downloaded once from Hugging
8
+ Face (spellspeak/tone, release rc3). With --fp16, only model_fp16.onnx, tokenizer.json and config.json are needed.
9
  """
10
  import pathlib
11
  import sys
 
23
  if not all((HERE / f).exists() for f in MODEL_FILES):
24
  from huggingface_hub import snapshot_download
25
 
26
+ model_dir = snapshot_download("spellspeak/tone", revision="rc3", allow_patterns=MODEL_FILES)
27
  tone = LineClassifier(model_dir, threads=4, precision=PRECISION)
28
 
29
  # Three characters and the player. Each line names who speaks, who it is to (None: the room) and the line before it.
model.onnx CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:ee4d0f23840e74d4b529de7ce3f5f057a2202fc95cb1166a0392c833919a3e80
3
  size 111170339
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:dc5aa341edd3e21f83f782900b7a9737f88e671fb2a9ce44b190236f65c4a199
3
  size 111170339
model.opt.onnx CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:d6809b08375415f0cd2bbc0282d9d3c8d8282b010001ee52559f7fae5b857e60
3
  size 111132591
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f53f61ddc0823ff6ffbecd5cdeac49888e08b5156adcebf7ce5ae6e69f3ed425
3
  size 111132591
model_fp16.onnx CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:08201561726a6e80cf454ab57e786b268ee428d152dc9299f80941e541ad11d7
3
  size 55694715
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:59754462606ddc50086c4cbba14b41bf336d504047f4160792f747cab257355b
3
  size 55694715