SpellSpeak Tone rc3
Browse files- MANIFEST.json +12 -12
- README.md +70 -77
- config.json +16 -16
- example.py +2 -2
- model.onnx +1 -1
- model.opt.onnx +1 -1
- model_fp16.onnx +1 -1
MANIFEST.json
CHANGED
|
@@ -1,33 +1,33 @@
|
|
| 1 |
{
|
| 2 |
"name": "SpellSpeak Tone",
|
| 3 |
-
"release": "
|
| 4 |
-
"package": "spellspeak-tone-
|
| 5 |
-
"created": "2026-10-
|
| 6 |
"contract": "line-tags-0.2",
|
| 7 |
"files": {
|
| 8 |
"README.md": {
|
| 9 |
-
"bytes":
|
| 10 |
-
"sha256": "
|
| 11 |
},
|
| 12 |
"config.json": {
|
| 13 |
-
"bytes":
|
| 14 |
-
"sha256": "
|
| 15 |
},
|
| 16 |
"example.py": {
|
| 17 |
-
"bytes":
|
| 18 |
-
"sha256": "
|
| 19 |
},
|
| 20 |
"model.onnx": {
|
| 21 |
"bytes": 111170339,
|
| 22 |
-
"sha256": "
|
| 23 |
},
|
| 24 |
"model.opt.onnx": {
|
| 25 |
"bytes": 111132591,
|
| 26 |
-
"sha256": "
|
| 27 |
},
|
| 28 |
"model_fp16.onnx": {
|
| 29 |
"bytes": 55694715,
|
| 30 |
-
"sha256": "
|
| 31 |
},
|
| 32 |
"requirements.txt": {
|
| 33 |
"bytes": 287,
|
|
|
|
| 1 |
{
|
| 2 |
"name": "SpellSpeak Tone",
|
| 3 |
+
"release": "rc3",
|
| 4 |
+
"package": "spellspeak-tone-rc3.1",
|
| 5 |
+
"created": "2026-10-13T18:24:17Z",
|
| 6 |
"contract": "line-tags-0.2",
|
| 7 |
"files": {
|
| 8 |
"README.md": {
|
| 9 |
+
"bytes": 13322,
|
| 10 |
+
"sha256": "02f6f2312c332c2e43995d0da70b04452b2c03e47ee516ff45066b20957ba793"
|
| 11 |
},
|
| 12 |
"config.json": {
|
| 13 |
+
"bytes": 1512,
|
| 14 |
+
"sha256": "a951d88defbccbe43fa103e8a24e9b233ea6cfe2cda3fd8fec574f67510f7ca8"
|
| 15 |
},
|
| 16 |
"example.py": {
|
| 17 |
+
"bytes": 2926,
|
| 18 |
+
"sha256": "efd2134c6eb28ac84bab1c9ad932bb61d8de8bc123e4a372ffd98629abdaad97"
|
| 19 |
},
|
| 20 |
"model.onnx": {
|
| 21 |
"bytes": 111170339,
|
| 22 |
+
"sha256": "dc5aa341edd3e21f83f782900b7a9737f88e671fb2a9ce44b190236f65c4a199"
|
| 23 |
},
|
| 24 |
"model.opt.onnx": {
|
| 25 |
"bytes": 111132591,
|
| 26 |
+
"sha256": "f53f61ddc0823ff6ffbecd5cdeac49888e08b5156adcebf7ce5ae6e69f3ed425"
|
| 27 |
},
|
| 28 |
"model_fp16.onnx": {
|
| 29 |
"bytes": 55694715,
|
| 30 |
+
"sha256": "59754462606ddc50086c4cbba14b41bf336d504047f4160792f747cab257355b"
|
| 31 |
},
|
| 32 |
"requirements.txt": {
|
| 33 |
"bytes": 287,
|
README.md
CHANGED
|
@@ -20,29 +20,28 @@ tags:
|
|
| 20 |
- dialogue
|
| 21 |
---
|
| 22 |
|
| 23 |
-
# SpellSpeak Tone ·
|
| 24 |
|
| 25 |
When several AI characters share one conversation, as in a game, an interactive story or any other multi-character voice experience, each of them needs a structured reading of every line: how the speaker sounds, what the line does to each person present, and whether it leaves a question open for someone else to answer. The language model voicing a character could produce that reading, but only through extra output tokens, training or tool calls. That adds latency to every turn, takes capacity from the dialogue, risks malformed output, and repeats the work for each character in the scene.
|
| 26 |
|
| 27 |
-
SpellSpeak Tone treats the reading as multi-task classification by a separate small model. One pass over a line predicts the speaker's emotion, the social act directed at each person present with a hostility probability per person, and
|
| 28 |
|
| 29 |
[Examples and source](https://github.com/spellspeak/models/tree/main/tone)
|
| 30 |
|
| 31 |
-
## What is new in
|
| 32 |
|
| 33 |
-
- **
|
| 34 |
-
- **
|
| 35 |
-
- **
|
| 36 |
-
|
| 37 |
-
Full precision is the reference: every other number on this card is full precision's, and it is the file to keep for any further training.
|
| 38 |
|
| 39 |
| | |
|
| 40 |
|---|---|
|
| 41 |
-
| **Released** | 2026-10-
|
| 42 |
| **What it does** | Reads one line of dialogue and tags the speaker's emotion, the social act toward each person present, and the line's move in the conversation |
|
| 43 |
-
| **What changed from
|
| 44 |
-
| **Size** | 27.9M parameters
|
| 45 |
-
| **Speed** | p95 per line on 4 CPU threads (Apple M1 Max):
|
| 46 |
| **Runtime** | ONNX Runtime (1.24 or later) and tokenizers, on CPU. A Python reference runtime ships with it |
|
| 47 |
| **Files** | `model.opt.onnx` (the graph to run), `model.onnx` (the same graph unoptimized), `model_fp16.onnx` (the graph to run with its weights in 16 bits), `tokenizer.json` and `config.json`, with the reference runtime (`runtime/`), `example.py` and `requirements.txt`. `MANIFEST.json` lists every file with its size and sha256 |
|
| 48 |
| **Licence** | Apache-2.0 |
|
|
@@ -51,7 +50,7 @@ Full precision is the reference: every other number on this card is full precisi
|
|
| 51 |
|
| 52 |
```bash
|
| 53 |
pip install -U huggingface_hub
|
| 54 |
-
hf download spellspeak/tone --revision
|
| 55 |
cd spellspeak-tone
|
| 56 |
pip install -r requirements.txt
|
| 57 |
python example.py
|
|
@@ -61,34 +60,30 @@ The example puts three characters and the player in a room. The player asks ever
|
|
| 61 |
|
| 62 |
```
|
| 63 |
player to everyone: 'Who knows where the mill key is?'
|
| 64 |
-
move asks (0.
|
| 65 |
Mara to player: "It's under the counter, love."
|
| 66 |
-
move answers (0.
|
| 67 |
Tomas to player: 'I heard the miller had it last.'
|
| 68 |
move hedges (0.98) · emotion neutral · acts: none
|
| 69 |
Wren to player: 'Why do you want to know?'
|
| 70 |
-
move withholds (0.
|
| 71 |
Tomas to player: 'No idea, sorry.'
|
| 72 |
move doesnt_know (0.99) · emotion neutral · acts: player apology
|
| 73 |
player to Mara: "thanks, that's all i needed"
|
| 74 |
-
move closes (0.
|
| 75 |
player to Tomas: "Touch my horse again and you'll regret it."
|
| 76 |
-
move
|
| 77 |
```
|
| 78 |
|
| 79 |
-
|
| 80 |
-
|
| 81 |
-
The small download leaves out the two full-precision graphs, about 56 MB in all, and runs the 16-bit file:
|
| 82 |
|
| 83 |
```bash
|
| 84 |
-
hf download spellspeak/tone --revision
|
| 85 |
cd spellspeak-tone-small
|
| 86 |
pip install -r requirements.txt
|
| 87 |
python example.py --fp16
|
| 88 |
```
|
| 89 |
|
| 90 |
-
It prints exactly the same output.
|
| 91 |
-
|
| 92 |
In your own code, with the downloaded folder as the working directory:
|
| 93 |
|
| 94 |
```python
|
|
@@ -100,49 +95,49 @@ from harness.expression.classifier import LineClassifier
|
|
| 100 |
|
| 101 |
tone = LineClassifier(".") # full precision (model.opt.onnx), the default
|
| 102 |
# tone = LineClassifier(".", precision="fp16") # the 16-bit file (model_fp16.onnx), for the small download
|
| 103 |
-
tags = tone.tag(LineInput(speaker="
|
| 104 |
-
text="
|
| 105 |
-
print(tags.
|
| 106 |
-
print(tags.emotion, tags.acts["Tomas"]) # neutral, act='none' ...
|
| 107 |
```
|
| 108 |
|
| 109 |
## Input and output
|
| 110 |
|
| 111 |
-
- **Input:** the line, the line before it, who speaks (the player or a character), who it is said to (or the room), and everyone present (tested up to 8 people; 160 tokens at most). Who it is said to and the line before are optional, and help: see the table below.
|
| 112 |
-
- **Per line:** an emotion (neutral, happy, amused, angry, sad, afraid, surprised, disgusted, contemptuous) with intensity (low, medium, high) and a confidence. Whether the line is only a listening sound such as "mm-hm" (backchannel).
|
| 113 |
- **Per person:** an act (insult, threat, accusation, tease, demand, dismiss, refusal, request, thanks, praise, apology, comfort, warning, none) with intensity and a confidence.
|
| 114 |
-
- **The moves.** asks: wants a reply or an action. answers: gives what was asked for, or offers information. hedges: offers something second-hand or unsure. doesnt_know: says it does not know. withholds: declines, deflects or dodges. closes: winds the exchange down and wants no reply. other: anything else.
|
| 115 |
-
- **Hostile** means insult, threat, accusation, or demand at high intensity. A tag is hostile when the hostility score reaches 0.
|
| 116 |
- **Confidence** is the chance the tag is right (calibrated). For a hostile tag it is the chance the line is hostile toward that person.
|
| 117 |
|
| 118 |
## Performance (held-out test sets, committee labels)
|
| 119 |
|
| 120 |
-
|
| 121 |
|
| 122 |
-
| Measure |
|
| 123 |
-
|---|---|
|
| 124 |
-
|
|
| 125 |
-
|
|
| 126 |
-
|
|
| 127 |
-
| The same, on characters' lines only (363) | 86.2% |
|
| 128 |
-
| Asks or not (824 lines) | 95.3% (93.6% to 96.5%) |
|
| 129 |
-
| Move macro-F1 on player lines (324), between characters (109), in settings never seen in training (293) | 0.737, 0.754, 0.712 |
|
| 130 |
-
| Move confidence calibration error | 0.04 |
|
| 131 |
|
| 132 |
-
|
| 133 |
|
| 134 |
-
The emotion
|
| 135 |
|
| 136 |
-
| Measure |
|
| 137 |
-
|---|---|
|
| 138 |
-
| Act macro-F1, main test (1,905 lines) | 0.678 (0.
|
| 139 |
-
| Emotion macro-F1, main test | 0.677 |
|
| 140 |
-
| Natural player speech (900 lines): act F1, emotion F1 | 0.761, 0.711 |
|
| 141 |
-
| Game settings never seen in training (900 lines): act F1 | 0.784 |
|
| 142 |
-
| Harmless lines called hostile: prose-style, player speech | 1.1%, 1.7% |
|
| 143 |
-
| Hostile lines caught: main test, player speech | 72%, 78% |
|
| 144 |
-
|
|
| 145 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 146 |
|
| 147 |
### With and without the optional inputs
|
| 148 |
|
|
@@ -150,16 +145,16 @@ The same test lines, scored again without who a line is said to, without the lin
|
|
| 150 |
|
| 151 |
| Measure | As given | Without `to` | Without the line before | Without either |
|
| 152 |
|---|---|---|---|---|
|
| 153 |
-
| Act macro-F1, main test | 0.
|
| 154 |
-
| Emotion macro-F1, main test | 0.
|
| 155 |
-
| Harmless lines called hostile, prose | 1.
|
| 156 |
-
|
|
| 157 |
-
|
|
| 158 |
-
|
|
| 159 |
-
|
|
| 160 |
|
| 161 |
-
- **Send who a line is said to when you know it.** Without it,
|
| 162 |
-
- **Send the line before
|
| 163 |
|
| 164 |
### The 16-bit file
|
| 165 |
|
|
@@ -167,28 +162,26 @@ The same 5,204 test lines, tagged by `model_fp16.onnx` and by full precision, in
|
|
| 167 |
|
| 168 |
| | As given | Without `to` | Without the line before | Without either |
|
| 169 |
|---|---|---|---|---|
|
| 170 |
-
| Lines with every tag the same as full precision | 100% |
|
| 171 |
|
| 172 |
-
|
| 173 |
|
| 174 |
## How it was made
|
| 175 |
|
| 176 |
-
- **Model:** MiniLM-L6 (`sentence-transformers/all-MiniLM-L6-v2` at 1110a24, Apache-2.0) with six small heads
|
| 177 |
-
- **Distillation:** the emotion and the acts from three Ettin-1B teachers (`jhu-clsp/ettin-encoder-1b`, MIT) over
|
| 178 |
-
- **Labels:** the majority vote of a committee of three large language models.
|
| 179 |
-
- **Lines:** our own game dialogue, Project Gutenberg dialogue, GoEmotions (Apache-2.0), Civil Comments (CC0), and lines written for this project by Qwen3.8-27B.
|
| 180 |
-
- **Calibration:** the hostility threshold
|
| 181 |
-
- **The 16-bit file:** full precision's graph with every weight tensor of 1,024 values or more stored in 16 bits and turned back into 32 bits by the graph itself
|
| 182 |
|
| 183 |
## Use and limits
|
| 184 |
|
| 185 |
- **Use it** for character reactions and turn-taking in multi-character conversations: faces, feeling meters, moods, and who speaks next. **Do not use it** for moderation, safety decisions or judging real people.
|
| 186 |
- **English only.** It reads one previous line.
|
| 187 |
-
- **
|
| 188 |
-
- **
|
| 189 |
-
- **
|
| 190 |
-
- **About 30% slower than rc1**, because two extra encoder layers run for the move.
|
| 191 |
-
- **The 16-bit file saves download and disk, not time or memory.** It has been timed on CPU only.
|
| 192 |
- **It has only been tested in game settings.** Its test lines are committee-labelled generated and game dialogue, not logs from real players or from other multi-character voice experiences.
|
| 193 |
|
| 194 |
-
The numbers come from the training repository's
|
|
|
|
| 20 |
- dialogue
|
| 21 |
---
|
| 22 |
|
| 23 |
+
# SpellSpeak Tone · RC3
|
| 24 |
|
| 25 |
When several AI characters share one conversation, as in a game, an interactive story or any other multi-character voice experience, each of them needs a structured reading of every line: how the speaker sounds, what the line does to each person present, and whether it leaves a question open for someone else to answer. The language model voicing a character could produce that reading, but only through extra output tokens, training or tool calls. That adds latency to every turn, takes capacity from the dialogue, risks malformed output, and repeats the work for each character in the scene.
|
| 26 |
|
| 27 |
+
SpellSpeak Tone treats the reading as multi-task classification by a separate small model. One pass over a line predicts the speaker's emotion, the social act directed at each person present with a hostility probability per person, and the move the line makes in the conversation: does it ask, answer, pass on a rumour, say it does not know, hold back, or close the exchange. The application uses the tags for faces, feelings and who speaks next. Its constraints are millisecond inference on a CPU, robustness to unstructured user input, and rare false hostility.
|
| 28 |
|
| 29 |
[Examples and source](https://github.com/spellspeak/models/tree/main/tone)
|
| 30 |
|
| 31 |
+
## What is new in RC3
|
| 32 |
|
| 33 |
+
- **It reads sarcasm.** On a held-out test of game dialogue in settings it never saw, it catches 73% of the sarcastic lines that are hostile toward someone (rc2: 52%), and reads 6% of sarcastic lines at face value, as thanks or praise (rc2: 14%). "Sure you will", "what gave it away?" and "wow, so helpful" now read as the jabs they are. Sarcasm about the weather, about oneself or between friends, and sincere lines with the same words ("thanks a lot" after good news), are called hostile less often than before: 3.3% against 5.5%.
|
| 34 |
+
- **Insults and teases in a row.** A short follow-up such as "a disgrace" after an insult is read as the insult it continues, and a complaint about a thing followed by the same words is not. Scripted feeling-meter scenes replayed with the model's own tags: 26 of 27 (rc2: 24).
|
| 35 |
+
- **A 16-bit file, as in rc2.1.** `model_fp16.onnx` is half the size (55.7 MB) and gives the same tags on 99.96% or more of the test lines in every input setting.
|
| 36 |
+
- **What it costs.** Emotion macro-F1 on the main test is 0.649 against rc2's 0.677, most of it in rare emotions (disgust, fear). The move on lines between characters is weaker (0.633 against 0.754). Without who a line is said to, harmless player lines are called hostile more often (3.9% against 1.9%).
|
|
|
|
| 37 |
|
| 38 |
| | |
|
| 39 |
|---|---|
|
| 40 |
+
| **Released** | 2026-10-13 · rc3 (package `spellspeak-tone-rc3.1`) |
|
| 41 |
| **What it does** | Reads one line of dialogue and tags the speaker's emotion, the social act toward each person present, and the line's move in the conversation |
|
| 42 |
+
| **What changed from rc2** | The encoder was trained again with sarcasm and its look-alikes, and the move head rebuilt on it. Same outputs, same size, same speed. The hostility threshold is 0.675 (rc2: 0.775), set so that harmless player lines are called hostile no more often than with rc2 |
|
| 43 |
+
| **Size** | 27.9M parameters · 111.1 MB ONNX in full precision (fp32), 55.7 MB with the weights in 16 bits (`model_fp16.onnx`) |
|
| 44 |
+
| **Speed** | p95 per line on 4 CPU threads (Apple M1 Max): 9.6 ms for a scene of 8 people, the same as rc2 timed side by side. On 1 thread: 28.6 ms. The 16-bit file takes the same time |
|
| 45 |
| **Runtime** | ONNX Runtime (1.24 or later) and tokenizers, on CPU. A Python reference runtime ships with it |
|
| 46 |
| **Files** | `model.opt.onnx` (the graph to run), `model.onnx` (the same graph unoptimized), `model_fp16.onnx` (the graph to run with its weights in 16 bits), `tokenizer.json` and `config.json`, with the reference runtime (`runtime/`), `example.py` and `requirements.txt`. `MANIFEST.json` lists every file with its size and sha256 |
|
| 47 |
| **Licence** | Apache-2.0 |
|
|
|
|
| 50 |
|
| 51 |
```bash
|
| 52 |
pip install -U huggingface_hub
|
| 53 |
+
hf download spellspeak/tone --revision rc3 --local-dir spellspeak-tone
|
| 54 |
cd spellspeak-tone
|
| 55 |
pip install -r requirements.txt
|
| 56 |
python example.py
|
|
|
|
| 60 |
|
| 61 |
```
|
| 62 |
player to everyone: 'Who knows where the mill key is?'
|
| 63 |
+
move asks (0.92) · emotion neutral · acts: none
|
| 64 |
Mara to player: "It's under the counter, love."
|
| 65 |
+
move answers (0.92) · emotion neutral · acts: none
|
| 66 |
Tomas to player: 'I heard the miller had it last.'
|
| 67 |
move hedges (0.98) · emotion neutral · acts: none
|
| 68 |
Wren to player: 'Why do you want to know?'
|
| 69 |
+
move withholds (0.61) · emotion neutral · acts: none
|
| 70 |
Tomas to player: 'No idea, sorry.'
|
| 71 |
move doesnt_know (0.99) · emotion neutral · acts: player apology
|
| 72 |
player to Mara: "thanks, that's all i needed"
|
| 73 |
+
move closes (0.98) · emotion happy · acts: Mara thanks
|
| 74 |
player to Tomas: "Touch my horse again and you'll regret it."
|
| 75 |
+
move other (0.71) · emotion angry · acts: Tomas threat
|
| 76 |
```
|
| 77 |
|
| 78 |
+
The small download leaves out the two full-precision graphs and runs the 16-bit file. It prints the same tags:
|
|
|
|
|
|
|
| 79 |
|
| 80 |
```bash
|
| 81 |
+
hf download spellspeak/tone --revision rc3 --local-dir spellspeak-tone-small --exclude model.onnx --exclude model.opt.onnx
|
| 82 |
cd spellspeak-tone-small
|
| 83 |
pip install -r requirements.txt
|
| 84 |
python example.py --fp16
|
| 85 |
```
|
| 86 |
|
|
|
|
|
|
|
| 87 |
In your own code, with the downloaded folder as the working directory:
|
| 88 |
|
| 89 |
```python
|
|
|
|
| 95 |
|
| 96 |
tone = LineClassifier(".") # full precision (model.opt.onnx), the default
|
| 97 |
# tone = LineClassifier(".", precision="fp16") # the 16-bit file (model_fp16.onnx), for the small download
|
| 98 |
+
tags = tone.tag(LineInput(speaker="player", speaker_kind="player", to="Mara", targets=["Mara", "Tomas"],
|
| 99 |
+
text="oh great, thanks a lot", previous="Mara: Sorry, we're out of rooms tonight."))
|
| 100 |
+
print(tags.emotion, tags.acts["Mara"]) # angry act='insult' ... after bad news; after good news the same words read as thanks
|
|
|
|
| 101 |
```
|
| 102 |
|
| 103 |
## Input and output
|
| 104 |
|
| 105 |
+
- **Input:** the line, the line before it, who speaks (the player or a character), who it is said to (or the room), and everyone present (tested up to 8 people; 160 tokens at most). Who it is said to and the line before are optional, and help: see the table below. Sarcasm is mostly read from the line before.
|
| 106 |
+
- **Per line:** an emotion (neutral, happy, amused, angry, sad, afraid, surprised, disgusted, contemptuous) with intensity (low, medium, high) and a confidence. Whether the line is only a listening sound such as "mm-hm" (backchannel). The **move**, with a confidence.
|
| 107 |
- **Per person:** an act (insult, threat, accusation, tease, demand, dismiss, refusal, request, thanks, praise, apology, comfort, warning, none) with intensity and a confidence.
|
| 108 |
+
- **The moves.** asks: wants a reply or an action. answers: gives what was asked for, or offers information. hedges: offers something second-hand or unsure. doesnt_know: says it does not know. withholds: declines, deflects or dodges. closes: winds the exchange down and wants no reply. other: anything else.
|
| 109 |
+
- **Hostile** means insult, threat, accusation, or demand at high intensity. A tag is hostile when the hostility score reaches 0.675.
|
| 110 |
- **Confidence** is the chance the tag is right (calibrated). For a hostile tag it is the chance the line is hostile toward that person.
|
| 111 |
|
| 112 |
## Performance (held-out test sets, committee labels)
|
| 113 |
|
| 114 |
+
Sarcasm, on 600 lines of game dialogue in six settings that no training line uses (397 sarcastic, 203 sincere look-alikes):
|
| 115 |
|
| 116 |
+
| Measure | rc2 | rc3 |
|
| 117 |
+
|---|---|---|
|
| 118 |
+
| Sarcastic lines hostile toward someone: caught (141 pairs) | 52% | 73% (21 points more; 95% interval 13 to 30) |
|
| 119 |
+
| Sarcastic lines read at face value (294) | 14% | 6% |
|
| 120 |
+
| Look-alikes and harmless sarcasm called hostile | 5.5% | 3.3% |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 121 |
|
| 122 |
+
On sarcasm from sitcoms and Reddit, which is mostly not aimed at anyone present, rc3 is about level with rc2.
|
| 123 |
|
| 124 |
+
The emotion, the acts and the move:
|
| 125 |
|
| 126 |
+
| Measure | rc2 | rc3 |
|
| 127 |
+
|---|---|---|
|
| 128 |
+
| Act macro-F1, main test (1,905 lines) | 0.678 | 0.661 (0.630 to 0.689) |
|
| 129 |
+
| Emotion macro-F1, main test | 0.677 | 0.649 |
|
| 130 |
+
| Natural player speech (900 lines): act F1, emotion F1 | 0.761, 0.711 | 0.781, 0.687 |
|
| 131 |
+
| Game settings never seen in training (900 lines): act F1 | 0.784 | 0.771 |
|
| 132 |
+
| Harmless lines called hostile: prose-style, player speech | 1.1%, 1.7% | 1.4%, 2.3% |
|
| 133 |
+
| Hostile lines caught: main test, player speech | 72%, 78% | 76%, 84% |
|
| 134 |
+
| Move macro-F1 (824 lines) | 0.812 | 0.816 (0.785 to 0.845) |
|
| 135 |
+
| Answered or still open (454 lines) | 82.6% | 83.0% |
|
| 136 |
+
| Asks or not (824 lines) | 95.3% | 96.0% |
|
| 137 |
+
| Move macro-F1 on player lines, between characters | 0.737, 0.754 | 0.774, 0.633 |
|
| 138 |
+
| Calibration error: act tags, emotion, move | 0.08, 0.05, 0.04 | 0.07, 0.05, 0.05 |
|
| 139 |
+
|
| 140 |
+
For scale, two frontier labellers agree with each other at 0.766 act F1, 0.709 emotion F1 and 0.845 move F1.
|
| 141 |
|
| 142 |
### With and without the optional inputs
|
| 143 |
|
|
|
|
| 145 |
|
| 146 |
| Measure | As given | Without `to` | Without the line before | Without either |
|
| 147 |
|---|---|---|---|---|
|
| 148 |
+
| Act macro-F1, main test | 0.661 | 0.564 | 0.656 | 0.552 |
|
| 149 |
+
| Emotion macro-F1, main test | 0.649 | 0.649 | 0.640 | 0.636 |
|
| 150 |
+
| Harmless lines called hostile, prose | 1.4% | 1.5% | 1.3% | 1.8% |
|
| 151 |
+
| Harmless lines called hostile, player speech | 2.3% | 3.9% | 2.9% | 5.7% |
|
| 152 |
+
| Hostile lines caught, main test | 76% | 77% | 76% | 77% |
|
| 153 |
+
| Move macro-F1 | 0.816 | 0.805 | 0.722 | 0.683 |
|
| 154 |
+
| Answered or still open | 83.0% | 80.0% | 66.1% | 60.1% |
|
| 155 |
|
| 156 |
+
- **Send who a line is said to when you know it.** Without it, more harmless player lines are called hostile.
|
| 157 |
+
- **Send the line before.** Sarcasm and the move are both read from it.
|
| 158 |
|
| 159 |
### The 16-bit file
|
| 160 |
|
|
|
|
| 162 |
|
| 163 |
| | As given | Without `to` | Without the line before | Without either |
|
| 164 |
|---|---|---|---|---|
|
| 165 |
+
| Lines with every tag the same as full precision | 100% | 99.98% (one act toward one person differs) | 100% | 99.96% (two lines differ) |
|
| 166 |
|
| 167 |
+
Every measure in the tables above is the same with the 16-bit file. It computes in 32 bits, so it saves the download and the disk, not time or memory (224 MB resident against 205 MB).
|
| 168 |
|
| 169 |
## How it was made
|
| 170 |
|
| 171 |
+
- **Model:** MiniLM-L6 (`sentence-transformers/all-MiniLM-L6-v2` at 1110a24, Apache-2.0) with six small heads. For the move, trainable copies of its top two layers read its fourth layer's output, and a small head reads them.
|
| 172 |
+
- **Distillation:** the emotion and the acts from three Ettin-1B teachers (`jhu-clsp/ettin-encoder-1b`, MIT) over 374,534 lines, plus 20,286 lines labelled directly. The move from rc2's three move teachers over 113,201 lines, plus 3,296 labelled training lines.
|
| 173 |
+
- **Labels:** the majority vote of a committee of three large language models.
|
| 174 |
+
- **Lines:** our own game dialogue, Project Gutenberg dialogue, GoEmotions (Apache-2.0), Civil Comments (CC0), and lines written for this project by Qwen3.8-27B. New in rc3: 40,758 lines of sarcasm in twelve kinds with sincere look-alikes, and 8,284 of a speaker's two lines in a row. Sitcom and Reddit sarcasm was used only as a test.
|
| 175 |
+
- **Calibration:** the hostility threshold was matched on a player-speech dev set; the act, emotion and move curves were fitted on dev.
|
| 176 |
+
- **The 16-bit file:** full precision's graph with every weight tensor of 1,024 values or more stored in 16 bits and turned back into 32 bits by the graph itself, so the same runtime and calibration serve both files.
|
| 177 |
|
| 178 |
## Use and limits
|
| 179 |
|
| 180 |
- **Use it** for character reactions and turn-taking in multi-character conversations: faces, feeling meters, moods, and who speaks next. **Do not use it** for moderation, safety decisions or judging real people.
|
| 181 |
- **English only.** It reads one previous line.
|
| 182 |
+
- **It misses things.** It misses about one in four hostile sarcastic lines in game dialogue, and most sarcasm in sitcom and Reddit text. It gets "answered or still open" wrong about one time in six.
|
| 183 |
+
- **Rare emotions are weaker than in rc2**, disgust and fear most.
|
| 184 |
+
- **The move on lines between characters is weaker than in rc2** (0.633 against 0.754).
|
|
|
|
|
|
|
| 185 |
- **It has only been tested in game settings.** Its test lines are committee-labelled generated and game dialogue, not logs from real players or from other multi-character voice experiences.
|
| 186 |
|
| 187 |
+
The numbers come from the training repository's report for this release, `2026-10-13-linecls-r9-sarcasm.md` (at commit ecaafed), with the open-data sarcasm test in `2026-10-13-linecls-r9-sarcasm-test.md`. The full-precision files were packaged from git tag `spellspeak-tone-rc3` (commit c85d633), and this release was packaged from git tag `spellspeak-tone-rc3.1`. That repository is private.
|
config.json
CHANGED
|
@@ -1,11 +1,11 @@
|
|
| 1 |
{
|
| 2 |
-
"run_id": "linecls-
|
| 3 |
"encoder": "minilm-l6",
|
| 4 |
"repo": "sentence-transformers/all-MiniLM-L6-v2",
|
| 5 |
"revision": "1110a243fdf4706b3f48f1d95db1a4f5529b4d41",
|
| 6 |
"max_len": 160,
|
| 7 |
"render": "linecls-input-0.1",
|
| 8 |
-
"tau": 0.
|
| 9 |
"acts": [
|
| 10 |
"insult",
|
| 11 |
"threat",
|
|
@@ -22,7 +22,7 @@
|
|
| 22 |
"warning",
|
| 23 |
"none"
|
| 24 |
],
|
| 25 |
-
"tau_trained": 0.
|
| 26 |
"tau_rule": "matched to r5's false hostility on player-speech dev (exp-235)",
|
| 27 |
"emotions": [
|
| 28 |
"neutral",
|
|
@@ -42,28 +42,28 @@
|
|
| 42 |
],
|
| 43 |
"calibration": {
|
| 44 |
"hostility": {
|
| 45 |
-
"a": 1.
|
| 46 |
-
"b": -1.
|
| 47 |
-
"items":
|
| 48 |
},
|
| 49 |
"act": {
|
| 50 |
-
"a": 1.
|
| 51 |
-
"b": 1.
|
| 52 |
-
"items":
|
| 53 |
},
|
| 54 |
"emotion": {
|
| 55 |
-
"a": 1.
|
| 56 |
-
"b": 0.
|
| 57 |
-
"items":
|
| 58 |
},
|
| 59 |
"exchange": {
|
| 60 |
-
"a": 1.
|
| 61 |
-
"b": 0.
|
| 62 |
"items": 680
|
| 63 |
}
|
| 64 |
},
|
| 65 |
"calibration_note": "Platt curves fitted on dev (exp-237); a confidence is the chance the tag is right; r8: the exchange curve fitted on the exchange dev lines (exp-247)",
|
| 66 |
-
"base_run_id": "linecls-r2-student-minilm-l6-
|
| 67 |
"exchanges": [
|
| 68 |
"asks",
|
| 69 |
"answers",
|
|
@@ -77,7 +77,7 @@
|
|
| 77 |
"variant": "t3-b2-soft",
|
| 78 |
"kind": "attn4",
|
| 79 |
"branch": 2,
|
| 80 |
-
"seed":
|
| 81 |
"params": 4290823
|
| 82 |
},
|
| 83 |
"contract": "line-tags-0.2"
|
|
|
|
| 1 |
{
|
| 2 |
+
"run_id": "linecls-r9-exchange",
|
| 3 |
"encoder": "minilm-l6",
|
| 4 |
"repo": "sentence-transformers/all-MiniLM-L6-v2",
|
| 5 |
"revision": "1110a243fdf4706b3f48f1d95db1a4f5529b4d41",
|
| 6 |
"max_len": 160,
|
| 7 |
"render": "linecls-input-0.1",
|
| 8 |
+
"tau": 0.675,
|
| 9 |
"acts": [
|
| 10 |
"insult",
|
| 11 |
"threat",
|
|
|
|
| 22 |
"warning",
|
| 23 |
"none"
|
| 24 |
],
|
| 25 |
+
"tau_trained": 0.325,
|
| 26 |
"tau_rule": "matched to r5's false hostility on player-speech dev (exp-235)",
|
| 27 |
"emotions": [
|
| 28 |
"neutral",
|
|
|
|
| 42 |
],
|
| 43 |
"calibration": {
|
| 44 |
"hostility": {
|
| 45 |
+
"a": 1.043348,
|
| 46 |
+
"b": -1.115049,
|
| 47 |
+
"items": 4119
|
| 48 |
},
|
| 49 |
"act": {
|
| 50 |
+
"a": 1.033672,
|
| 51 |
+
"b": 1.253065,
|
| 52 |
+
"items": 3422
|
| 53 |
},
|
| 54 |
"emotion": {
|
| 55 |
+
"a": 1.136742,
|
| 56 |
+
"b": 0.192477,
|
| 57 |
+
"items": 2793
|
| 58 |
},
|
| 59 |
"exchange": {
|
| 60 |
+
"a": 1.063282,
|
| 61 |
+
"b": 0.528266,
|
| 62 |
"items": 680
|
| 63 |
}
|
| 64 |
},
|
| 65 |
"calibration_note": "Platt curves fitted on dev (exp-237); a confidence is the chance the tag is right; r8: the exchange curve fitted on the exchange dev lines (exp-247)",
|
| 66 |
+
"base_run_id": "linecls-r2-student-minilm-l6-r9bs1-seed1",
|
| 67 |
"exchanges": [
|
| 68 |
"asks",
|
| 69 |
"answers",
|
|
|
|
| 77 |
"variant": "t3-b2-soft",
|
| 78 |
"kind": "attn4",
|
| 79 |
"branch": 2,
|
| 80 |
+
"seed": 0,
|
| 81 |
"params": 4290823
|
| 82 |
},
|
| 83 |
"contract": "line-tags-0.2"
|
example.py
CHANGED
|
@@ -5,7 +5,7 @@
|
|
| 5 |
python example.py --fp16 # the 16-bit file: model_fp16.onnx, half the size
|
| 6 |
|
| 7 |
The model files are read from this folder. Without them, as in a git checkout, they are downloaded once from Hugging
|
| 8 |
-
Face (spellspeak/tone, release
|
| 9 |
"""
|
| 10 |
import pathlib
|
| 11 |
import sys
|
|
@@ -23,7 +23,7 @@ model_dir = HERE
|
|
| 23 |
if not all((HERE / f).exists() for f in MODEL_FILES):
|
| 24 |
from huggingface_hub import snapshot_download
|
| 25 |
|
| 26 |
-
model_dir = snapshot_download("spellspeak/tone", revision="
|
| 27 |
tone = LineClassifier(model_dir, threads=4, precision=PRECISION)
|
| 28 |
|
| 29 |
# Three characters and the player. Each line names who speaks, who it is to (None: the room) and the line before it.
|
|
|
|
| 5 |
python example.py --fp16 # the 16-bit file: model_fp16.onnx, half the size
|
| 6 |
|
| 7 |
The model files are read from this folder. Without them, as in a git checkout, they are downloaded once from Hugging
|
| 8 |
+
Face (spellspeak/tone, release rc3). With --fp16, only model_fp16.onnx, tokenizer.json and config.json are needed.
|
| 9 |
"""
|
| 10 |
import pathlib
|
| 11 |
import sys
|
|
|
|
| 23 |
if not all((HERE / f).exists() for f in MODEL_FILES):
|
| 24 |
from huggingface_hub import snapshot_download
|
| 25 |
|
| 26 |
+
model_dir = snapshot_download("spellspeak/tone", revision="rc3", allow_patterns=MODEL_FILES)
|
| 27 |
tone = LineClassifier(model_dir, threads=4, precision=PRECISION)
|
| 28 |
|
| 29 |
# Three characters and the player. Each line names who speaks, who it is to (None: the room) and the line before it.
|
model.onnx
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
size 111170339
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:dc5aa341edd3e21f83f782900b7a9737f88e671fb2a9ce44b190236f65c4a199
|
| 3 |
size 111170339
|
model.opt.onnx
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
size 111132591
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:f53f61ddc0823ff6ffbecd5cdeac49888e08b5156adcebf7ce5ae6e69f3ed425
|
| 3 |
size 111132591
|
model_fp16.onnx
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
size 55694715
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:59754462606ddc50086c4cbba14b41bf336d504047f4160792f747cab257355b
|
| 3 |
size 55694715
|