KoljaB/backchannel-cue-32m

Backchannel Cue 32M

GitHub · X · YouTube · Discord

When people listen, they say "mhm", "right" or "oh no" while the other person is still talking. These small signals are called backchannels. Voice assistants usually stay silent until the user finishes, which feels like talking to a wall.

This model adds that listening behavior. Give it what the user has said so far when they pause, and it tells you whether a listener would respond, and how. It runs on a CPU in milliseconds, so you can call it at every pause without an LLM.

• Input: the words of the current user turn, plus a few counters. No audio.
• Output: a cue probability, fitting reactions, the mood of the text and the likely wording.
• Speed: about 5 ms on short text and 12 ms on 40 words, on one CPU thread.
• Size: 129 MB (ONNX, float32).
• Language: English.

Quick start

pip install onnxruntime tokenizers numpy huggingface_hub
python example.py

example.py is a complete policy you can copy. Its main call is:

from example import decide

decide("my dad got really sick last week")   # e.g. "Oh no."
decide("i wanted to tell you about")         # "About what?"
decide("what time is it")                    # usually None: stay quiet

Call it about 0.45 seconds into a pause, on your live transcript. Play the returned text from pre-rendered audio if you can: generating it on the fly usually arrives too late. If the user has clearly finished their turn, let your assistant reply instead.

Outputs

Output	Meaning
gate	Probability that a listener would respond at this moment
reactions	A score for each kind of response (table below)
mood	neutral, bad, good or funny
forms	The wording, 61 classes such as "mhm", "oh no" or "right"
echo	Which ask-back question fits

Reaction	Example
continuer	"Mhm."
receipt	"I see."
agree	"Exactly."
surprise	"Really?"
wow	"Wow."
sympathy	"Oh no."
praise	"That's great."
laugh	"Haha."
echo	"About what?" after "I wanted to tell you about"

config.json holds all labels and which wording belongs to which reaction. (meta.json is the same file, kept for older copies of the example.)

Timing inputs

The model takes five numbers, which example.py computes for you from these values:

• words in the current turn
• seconds since your last cue (None if there was none)
• words since your last cue
• cues so far in this turn

Recommended rules

The raw scores aren't a finished policy. These rules worked well in testing, and example.py applies the first three:

• Cue bar: use 0.09 at the end of a clause and 0.018 mid-sentence. Scores are low overall, especially mid-sentence, because people rarely respond there. A user who stops mid-sentence still likes to hear that you're listening.
• Echo questions: only ask back after a word that leaves the sentence hanging ("about", "like", "if"), and only with an echo score of 0.7 or more.
• Safety with mood: no laugh, praise or wow when bad is above 0.3. Only laugh when funny is at least 0.5 and sympathy is at most 0.25.
• Spacing: leave at least 6 seconds between cues and at most about 4 cues a minute. Without this, a talkative pause pattern gets several cues in a few seconds.
• Mid-sentence wording: keep to "mhm", "yeah" and "uh-huh". "Okay" or "alright" sound as if the thought were finished.

Input tips

Send only the current turn, not the whole conversation. On messy speech recognition text, earlier turns pulled echo scores down sharply. Use lowercase words without punctuation, newest 30 words. The clean() function in example.py does this.

Training

• Encoder: jhu-clsp/ettin-encoder-32m (MIT).
• Real conversations: about 295,000 pauses, 63% from Open Yap 1K and 37% from Switchboard (MS-State transcripts). These teach when people respond and in which words.
• Teacher labels: about 7,100 pauses and 18,400 written sentences, labeled for fitting reactions, mood and echoes from the visible words only. The labels came from a local Qwen model and OpenAI GPT-5.4 models. OpenAI models never saw Open Yap text.

This repository holds model weights only, no training data.

Results

Check	Result
Gate AUC, held-out corpus pauses	0.733
Gate AUC, short turns, current-turn text	0.714 (Switchboard 0.76, Open Yap 0.61)
Mood accuracy, written dev set	86%
New sad sentences	12 of 12 read as bad, 0 laughs
Echoes at 0.7, hand-written set	14 of 14 good fired, 2 of 12 bad fired
Same echoes with synthetic transcript errors	99% good fired, 30% bad fired

The hand-written sets are small and some were used during tuning, so treat these numbers as rough.

Limits

• English only.
• It can't detect humor from text, so it rarely laughs at funny stories.
• It reads words only, so tone of voice and pitch are invisible to it.
• The gate is weak on its own. Expect some misplaced cues.
• With current-turn input it forgets earlier turns, so sad news from a previous turn won't stop a cheerful cue.
• Echo questions get less reliable on messy transcripts, and "Like what?" can fire after a filler "like".
• It has been tested in simulated talks and by one person, not in a user study.

Credits

Open Yap 1K by The Agentic Data Company. Encoder by JHU CLSP. Switchboard transcripts by Mississippi State University.