TYGER

Speech recognition and duplex turn control in one model.

TYGER is NVIDIA's Nemotron speech recognizer with a GRU on top. For every 80 ms of audio, the GRU tells a voice agent whether to answer (the user has finished), stop (the user is cutting in) or say "mm-hm" (the user goes on talking). It hears both sides of the call: the user's mic and the agent's own voice. Both parts load from one model.safetensors. It makes no speech itself: your app brings the LLM and the voice.

Use it

tyger-s2s is our open-source voice agent built on this model (Apache-2.0). You talk to it in the browser, as on a phone call:

• this model is its ears: it transcribes you, and knows when you've finished, when you cut in, and when to say "mm-hm";
• Gemma 4 E4B writes the replies, VoxCPM2 speaks them in a cloned voice, and a hosted model can take the hard questions;
• it runs on one NVIDIA GPU with 32GB VRAM, for up to 4 conversations at once, and answers ~0.7 s after you stop speaking;
• it comes as a Docker image, and can be the voice backend of your own web app.

To run the model in your own code, start from tyger-s2s's runtime (app/plugins/ears/tyger/): it checks the files against the SHA-256s in config.json, makes sure every tensor loads, and serves several conversations on one GPU. Transformers' AutoModel and pipeline can't load this model.

The model

Speech recognition	NVIDIA Nemotron 3.5 ASR Streaming 0.6B, unchanged, with its decoder and tokenizer
Turn-taker	a causal GRU: 3 layers, width 768
Parameters	652M: 638M Nemotron + 14M GRU, in FP32 (2.61 GB)
Input	two aligned mono 16 kHz streams: the user's mic and the agent's playback
Features	Nemotron's layer 15 (1,024 dims), one frame per 80 ms, no lookahead
Streaming	320 ms hops; Nemotron re-encodes an 8 s window per stream at each hop
Output, per frame	the turn state (listen, hold, speak, backchannel) and hazard, barge-in, backchannel and endpoint scores
State per conversation	9 KiB of GRU state, besides the audio windows
Language	English: the GRU was trained and tested on English only

This repo holds weights and configs, no code. The weights have two namespaces: nemotron.* (its configs, processor and tokenizer are in nemotron/) and turn_taker.*. config.json describes both, with the default event policy. Nemotron comes inside: no separate download, and it runs offline. Keep both parts in FP32 with TF32 off: half precision and quantization aren't tested.

Events

An event fires when its score stays over the threshold for 2 frames in a row:

Event	When	The app	Default threshold
Endpoint	the user has finished	answers	0.91
Barge-in	the user cuts in while the agent talks	stops the reply at once	0.50
Backchannel	a "mm-hm" fits while the user talks	plays one; the user keeps the turn	0.40

Guards: 6 warm-up frames, at least 350 ms of speech above -50 dB, 1.2 s between backchannels, and an endpoint after 1.2 s of silence even if the score stays low. The state input is LISTEN while the agent listens and SPEAK while it talks, never the model's own guess. The barge-in and backchannel defaults are looser than the evaluation's (0.93 and 0.80) and have no measurements of their own: tune them in your app (tyger-s2s's page has sliders for that).

Training

The GRU, its input layers and its heads were trained from scratch on Open Yap 1K v1.0 (1,395 training, 107 development and 100 test conversations), with Nemotron frozen. The labels come automatically from when each side speaks, not from human annotators: the turn state, an upcoming speaking onset (hazard), when to yield to an interruption, when a backchannel fits, and whether a pause ends the turn. One training seed.

Evaluation

Offline scores for the released weights, at endpoint 0.91, barge-in 0.93 and backchannel 0.80, set on development data and frozen before test scoring. evaluation/README.md has the definitions and caveats, and metrics.json every number.

Endpoints, with the 1.2 s silence fallback:

Set	Turns	Latency, median / mean	Early cut-ins	No endpoint within 1.2 s	Pauses with a false endpoint
Open Yap 1K test	1,000	252.5 / 465.0 ms	29.90%	7.40%	415 of 1,000
Full-Duplex-Bench v1.0, CANDOR	119	307.5 / 383.0 ms	25.21%	5.04%	123 of 277
Internal, not public	242	252.5 / 286.5 ms	12.40%	0.00%	186 of 539

Latency runs from the end of the user's turn, and counts only endpoints that fired: read it with the early cut-ins (an endpoint before the turn really ended) and the misses. The sets differ, so don't rank them against each other. With the fallback off, Open Yap's mean latency is 436.5 ms, and 16.0% of turns get no endpoint within 1.2 s.

Turn state, barge-in and backchannel (Open Yap 1K test):

Metric	Value
Turn-state macro-F1	0.9436
Endpoint ROC-AUC / AP	0.9839 / 0.8149
Barge-in ROC-AUC / AP	0.9797 / 0.6657
Backchannel ROC-AUC / AP	0.8976 / 0.3004
Barge-ins caught within 640 ms / median delay	25.32% / 400 ms
False barge-ins per hour of clean speech	0.00 (none in 4.72 h; not a guarantee)
Backchannels caught / median delay	14.86% / -160 ms (before the labeled moment)
Backchannels fired per hour of listening	135.08

Speed: the GRU takes 0.805 ms per frame (median; p90 0.823 ms) on an RTX 4090 at batch 1, with 108.76 MiB of peak GPU memory. That leaves out Nemotron and waiting for audio. In tyger-s2s, the whole model takes ~4 GB of GPU memory for 4 conversations.

Limits:

• Offline diagnostics only: no live voice-agent benchmark, and no new ASR error rates (NVIDIA's card has its own).
• One training seed, and the evaluation sets also guided development. The internal set can't be reproduced.
• The Full-Duplex-Bench rows are adapted v1.0 CANDOR replays with a silent agent channel, not official full-agent scores. FDB-v3 isn't evaluated.

License

TYGER's weights and files are OpenMDW-1.1. NVIDIA's original card, figures and license are unchanged in upstream/ and licenses/, and manifest.json lists every file's size and SHA-256. See LICENSE.md and NOTICE.md.

Links

• Voice agent: nineninesix-ai/tyger-s2s
• Website: nineninesix.ai
• Discord: NineNineSix community
• Training data: Open Yap 1K
• ASR: NVIDIA Nemotron

Acknowledgments

We thank NVIDIA for Nemotron, The Agentic Data Company for Open Yap 1K, and the authors of Full-Duplex-Bench for its evaluation probes. Related work, not bundled or reproduced here: Full-Duplex-Bench-v3, Moshi, SALM-Duplex, Freeze-Omni, PersonaPlex and Audio Flamingo 3.

Citation

If you use TYGER, please cite the model and its training data:

@misc{tyger_2026,
  author    = {{NineNineSix}},
  title     = {{TYGER}: Speech Recognition and Duplex Turn Control},
  year      = {2026},
  publisher = {NineNineSix},
  url       = {https://huggingface.co/nineninesix/tyger}
}

@misc{openyap1k,
  title     = {Open Yap 1K: Channel-Separated English Natural Two-Speaker Conversations},
  author    = {{The Agentic Data Company}},
  year      = {2026},
  version   = {1.0},
  publisher = {The Agentic Data Company},
  url       = {https://theagenticdatacompany.com/open-yap-1k}
}

References

BibTeX for the ASR component and related work

@misc{nvidia_nemotron_3_5_asr,
  author = {{NVIDIA}},
  title  = {Nemotron 3.5 ASR Streaming 0.6B},
  url    = {https://huggingface.co/nvidia/nemotron-3.5-asr-streaming-0.6b},
  note   = {Bundled revision ea30d66debe3740a08b573244286791d423d6b3e}
}

@misc{lin2025fullduplexbench,
  title = {Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilities},
  author = {Lin, Guan-Ting and Lian, Jiachen and Li, Tingle and Wang, Qirui and Anumanchipalli, Gopala and Liu, Alexander H. and Lee, Hung-yi},
  year = {2025}, eprint = {2503.04721}, archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.04721}
}

@misc{lin2026fullduplexbenchv3,
  title = {Full-Duplex-Bench-v3: Benchmarking Tool Use for Full-Duplex Voice Agents Under Real-World Disfluency},
  author = {Lin, Guan-Ting and Chen, Chen and Chen, Zhehuai and Lee, Hung-yi},
  year = {2026}, eprint = {2604.04847}, archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.04847}
}

@misc{defossez2024moshi,
  title = {Moshi: a speech-text foundation model for real-time dialogue},
  author = {D\'{e}fossez, Alexandre and Mazar\'{e}, Laurent and Orsini, Manu and Royer, Am\'{e}lie and P\'{e}rez, Patrick and J\'{e}gou, Herv\'{e} and Grave, Edouard and Zeghidour, Neil},
  year = {2024}, eprint = {2410.00037}, archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.00037}
}

@misc{hu2025salmduplex,
  title = {SALM-Duplex: Efficient and Direct Duplex Modeling for Speech-to-Speech Language Model},
  author = {Hu, Ke and Hosseini-Asl, Ehsan and Chen, Chen and Casanova, Edresson and Ghosh, Subhankar and {\.{Z}}elasko, Piotr and Chen, Zhehuai and Li, Jason and Balam, Jagadeesh and Ginsburg, Boris},
  year = {2025}, eprint = {2505.15670}, archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.15670}
}

@misc{wang2024freezeomni,
  title = {Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLM},
  author = {Wang, Xiong and Li, Yangze and Fu, Chaoyou and Shen, Yunhang and Xie, Lei and Li, Ke and Sun, Xing and Ma, Long},
  year = {2024}, eprint = {2411.00774}, archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.00774}
}

@misc{roy2026personaplex,
  title = {PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Models},
  author = {Roy, Rajarshi and Raiman, Jonathan and Lee, Sang-gil and Ene, Teodor-Dumitru and Kirby, Robert and Kim, Sungwon and Kim, Jaehyeon and Catanzaro, Bryan},
  year = {2026}, eprint = {2602.06053}, archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.06053}
}

@misc{goel2025audioflamingo3,
  title = {Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models},
  author = {Goel, Arushi and Ghosh, Sreyan and Kim, Jaehyeon and Kumar, Sonal and Kong, Zhifeng and Lee, Sang-gil and Yang, Chao-Han Huck and Duraiswami, Ramani and Manocha, Dinesh and Valle, Rafael and Catanzaro, Bryan},
  year = {2025}, eprint = {2507.08128}, archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.08128}
}