model card: released checkpoint and results
Browse files
README.md
CHANGED
|
@@ -7,7 +7,6 @@ pipeline_tag: audio-to-audio
|
|
| 7 |
language:
|
| 8 |
- en
|
| 9 |
library_name: contextspan
|
| 10 |
-
# TODO(jaeho): thumbnail: <absolute URL of assets/figures/architecture.png once the repo is public>
|
| 11 |
tags:
|
| 12 |
- full-duplex
|
| 13 |
- spoken-dialogue
|
|
@@ -23,13 +22,12 @@ tags:
|
|
| 23 |
# Context Spanning · DuetaSpan-7B
|
| 24 |
|
| 25 |
> [!Note]
|
| 26 |
-
> This repository holds the **DuetaSpan
|
| 27 |
> *Context Spanning: A Communication Framework for Full-Duplex Speech Models and External LLM Backends*
|
| 28 |
> (Go, Kim, Cha, Shin — Mindlogic, 2026). The inference and training code lives at
|
| 29 |
> [mindlogic-ai/ContextSpanning](https://github.com/mindlogic-ai/ContextSpanning); the weights load only
|
| 30 |
> through that package.
|
| 31 |
|
| 32 |
-
<!-- TODO(seonghyeon): replace XXXX.XXXXX with the arXiv id -->
|
| 33 |
[](https://arxiv.org/abs/XXXX.XXXXX) [](https://github.com/mindlogic-ai/ContextSpanning) [](https://huggingface.co/nvidia/personaplex-7b-v1) [](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/) [](https://github.com/mindlogic-ai/ContextSpanning/blob/main/LICENSE)
|
| 34 |
|
| 35 |
A full-duplex speech model that keeps listening and talking while it calls an external backend. When the
|
|
@@ -39,7 +37,7 @@ questions) returns one reference, and that reference is written into the model's
|
|
| 39 |
(time, weather, prices, distances) instead of a latent summary.
|
| 40 |
|
| 41 |
<p align="center">
|
| 42 |
-
<img src="assets/figures/architecture.png" alt="Context Spanning architecture: the
|
| 43 |
</p>
|
| 44 |
|
| 45 |
<!-- assets/figures/architecture.png is uploaded to this model repo together with the card -->
|
|
@@ -52,7 +50,7 @@ questions) returns one reference, and that reference is written into the model's
|
|
| 52 |
its cache and keeps stepping at the 80 ms frame clock.
|
| 53 |
- **Real tools.** The shipped bank is 60 tools whose result is something a voice assistant says — time,
|
| 54 |
weather, prices, web search, places and routes, SGD-seeded bookings — plus four MCP servers.
|
| 55 |
-
- **
|
| 56 |
the user's name and city.
|
| 57 |
|
| 58 |
## Files
|
|
@@ -60,7 +58,7 @@ questions) returns one reference, and that reference is written into the model's
|
|
| 60 |
| file | content |
|
| 61 |
|---|---|
|
| 62 |
| `context_spanning_7b.pt` | `{"model": state_dict}` in bf16, loadable with `main.py infer --checkpoint` or fetched automatically |
|
| 63 |
-
| `voices/{f0,f1,f2,f3,m0,m1,m2,m3}.pt` | voice prompts (`{"codes": LongTensor[8, P]}`, agent-voice Mimi codes; 8 s each) |
|
| 64 |
| `assets/figures/` | the paper's figures used by this card |
|
| 65 |
|
| 66 |
Mimi and the text tokenizer are taken from the PersonaPlex repository at load time; the `moshi` package is
|
|
@@ -73,56 +71,56 @@ vendored in the code repository (`contextspan/moshi/`).
|
|
| 73 |
- Parameters: 7B
|
| 74 |
- Audio codec: Mimi, 12.5 Hz, 8 codebooks per channel (user and agent)
|
| 75 |
- Frame clock: 80 ms per step
|
| 76 |
-
- Sequence convention: `<ret>` = 4, span open = 12, span close = 13
|
| 77 |
-
-
|
| 78 |
-
- Training objective: masked cross-entropy on the text row and on the audio codebooks as in PersonaPlex, with the span and prefix columns masked out
|
| 79 |
-
- Optimisation: 8-bit AdamW, context 3,000 frames, lr 2e-6 (temporal transformer) / 4e-6 (depth transformer), 32 dialogues per update, 4× NVIDIA RTX Pro 6000
|
| 80 |
-
- Released checkpoint: DuetaSpan v7, step 8000 (2026-09-11)
|
| 81 |
- Backends the numbers were measured with: router `google/gemma-4-26B-A4B-it` on vLLM, ASR Qwen3-ASR-1.7B
|
| 82 |
- Language: English
|
| 83 |
|
| 84 |
## Benchmark Results
|
| 85 |
|
| 86 |
-
|
| 87 |
-
|
| 88 |
-
|
| 89 |
-
|
| 90 |
-
|
| 91 |
-
|
| 92 |
-
|
| 93 |
-
|
| 94 |
-
|
| 95 |
-
|
| 96 |
-
|
| 97 |
-
|
| 98 |
-
|
| 99 |
-
|
| 100 |
-
|
| 101 |
-
|
| 102 |
-
|
| 103 |
-
|
| 104 |
-
|
| 105 |
-
|
| 106 |
-
|
| 107 |
-
|
| 108 |
-
|
| 109 |
-
|
| 110 |
-
|
| 111 |
-
|
| 112 |
-
|
| 113 |
-
|
| 114 |
-
|
| 115 |
-
|
| 116 |
-
|
| 117 |
-
|
| 118 |
-
|
| 119 |
-
|
| 120 |
-
|
| 121 |
-
|
| 122 |
-
|
| 123 |
-
|
| 124 |
-
|
| 125 |
-
|
|
|
|
|
|
|
|
|
|
| 126 |
|
| 127 |
| Model | LlamaQ ref. | LlamaQ resp. | WebQ ref. | WebQ resp. | TriviaQA ref. | TriviaQA resp. | HaluEval ref. | HaluEval resp. |
|
| 128 |
|---|---:|---:|---:|---:|---:|---:|---:|---:|
|
|
@@ -132,7 +130,7 @@ model's response. Our router is Gemma 4.
|
|
| 132 |
| MoshiRAG (GPT-4.1)† | 87.8 | 80.6 | 77.7 | 68.9 | 86.8 | 78.2 | 61.2 | 51.3 |
|
| 133 |
| MoshiRAG (Tavily)† | 84.6 | 78.2 | 73.5 | 66.1 | 84.9 | 77.5 | 54.3 | 47.0 |
|
| 134 |
| Vanilla Moshi† | | 62.3 | | 26.6 | | 22.8 | | 10.5 |
|
| 135 |
-
| **Ours (Gemma 4)** |
|
| 136 |
|
| 137 |
| Model | AddSub ref. | AddSub resp. | MultiArith ref. | MultiArith resp. | SinglEq ref. | SinglEq resp. | SVAMP ref. | SVAMP resp. | GSM8K ref. | GSM8K resp. |
|
| 138 |
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
|
|
@@ -141,12 +139,7 @@ model's response. Our router is Gemma 4.
|
|
| 141 |
| MoshiRAG (Gemma 3)† | 76.6 | 61.7 | 87.1 | 69.0 | 83.2 | 68.2 | 74.1 | 55.0 | 66.2 | 33.9 |
|
| 142 |
| MoshiRAG (GPT-4.1)† | 87.9 | 64.8 | 87.1 | 76.0 | 89.6 | 72.9 | 80.5 | 61.1 | 70.8 | 43.2 |
|
| 143 |
| Vanilla Moshi† | | 8.3 | | 9.8 | | 18.4 | | 9.7 | | 2.1 |
|
| 144 |
-
| **Ours (Gemma 4)** |
|
| 145 |
-
|
| 146 |
-
The step-8000 checkpoint in this repository was measured again after the paper runs on the `semi`
|
| 147 |
-
protocol (120 items per set): HaluEvalAudio resp 0.642 / ref 0.725 / P(resp | ref) 0.851, math word
|
| 148 |
-
problems (40) P(resp | ref) 1.00, `<ret>` rate 0.925. Protocols and scorers:
|
| 149 |
-
[`eval/README.md`](https://github.com/mindlogic-ai/ContextSpanning/blob/main/eval/README.md).
|
| 150 |
|
| 151 |
## Quickstart
|
| 152 |
|
|
@@ -185,7 +178,7 @@ the ASR. Servers, GPU layout, vLLM flags and environment variables:
|
|
| 185 |
- Context Spans occupy one text token per frame, so long references spend context; see the paper's
|
| 186 |
limitations section.
|
| 187 |
- Out of scope: text-only use, batch transcription, or any use of the voice prompts to imitate a real
|
| 188 |
-
person. The
|
| 189 |
their consent.
|
| 190 |
|
| 191 |
## License
|
|
@@ -197,12 +190,9 @@ the ASR. Servers, GPU layout, vLLM flags and environment variables:
|
|
| 197 |
project (volunteers who released their voice under CC0; no attribution or consent condition attaches).
|
| 198 |
- **Code:** [mindlogic-ai/ContextSpanning](https://github.com/mindlogic-ai/ContextSpanning) is MIT; the
|
| 199 |
vendored `contextspan/moshi/` carries Kyutai's own license files.
|
| 200 |
-
<!-- TODO(seonghyeon): decide gated: true / extra_gated_prompt before the repo goes public -->
|
| 201 |
|
| 202 |
## Citation
|
| 203 |
|
| 204 |
-
<!-- TODO(seonghyeon): fill in the arXiv id / venue once known -->
|
| 205 |
-
|
| 206 |
```bibtex
|
| 207 |
@article{go2026contextspanning,
|
| 208 |
title = {Context Spanning: A Communication Framework for Full-Duplex Speech Models and External LLM Backends},
|
|
|
|
| 7 |
language:
|
| 8 |
- en
|
| 9 |
library_name: contextspan
|
|
|
|
| 10 |
tags:
|
| 11 |
- full-duplex
|
| 12 |
- spoken-dialogue
|
|
|
|
| 22 |
# Context Spanning · DuetaSpan-7B
|
| 23 |
|
| 24 |
> [!Note]
|
| 25 |
+
> This repository holds the released **DuetaSpan-7B** weights described in
|
| 26 |
> *Context Spanning: A Communication Framework for Full-Duplex Speech Models and External LLM Backends*
|
| 27 |
> (Go, Kim, Cha, Shin — Mindlogic, 2026). The inference and training code lives at
|
| 28 |
> [mindlogic-ai/ContextSpanning](https://github.com/mindlogic-ai/ContextSpanning); the weights load only
|
| 29 |
> through that package.
|
| 30 |
|
|
|
|
| 31 |
[](https://arxiv.org/abs/XXXX.XXXXX) [](https://github.com/mindlogic-ai/ContextSpanning) [](https://huggingface.co/nvidia/personaplex-7b-v1) [](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/) [](https://github.com/mindlogic-ai/ContextSpanning/blob/main/LICENSE)
|
| 32 |
|
| 33 |
A full-duplex speech model that keeps listening and talking while it calls an external backend. When the
|
|
|
|
| 37 |
(time, weather, prices, distances) instead of a latent summary.
|
| 38 |
|
| 39 |
<p align="center">
|
| 40 |
+
<img src="assets/figures/architecture.png" alt="Context Spanning architecture: the frontend streams user audio, agent audio and agent text on one timeline; after the ret token the agent keeps talking (lead portion) while the backend passes the Streaming ASR user text and the Context DB to an LLM, whose tool result is written back as a Context Span (sine wave on user audio, silence on agent audio, sos … eos on agent text) before the body portion." width="100%">
|
| 41 |
</p>
|
| 42 |
|
| 43 |
<!-- assets/figures/architecture.png is uploaded to this model repo together with the card -->
|
|
|
|
| 50 |
its cache and keeps stepping at the 80 ms frame clock.
|
| 51 |
- **Real tools.** The shipped bank is 60 tools whose result is something a voice assistant says — time,
|
| 52 |
weather, prices, web search, places and routes, SGD-seeded bookings — plus four MCP servers.
|
| 53 |
+
- **Nine released voices** (`f0`-`f3` female, `m0`-`m3` male, plus `seonghyeon`, a team member's own recorded voice) and PersonaPlex-style persona prompts carrying
|
| 54 |
the user's name and city.
|
| 55 |
|
| 56 |
## Files
|
|
|
|
| 58 |
| file | content |
|
| 59 |
|---|---|
|
| 60 |
| `context_spanning_7b.pt` | `{"model": state_dict}` in bf16, loadable with `main.py infer --checkpoint` or fetched automatically |
|
| 61 |
+
| `voices/{f0,f1,f2,f3,m0,m1,m2,m3,seonghyeon}.pt` | voice prompts (`{"codes": LongTensor[8, P]}`, agent-voice Mimi codes; 8 s each) |
|
| 62 |
| `assets/figures/` | the paper's figures used by this card |
|
| 63 |
|
| 64 |
Mimi and the text tokenizer are taken from the PersonaPlex repository at load time; the `moshi` package is
|
|
|
|
| 71 |
- Parameters: 7B
|
| 72 |
- Audio codec: Mimi, 12.5 Hz, 8 codebooks per channel (user and agent)
|
| 73 |
- Frame clock: 80 ms per step
|
| 74 |
+
- Sequence convention: `<ret>` = 4, span open = 12, span close = 13, text pad = 3
|
| 75 |
+
- Released checkpoint: 2026-09-23
|
|
|
|
|
|
|
|
|
|
| 76 |
- Backends the numbers were measured with: router `google/gemma-4-26B-A4B-it` on vLLM, ASR Qwen3-ASR-1.7B
|
| 77 |
- Language: English
|
| 78 |
|
| 79 |
## Benchmark Results
|
| 80 |
|
| 81 |
+
Full runs of the released checkpoint with the released router (Gemma-4-26B-A4B on vLLM) and ASR (Qwen3-ASR-1.7B);
|
| 82 |
+
protocols and scorers are in [`benchmark/README.md`](https://github.com/mindlogic-ai/ContextSpanning/blob/main/benchmark/README.md). Rows marked † are reprinted from the
|
| 83 |
+
original benchmark papers (Full-Duplex-Bench, PersonaPlex) or from MoshiRAG. Arrows give the direction of better.
|
| 84 |
+
|
| 85 |
+
**Table 1. Full-Duplex-Bench v1 (727 clips).**
|
| 86 |
+
|
| 87 |
+
| Task | Metric | **Ours** | PersonaPlex† | Moshi† | Gemini Live 2.5† |
|
| 88 |
+
|---|---|---:|---:|---:|---:|
|
| 89 |
+
| Pause Handling | TOR, Candor ↓ | 0.824 | 0.431 | 0.980 | 0.310 |
|
| 90 |
+
| | TOR, synthetic ↓ | 0.861 | 0.358 | 0.985 | 0.255 |
|
| 91 |
+
| Backchannel | TOR ↓ | 0.782 | 0.273 | 1.000 | 0.091 |
|
| 92 |
+
| | Frequency (per s) ↑ | 0.150 | 0.042 | 0.001 | 0.012 |
|
| 93 |
+
| | JSD ↓ | 0.716 | 0.662 | 0.957 | 0.896 |
|
| 94 |
+
| Turn-Taking | TOR ↑ | 0.899 | 0.908 | 0.941 | 0.655 |
|
| 95 |
+
| | Latency (s) ↓ | 0.064 | 0.170 | 0.265 | 1.301 |
|
| 96 |
+
| User Interruption | TOR ↑ | 0.925 | 0.950 | 1.000 | 0.891 |
|
| 97 |
+
| | GPT-4o rating (0-5) ↑ | 3.924 | 4.290 | 0.765 | 3.376 |
|
| 98 |
+
| | Latency (s) ↓ | 0.598 | 0.240 | 0.257 | 1.183 |
|
| 99 |
+
|
| 100 |
+
TOR is the take-over rate, the fraction of clips in which the model takes the turn; JSD is the Jensen-Shannon
|
| 101 |
+
divergence from the human backchannel timing distribution.
|
| 102 |
+
|
| 103 |
+
**Table 2. Full-Duplex-Bench v3 (100 scenarios, 12 mock tools).** MoshiRAG is the released `kyutai/moshika-rag`
|
| 104 |
+
checkpoint run with its own inference code, its reference LLM replaced by the same tool router and prompt as Ours
|
| 105 |
+
(router LLM Gemma-3-27B-it).
|
| 106 |
+
|
| 107 |
+
| Task | Metric | **Ours** | MoshiRAG | GPT-Realtime† | Gemini Live 3.1† |
|
| 108 |
+
|---|---|---:|---:|---:|---:|
|
| 109 |
+
| Tool Use | Tool selection ↑ | 0.855 | 0.738 | 0.876 | 0.817 |
|
| 110 |
+
| | Argument accuracy ↑ | 0.567 | 0.440 | 0.680 | 0.588 |
|
| 111 |
+
| | Response quality ↑ | 0.411 | 0.255 | 0.792 | 0.718 |
|
| 112 |
+
| | Pass rate ↑ | 0.470 | 0.280 | 0.600 | 0.540 |
|
| 113 |
+
| Turn-Taking Dynamics | Take-turn rate (%) ↑ | 95.0 | 94.0 | 96.0 | 78.0 |
|
| 114 |
+
| | Latency, task completion (s) ↓ | 5.83 | 7.69 | 6.89 | 4.25 |
|
| 115 |
+
| | Interruption rate (%) ↓ | 73.7 | 58.5 | 13.5 | 19.2 |
|
| 116 |
+
| | Filler rate (%) ↓ | 96.0 | 92.3 | 16.9 | 31.7 |
|
| 117 |
+
|
| 118 |
+
Tool-use metrics are fractions of the 100 scenarios; rates are percentages; latency is the task-completion
|
| 119 |
+
latency in seconds.
|
| 120 |
+
|
| 121 |
+
**Table 3. Spoken QA and math reasoning (accuracy, %).** `ref.` = the router's reference answer is judged;
|
| 122 |
+
`resp.` = the model's spoken response is judged. LlamaQ 300, WebQ 1000, TriviaQA 1000, HaluEval 1000; math
|
| 123 |
+
3,822 (AddSub 395, MultiArith 600, SinglEq 508, SVAMP 1000, GSM8K 1319).
|
| 124 |
|
| 125 |
| Model | LlamaQ ref. | LlamaQ resp. | WebQ ref. | WebQ resp. | TriviaQA ref. | TriviaQA resp. | HaluEval ref. | HaluEval resp. |
|
| 126 |
|---|---:|---:|---:|---:|---:|---:|---:|---:|
|
|
|
|
| 130 |
| MoshiRAG (GPT-4.1)† | 87.8 | 80.6 | 77.7 | 68.9 | 86.8 | 78.2 | 61.2 | 51.3 |
|
| 131 |
| MoshiRAG (Tavily)† | 84.6 | 78.2 | 73.5 | 66.1 | 84.9 | 77.5 | 54.3 | 47.0 |
|
| 132 |
| Vanilla Moshi† | | 62.3 | | 26.6 | | 22.8 | | 10.5 |
|
| 133 |
+
| **Ours (Gemma 4)** | 85.3 | 81.7 | 67.7 | 59.1 | 71.5 | 68.9 | 39.6 | 33.7 |
|
| 134 |
|
| 135 |
| Model | AddSub ref. | AddSub resp. | MultiArith ref. | MultiArith resp. | SinglEq ref. | SinglEq resp. | SVAMP ref. | SVAMP resp. | GSM8K ref. | GSM8K resp. |
|
| 136 |
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
|
|
|
|
| 139 |
| MoshiRAG (Gemma 3)† | 76.6 | 61.7 | 87.1 | 69.0 | 83.2 | 68.2 | 74.1 | 55.0 | 66.2 | 33.9 |
|
| 140 |
| MoshiRAG (GPT-4.1)† | 87.9 | 64.8 | 87.1 | 76.0 | 89.6 | 72.9 | 80.5 | 61.1 | 70.8 | 43.2 |
|
| 141 |
| Vanilla Moshi† | | 8.3 | | 9.8 | | 18.4 | | 9.7 | | 2.1 |
|
| 142 |
+
| **Ours (Gemma 4)** | 78.5 | 76.2 | 94.5 | 89.7 | 82.1 | 77.6 | 86.0 | 81.1 | 70.4 | 62.5 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 143 |
|
| 144 |
## Quickstart
|
| 145 |
|
|
|
|
| 178 |
- Context Spans occupy one text token per frame, so long references spend context; see the paper's
|
| 179 |
limitations section.
|
| 180 |
- Out of scope: text-only use, batch transcription, or any use of the voice prompts to imitate a real
|
| 181 |
+
person. The `f`/`m` voices come from volunteers who released their recordings under CC0 and `seonghyeon` is a team member's own voice, released with consent; do not condition on a real person's voice without
|
| 182 |
their consent.
|
| 183 |
|
| 184 |
## License
|
|
|
|
| 190 |
project (volunteers who released their voice under CC0; no attribution or consent condition attaches).
|
| 191 |
- **Code:** [mindlogic-ai/ContextSpanning](https://github.com/mindlogic-ai/ContextSpanning) is MIT; the
|
| 192 |
vendored `contextspan/moshi/` carries Kyutai's own license files.
|
|
|
|
| 193 |
|
| 194 |
## Citation
|
| 195 |
|
|
|
|
|
|
|
| 196 |
```bibtex
|
| 197 |
@article{go2026contextspanning,
|
| 198 |
title = {Context Spanning: A Communication Framework for Full-Duplex Speech Models and External LLM Backends},
|