Configuration Parsing Warning:In tokenizer_config.json: "tokenizer_config.chat_template" does not match any of the allowed types

PersonaPlex 7B Extended

How to run PersonaPlex 7B (nvidia/personaplex-7b-v1) on duplex conversations much longer than the 3,000 frames (4 minutes at 12.5 frames per second) it was trained on, and what we found trying to extend it to 40,960 frames (about 55 minutes).

Use rebase_inference.py with the original weights. It keeps the persona prefix and the latest frames, at most 3,000 in all, and reads them at positions from 0, the length and positions PersonaPlex was trained on. Generation stays fluent at any length, with no training.

The model.safetensors in this repository is a research checkpoint (RoPE change with trained query and key terms, see Approach 1). Its free generation is worse than the original model's. It is kept for reference only. rebase_inference.py does not use it; it loads nvidia/personaplex-7b-v1.

Summary

Three approaches were measured. All start from the original PersonaPlex weights.

Approach Training Generation past 3,000 frames Reads past 3,000 frames Verdict
1. RoPE change + trained query/key terms (weights here) yes breaks yes does not work
2. Rebased 3,000 frame window (rebase_inference.py) no fluent no recommended
3. Self-Extend with length temperature no fluent up to 32,000 frames checked yes, with no measured gain research result

Why the original model breaks: PersonaPlex was trained with sliding_window 3,000. With the window widened to 40,960 it sees query to key distances it never saw, and generation breaks from about 4,000 frames on (silence or unrelated words). With its own 3,000 window it never sees such distances, but it then drops the persona prefix, the first frames of the conversation, and breaks as well (word CE 4.6 at 4k to 8k frames, table below). Keeping the prefix is what makes approach 2 work.

Results

Teacher forced cross entropy

Setup: English self-play conversations of PersonaPlex with itself that no training saw. Two rows are built by joining the 64 evaluation conversations of one speaker side, each cut at a random point 0 to 300 frames before its end, up to 32,768 frames. Rows are laid out as generate reads a prompt. bfloat16, measured through our research harness, which computes PersonaPlex's backbone exactly (the RoPE and attention changes applied to it). Numbers are means over the two rows, in nats, over the frames whose text id is a word piece (padding frames excluded). Columns are spans of frame positions.

Model 0-512 512-1k 1k-2k 2k-3k 3k-4k 4k-8k 8k-16k 16k-32k
Original, window 40,960 1.18 1.85 1.27 1.24 1.36 13.95 13.61 12.54
Original, window 3,000 (prefix dropped) 1.38 4.60 8.88 9.43
1. RoPE change + query/key terms (rank 32, 236 steps) 1.20 1.85 1.33 1.28 1.49 4.00 5.16 6.86
2. Rebased 3,000 window 1.18 1.84 1.27 1.24 1.34 1.30 1.38 1.51
3. Self-Extend 1024/24 + length temperature 1.33 1.31 1.42 1.64

Below 3,000 frames all rows read the same frames as the original and agree within 0.06. Empty cells were not measured separately.

Generation

Setup: the joined evaluation rows were given to generate up to the context length, then the model generated 375 frames (30 seconds) of its own text and speech while hearing the recorded other side, sampled with temperature 0.8 and top-k 250. "KL to rebased" is the text KL from the model's distributions to approach 2's on the 30 generated seconds, row 0 / row 1.

Model Context (frames) What the generated text looks like KL to rebased
Original, window 40,960 4,300 broken ("a a is easy, a a a is a a is") 7.21 / 2.03
Original, window 40,960 8,300 silent
Original, window 40,960 16,400 unrelated words 5.12 / 2.66
1. RoPE change + query/key terms (236 steps) 4,300 broken 4.00 / 1.45
1. RoPE change + query/key terms (236 steps) 16,400 nearly silent 7.86 / 4.75
3. Self-Extend 1024/8 + length temperature 4,300 fluent, on topic 0.08 / 0.13
3. Self-Extend 1024/8 + length temperature 16,400 fluent, some short repeats 0.08 / 0.19
3. Self-Extend 1024/24 + length temperature 32,000 fluent, on topic 0.17 / 0.17

Approach 2 is the reference of this column. Its generations are fluent at every length checked.

Does the model use context past 3,000 frames?

Not on this data. Approach 3 reads the whole conversation, approach 2 only the prefix and the latest 3,000 frames. Their teacher forced word CE is the same up to 16,000 frames (difference 0.01 to 0.04), and approach 3 is 0.13 worse from 16,000 to 32,768. The evaluation rows join unrelated conversations, so nothing far back helps predict what comes next. Data whose later turns depend on early ones (a name, a fact) would be needed to train or measure long range use.

Approach 2: rebased 3,000 frame window

rebase_inference.py generates the assistant side of a conversation of any length:

  1. The persona prefix (text prompt, optional voice prompt) is laid out by PersonaPlexProcessor.
  2. The first chunk (500) frames are generated as usual.
  3. Before each further chunk, the model is given the persona prefix followed by the latest window - chunk - prefix frames it generated and heard, at positions from 0, and generates the next chunk. The model always reads between 2,500 and 3,000 frames.

The cost is one prompt pass of up to 2,500 frames every 500 generated frames (40 seconds), on top of normal generation. Nothing is trained and the original weights and configuration are used as they are.

Check: with the other side of an evaluation conversation, 5,000 frames (6 minutes 40 seconds) long, as --user-audio, the script generated fluent, on topic speech and text throughout, the last 2,500 frames under rebasing. It took 4 minutes 10 seconds on one B300 in bfloat16, faster than real time.

python rebase_inference.py \
  --user-audio other_side.wav \
  --persona "You are a wise and friendly teacher. Answer questions or provide advice in a clear and engaging way." \
  --output conversation.wav
  • --user-audio: the other side, any length. It is encoded with the Mimi codec 60 seconds at a time.
  • --voice-prompt: optional voice sample, as in PersonaPlex. Voice prompts are in voices.tgz of nvidia/personaplex-7b-v1.
  • --window (3,000) and --chunk (500): frames read at most, and frames generated between two rebases.
  • Output: a stereo WAV, the model on the left and the other side on the right, and the generated text on stdout.

It needs a transformers build with PersonaPlex, for example latentforge/transformers-moshi on branch feat/personaplex, which loads nvidia/personaplex-7b-v1 directly. The processor is read from nvidia/personaplex-7b-v1 as well. The script runs offline on a recorded other side; a live streaming server would apply the same rebase every chunk.

Approach 3: Self-Extend with length temperature

Self-Extend changes only how attention scores are computed, in the 32 backbone layers. A key closer than W (1,024) frames is read at its own position. A key further away is read with positions merged in groups of G: the query at $\lfloor i/G \rfloor + W - \lfloor W/G \rfloor$, the key at $\lfloor j/G \rfloor$, so every distance stays under 3,000. The two scores are chosen by distance and go through one softmax, so the whole conversation is read. The length temperature multiplies the scores of a query past position 3,000 by $\log i / \log 3000$, as in Qwen's log-n attention. Weights and the depth decoder are unchanged.

It keeps generation fluent up to 32,000 frames, close to approach 2 (tables above), but it shows no gain from the far context on this data, it keeps the whole conversation in the cache, and it needs a custom attention kernel. Its code is not included here.

Approach 1: RoPE change and trained query and key terms (the weights in this repository)

The checkpoint here changes two things from the original:

  1. RoPE frequencies. The 18 slowest of the 64 rotary pairs (46 to 63) move halfway, in log scale, toward period 1e6 (bound), set as longrope parameters in config.json.
  2. Query and key weights. Every layer's q_proj and k_proj carries a trained rank 16 term folded into the weight. It was trained for 100 steps, everything else frozen, to give the original model's outputs on conversations up to 3,000 frames under bound.

sliding_window and max_position_embeddings are 40,960. In free generation it loops even inside 3,000 frames ("stay consistent, stay consistent, stay consistent") and its sentences break past 8,000 frames.

Later variants of the same method, not uploaded, did not fix generation either:

  • Moving only the 9 pairs whose wavelength is at least 16,384 (keep-16384) to 1e6 fixed the short range loops.
  • Training the long rows toward approach 2 as a teacher (KL on chunks of 500 frames) with rank 32 for 236 steps brought the teacher forced word CE past 4,000 frames from 12.5 to 14 down to 4.0 to 6.9 (table above), but generation still broke from 4,300 frames on.
  • The training data ran out at 236 steps (every self-play conversation is read once).

The gap between teacher forced numbers and generation was the main lesson: lower cross entropy on recorded conversations did not keep the model stable when it reads its own speech.

Limitations

  • English only, self-play only. Training and evaluation used English conversations of PersonaPlex with itself, no human conversations.
  • Small evaluation. The long range numbers come from two joined rows.
  • No long range use shown. Approach 2 forgets everything older than about 2,500 frames apart from the persona prefix. No approach was shown to use older context on this data.
  • Offline script. rebase_inference.py reads a recorded other side; it is not a real time server.
  • Precision. Measured in bfloat16 only.

License

This repository is a derivative of nvidia/personaplex-7b-v1 and is governed by the NVIDIA Open Model License Agreement. PersonaPlex builds on Moshi (CC-BY-4.0). The training and evaluation conversations were generated by PersonaPlex itself.

Downloads last month
9
Safetensors
Model size
8B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for RetentionLabs/personaplex-7b-extended

Finetuned
(52)
this model

Collection including RetentionLabs/personaplex-7b-extended

Paper for RetentionLabs/personaplex-7b-extended