Instructions to use RetentionLabs/personaplex-7b-extended with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use RetentionLabs/personaplex-7b-extended with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForSeq2SeqLM model = AutoModelForSeq2SeqLM.from_pretrained("RetentionLabs/personaplex-7b-extended", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Configuration Parsing Warning:In tokenizer_config.json: "tokenizer_config.chat_template" does not match any of the allowed types
PersonaPlex 7B Extended
How to run PersonaPlex 7B (nvidia/personaplex-7b-v1) on duplex conversations much longer than the 3,000 frames
(4 minutes at 12.5 frames per second) it was trained on, and what we found trying to extend it to 40,960 frames
(about 55 minutes).
Use rebase_inference.py with the original weights. It keeps the persona prefix and the
latest frames, at most 3,000 in all, and reads them at positions from 0, the length and positions PersonaPlex was
trained on. Generation stays fluent at any length, with no training.
The
model.safetensorsin this repository is a research checkpoint (RoPE change with trained query and key terms, see Approach 1). Its free generation is worse than the original model's. It is kept for reference only.rebase_inference.pydoes not use it; it loadsnvidia/personaplex-7b-v1.
Summary
Three approaches were measured. All start from the original PersonaPlex weights.
| Approach | Training | Generation past 3,000 frames | Reads past 3,000 frames | Verdict |
|---|---|---|---|---|
| 1. RoPE change + trained query/key terms (weights here) | yes | breaks | yes | does not work |
2. Rebased 3,000 frame window (rebase_inference.py) |
no | fluent | no | recommended |
| 3. Self-Extend with length temperature | no | fluent up to 32,000 frames checked | yes, with no measured gain | research result |
Why the original model breaks: PersonaPlex was trained with sliding_window 3,000. With the window widened to
40,960 it sees query to key distances it never saw, and generation breaks from about 4,000 frames on (silence or
unrelated words). With its own 3,000 window it never sees such distances, but it then drops the persona prefix,
the first frames of the conversation, and breaks as well (word CE 4.6 at 4k to 8k frames, table below). Keeping the
prefix is what makes approach 2 work.
Results
Teacher forced cross entropy
Setup: English self-play conversations of PersonaPlex with itself that no training saw. Two rows are built by
joining the 64 evaluation conversations of one speaker side, each cut at a random point 0 to 300 frames before its
end, up to 32,768 frames. Rows are laid out as generate reads a prompt. bfloat16, measured through our research
harness, which computes PersonaPlex's backbone exactly (the RoPE and attention changes applied to it). Numbers are means over the two
rows, in nats, over the frames whose text id is a word piece (padding frames excluded). Columns are spans of frame
positions.
| Model | 0-512 | 512-1k | 1k-2k | 2k-3k | 3k-4k | 4k-8k | 8k-16k | 16k-32k |
|---|---|---|---|---|---|---|---|---|
| Original, window 40,960 | 1.18 | 1.85 | 1.27 | 1.24 | 1.36 | 13.95 | 13.61 | 12.54 |
| Original, window 3,000 (prefix dropped) | 1.38 | 4.60 | 8.88 | 9.43 | ||||
| 1. RoPE change + query/key terms (rank 32, 236 steps) | 1.20 | 1.85 | 1.33 | 1.28 | 1.49 | 4.00 | 5.16 | 6.86 |
| 2. Rebased 3,000 window | 1.18 | 1.84 | 1.27 | 1.24 | 1.34 | 1.30 | 1.38 | 1.51 |
| 3. Self-Extend 1024/24 + length temperature | 1.33 | 1.31 | 1.42 | 1.64 |
Below 3,000 frames all rows read the same frames as the original and agree within 0.06. Empty cells were not measured separately.
Generation
Setup: the joined evaluation rows were given to generate up to the context length, then the model generated 375
frames (30 seconds) of its own text and speech while hearing the recorded other side, sampled with temperature 0.8
and top-k 250. "KL to rebased" is the text KL from the model's distributions to approach 2's on the 30 generated
seconds, row 0 / row 1.
| Model | Context (frames) | What the generated text looks like | KL to rebased |
|---|---|---|---|
| Original, window 40,960 | 4,300 | broken ("a a is easy, a a a is a a is") | 7.21 / 2.03 |
| Original, window 40,960 | 8,300 | silent | |
| Original, window 40,960 | 16,400 | unrelated words | 5.12 / 2.66 |
| 1. RoPE change + query/key terms (236 steps) | 4,300 | broken | 4.00 / 1.45 |
| 1. RoPE change + query/key terms (236 steps) | 16,400 | nearly silent | 7.86 / 4.75 |
| 3. Self-Extend 1024/8 + length temperature | 4,300 | fluent, on topic | 0.08 / 0.13 |
| 3. Self-Extend 1024/8 + length temperature | 16,400 | fluent, some short repeats | 0.08 / 0.19 |
| 3. Self-Extend 1024/24 + length temperature | 32,000 | fluent, on topic | 0.17 / 0.17 |
Approach 2 is the reference of this column. Its generations are fluent at every length checked.
Does the model use context past 3,000 frames?
Not on this data. Approach 3 reads the whole conversation, approach 2 only the prefix and the latest 3,000 frames. Their teacher forced word CE is the same up to 16,000 frames (difference 0.01 to 0.04), and approach 3 is 0.13 worse from 16,000 to 32,768. The evaluation rows join unrelated conversations, so nothing far back helps predict what comes next. Data whose later turns depend on early ones (a name, a fact) would be needed to train or measure long range use.
Approach 2: rebased 3,000 frame window
rebase_inference.py generates the assistant side of a conversation of any length:
- The persona prefix (text prompt, optional voice prompt) is laid out by
PersonaPlexProcessor. - The first
chunk(500) frames are generated as usual. - Before each further chunk, the model is given the persona prefix followed by the latest
window - chunk - prefixframes it generated and heard, at positions from 0, and generates the next chunk. The model always reads between 2,500 and 3,000 frames.
The cost is one prompt pass of up to 2,500 frames every 500 generated frames (40 seconds), on top of normal generation. Nothing is trained and the original weights and configuration are used as they are.
Check: with the other side of an evaluation conversation, 5,000 frames (6 minutes 40 seconds) long, as --user-audio,
the script generated fluent, on topic speech and text throughout, the last 2,500 frames under rebasing. It took
4 minutes 10 seconds on one B300 in bfloat16, faster than real time.
python rebase_inference.py \
--user-audio other_side.wav \
--persona "You are a wise and friendly teacher. Answer questions or provide advice in a clear and engaging way." \
--output conversation.wav
--user-audio: the other side, any length. It is encoded with the Mimi codec 60 seconds at a time.--voice-prompt: optional voice sample, as in PersonaPlex. Voice prompts are invoices.tgzofnvidia/personaplex-7b-v1.--window(3,000) and--chunk(500): frames read at most, and frames generated between two rebases.- Output: a stereo WAV, the model on the left and the other side on the right, and the generated text on stdout.
It needs a transformers build with PersonaPlex, for example
latentforge/transformers-moshi on branch feat/personaplex,
which loads nvidia/personaplex-7b-v1 directly. The processor is read from nvidia/personaplex-7b-v1 as well. The script runs
offline on a recorded other side; a live streaming server would apply the same rebase every chunk.
Approach 3: Self-Extend with length temperature
Self-Extend changes only how attention scores are computed, in the 32 backbone
layers. A key closer than W (1,024) frames is read at its own position. A key further away is read with positions
merged in groups of G: the query at $\lfloor i/G \rfloor + W - \lfloor W/G \rfloor$, the key at
$\lfloor j/G \rfloor$, so every distance stays under 3,000. The two scores are chosen by distance and go through one
softmax, so the whole conversation is read. The length temperature multiplies the scores of a query past position
3,000 by $\log i / \log 3000$, as in Qwen's log-n attention. Weights and the depth decoder are unchanged.
It keeps generation fluent up to 32,000 frames, close to approach 2 (tables above), but it shows no gain from the far context on this data, it keeps the whole conversation in the cache, and it needs a custom attention kernel. Its code is not included here.
Approach 1: RoPE change and trained query and key terms (the weights in this repository)
The checkpoint here changes two things from the original:
- RoPE frequencies. The 18 slowest of the 64 rotary pairs (46 to 63) move halfway, in log scale, toward period
1e6 (
bound), set aslongropeparameters inconfig.json. - Query and key weights. Every layer's
q_projandk_projcarries a trained rank 16 term folded into the weight. It was trained for 100 steps, everything else frozen, to give the original model's outputs on conversations up to 3,000 frames underbound.
sliding_window and max_position_embeddings are 40,960. In free generation it loops even inside 3,000 frames
("stay consistent, stay consistent, stay consistent") and its sentences break past 8,000 frames.
Later variants of the same method, not uploaded, did not fix generation either:
- Moving only the 9 pairs whose wavelength is at least 16,384 (
keep-16384) to 1e6 fixed the short range loops. - Training the long rows toward approach 2 as a teacher (KL on chunks of 500 frames) with rank 32 for 236 steps brought the teacher forced word CE past 4,000 frames from 12.5 to 14 down to 4.0 to 6.9 (table above), but generation still broke from 4,300 frames on.
- The training data ran out at 236 steps (every self-play conversation is read once).
The gap between teacher forced numbers and generation was the main lesson: lower cross entropy on recorded conversations did not keep the model stable when it reads its own speech.
Limitations
- English only, self-play only. Training and evaluation used English conversations of PersonaPlex with itself, no human conversations.
- Small evaluation. The long range numbers come from two joined rows.
- No long range use shown. Approach 2 forgets everything older than about 2,500 frames apart from the persona prefix. No approach was shown to use older context on this data.
- Offline script.
rebase_inference.pyreads a recorded other side; it is not a real time server. - Precision. Measured in bfloat16 only.
License
This repository is a derivative of nvidia/personaplex-7b-v1 and is governed by the NVIDIA Open Model License Agreement. PersonaPlex builds on Moshi (CC-BY-4.0). The training and evaluation conversations were generated by PersonaPlex itself.
- Downloads last month
- 9