sadmN/moshi-base7b-rag

PersonaPlex 7B RAG — base fine-tune (step 1500)

A full 32-layer PersonaPlex 7B (8.37 B params) fine-tuned with RAG, served as a full-duplex streaming voice assistant whose factual answers come from a retrieved passage injected into the pause — zero extra frames, zero added silence.

• base: nvidia/personaplex-7b-v1 (gated, 32/32 layers, dim 4096)
• fine-tune: LoRA (r=64, alpha=128) on 128 matrices, 1500 steps, the RAG recipe (ref_mode=text, ref_pos=pause, K=16, lookup_weight=6)
• text vocabulary gains: <ref_start> 32000, <ref_end> 32001, <lookup> 32002 (pad/initial row moved to 32003)
• reference: the passage's own tokens are written as a 16-token text card into the pause the model already produces before its answer, so retrieval adds zero frames and zero silence — the audio is the model's own pause.
• card selection: contiguous 16-token window of the retrieved passage, 4-gram Jaccard against the question with an anchor on the question's rarest content word (rag_ref_select.select_window_ids).

IMPORTANT — streaming server required

This checkpoint is mute on the state-machine server (server_student_rag.py) that ships with the S28 student: that server forces the model's own audio stream to a constant SILENCE_TOKENS vector while the user speaks, a self-context the fine-tune never saw in training, so it never emits <lookup> and waits forever. This is fixed by the included single-path streaming server (server_rag_stream.py): the LM steps every frame with only the user's audio, generates its own audio + text freely (full-duplex, like the original server.py), and injects the card when the model emits <lookup>. No state machine, no forced turns, no cache reset — long conversations work because the KV cache persists for the whole session.

Live-tested on this checkpoint: it speaks and grounds correctly. Example with q_dune.wav:

"That detail isn't in the material I have available. Let me see if I can find that... The novel Dune was written by Frank Herbert."

Files

path	what
ckpt/rag_base7b/step_1500/student.pt	the base 7B fine-tune (16 GB, bf16, merged LoRA)
tokenizer.model	SentencePiece text tokenizer (32003 ids)
rag_index_dense/	local fallback index: 4998 passages + MiniLM embeddings
moshi_base7b_server.zip	server code + scripts + index + tokenizer (run anywhere)
setup_env.sh	creates the Python env (torch 2.5.1/cu121 + deps)
download_assets.sh	pulls this repo + the gated base weights
serve_base7b.sh	starts the streaming server (web search + local fallback)

Quick start (new machine)

unzip moshi_base7b_server.zip && cd moshi_base7b_server
bash setup_env.sh                                # python env -> /workspace/menv
HF_TOKEN=hf_... bash download_assets.sh
HF_TOKEN=hf_... bash serve_base7b.sh
# open http://<host>:8998

serve_base7b.sh reads HF_TOKEN only to resolve the gated base weights; the built-in Tavily key covers web search (override with TAVILY_API_KEY).

Hardware

One 80 GB GPU is comfortable (21.3 GB used with the local index loaded; RETRIEVER=web drops the sentence-transformer + cross-encoder). The frame loop is 80 ms (25.4 ms LM step), so any GPU that keeps a 12.5 Hz Moshi step under ~70 ms works.

Test

The server only steps the LM when client audio arrives — a client that stops sending starves it. Use the included ws_client_test2.py which streams 80 ms silence frames during gaps:

WS_URL='http://127.0.0.1:8998/api/chat?voice_prompt=NATF0.pt' \
  /workspace/menv/bin/python ws_client_test2.py 1 q_dune.wav 0 2.0 12

Limits

• The card budget is 16 tokens (the pause is ~17 frames), so a very long passage is truncated to the 16-token query-matched window. A 2× text-bag compressor for long context is in ref_compressor.py (not yet trained).
• The <lookup> emit rate is not 100% — the fine-tune's trigger pathway is still weak from the high learning rate (lr=1e-4). A retrain with lr=2e-5 + the streaming-consistent data is the next step.
• If no passage is retrieved the model answers from general knowledge.

Licence

Derived from nvidia/personaplex-7b-v1; use is subject to that model's licence. The retrieval index is built from public encyclopedic text.