InnerVoice-Qwen3-4B (SFT)

InnerVoice is an interactive-assistant model that thinks while it talks. Instead of a long silent <think> block followed by the answer, it interleaves two readable token streams at token granularity:

• an inner voice (private reasoning, readable but not spoken / not sent to TTS), and
• an outer voice (the public answer, emitted in short chunks as soon as each statement is certain).

Architecture: Qwen3-4B with LoRA (r=16, all layers) and the last decoder layer's FFN replaced by a two-expert MoE (think expert / answer expert, both initialised from the pretrained FFN) selected per token by a disclosure router (small MLP on the last-layer hidden state). The router's decision at position t chooses the expert that produces token t+1, so the mode of every generated token is a single bit that comes for free during decoding - no special tokens, no decode-time controller. LoRA is folded into the weights here.

This checkpoint is the SFT stage trained on 1,550 DeepMath-103K problems with interleaved think/answer trajectories distilled from Qwen3-32B (high reasoning effort; private reasoning + reader-facing solution paragraphs aligned by anchor coverage). An RL stage (GRPO with correctness, premature-commitment and first-content-onset rewards) is in progress. Code, data recipe and report: https://github.com/lqxiao/InnerVoice

Usage (transformers, remote code)

from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "liqiangx/InnerVoice-Qwen3-4B"
tok = AutoTokenizer.from_pretrained(repo)
m = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, torch_dtype="bfloat16").cuda()
out = m.generate_streams(tok, "Find the sum of all positive divisors of 360.", max_new_tokens=2048)
print("INNER :", out["think"])    # private reasoning (do not speak)
print("OUTER :", out["speak"])    # public answer, newline-separated chunks

out["interleaved"] gives the token-level (mode, text) sequence for a streaming UI. Plain m.generate() works too but returns both streams mixed in one text; read m.moe.last_route to split them.

For math problems the model was trained with the instruction appended to the user turn: Solve the problem. Reason privately; publicly present the solution in short paragraphs as it becomes certain, and end with 'Final answer: \boxed{<answer>}'.

Usage (vLLM)

A vLLM plugin (custom InnerVoiceQwen3ForCausalLM class, vLLM 0.9.x) is in the GitHub repo under latent/vllm_plugin (pip install latent/vllm_plugin). It reproduces the transformers decode token-for-token; the per-token mode is recovered from the router (see latent/sxs_rl_engine.py::label_types).

Evaluation (AIME 2025, Average@4, temperature 1.0, 12k-token budget, 30 problems x 4 samples, +-8 pt)

All rows below are our own runs with the same evaluation code (accuracy = last \boxed{} in the PUBLIC stream; 12k new-token cap); "unfinished" = samples that hit the cap. The paper's Qwen3-4B numbers (Base 66.3, SxS SFT 38.3) use a 32k budget and their own training set and are not directly comparable.

model	AIME25 Avg@4	unfinished	think tokens	FCO	public blocks
Qwen3-4B base, native thinking mode	0.342	0.617	10,300	10,301	0.4
Qwen3-4B + Standard CoT SFT (think-then-speak, same 1,550 Qwen3-32B traces)	0.083	0.008	1,074	1,047	1.6
Qwen3-4B InnerVoice SFT, research variant with e_think (not released)	0.208	0.008	3,570	614	3.0
This checkpoint: Qwen3-4B InnerVoice SFT without e_think (vLLM decode)	0.158	0.058	3,094	1,205	3.7

The released checkpoint and the e_think research variant differ by 5 points, within the +-8 pt noise of 30 problems x 4 samples. InnerVoice starts speaking after ~600-1,200 tokens (FCO) and emits ~3 public chunks per problem, while the think-then-speak baseline trained on identical data speaks only once at the end and loses most of the accuracy; the base model keeps its accuracy only because it may think for 10k+ tokens before saying anything (and still fails to finish 62% of the time within the budget).

Metric definitions follow When to Think, When to Speak: Learning Disclosure Policies for LLM Reasoning (ICML 2026, arXiv 2605.03314): FCO = index of the first public token with task content, AIRW = mean private span before each public block.

Limitations

• Trained on 1,550 math problems only; not instruction-tuned for general chat beyond what Qwen3-4B already does.
• Think tokens are readable; the model is small (4B) and its inner voice can be wrong - treat it as a transcript, not a guarantee.
• The e_think input embedding of the research code was dropped (ablation showed no effect), so this checkpoint is a plain per-token two-expert model and runs on any engine that can host a custom Qwen3 class.

Citation

@misc{innervoice2026, title={InnerVoice: think while talking with a last-layer disclosure MoE},
      author={Liqiang Xiao}, year={2026}, url={https://github.com/lqxiao/InnerVoice}}