live-vibe timing LoRA

What it is

This is the timing LoRA for the micro-turn front in live-vibe, my Claude Code plugin in potto007/ottomation under plugins/live-vibe. Every 200 ms, a 4B text model on llama-server gets the user's partial transcript in a window such as [voice] can you run the tests [p=0.8], [no voice] or [speaking], and decides whether to listen or speak. When it speaks, it returns one short line and an optional delegation to Claude Code as JSON {"action","delegate","say"}. Putting the duplex decision on a text LLM comes from the DuplexCascade idea (arXiv 2603.09180).

I trained the adapter on timing, so that the model waits while the user is mid-sentence, answers at the end of the utterance, stays quiet after its own answer, and never claims work it has not done. On the held-out windows below, the base model gets 37% of the mid-utterance decisions right and 1% of the decisions right after its own answer.

How to use it

In live-vibe, set frontServerLora to timing with microturnMs on, and the plugin downloads this file pinned by revision and sha256.

In plain llama.cpp, start the server with the adapter loaded:

llama-server -m Qwen3-4B-Instruct-2507-Q4_K_M.gguf --lora timing-lora.gguf

Then set its global scale to zero with POST /lora-adapters:

[{"id":0,"scale":0}]

and send "lora":[{"id":0,"scale":1}] only on the micro-turn requests. When I applied the first adapter to every request, it made the front's other turns worse, which the relay eval in the results shows. The POST is needed because in b11146, --lora-init-without-apply does not stop the adapter applying to requests that name no adapter.

Prompt coupling

I trained the adapter on one exact front prompt, micro-turn protocol and window format, so it is tied to all three. The fingerprint in training.json is a sha256 over them, computed by sidecar/timing_lora.py:fingerprint() in the plugin, and the plugin's unit checks fail when the code's fingerprint differs. On another prompt the adapter still loads, but I have not tested its timing there.

Training

The data is synthetic and templated. make_windows.py simulates 600 calls and turns them into 25,537 windows. Training used 4,427 rows (1,771 speak, 2,656 listen), and I held out 2,591 windows from calls the training never saw. No real user audio or transcripts went into it.

This is the second adapter. The first one, at revision b623743, scored 96% on its held-out set, and the misses showed me two faults in the generator rather than in the model. The mid-utterance p rounded up to 0.6 about one window in fifty, so the finished sentence appeared at [p=0.6] labelled listen and then again on the end-of-utterance window labelled speak. And the dictation windows carried "Mm-hm." on one pause in four, drawn at random, which no model can learn and which the plugin's loop could not speak anyway, since it holds every speak while the user is talking. I removed both, and the plugin's own clips do the backchannelling.

I trained with TRL 1.5.1 SFTTrainer, PEFT 0.19.1, transformers 5.12.1 and torch 2.10 on a bf16 base, with LoRA r=16, alpha 32 and dropout 0.05 on q, k, v, o, gate, up and down, which comes to 33.0M parameters. The loss covers the assistant JSON only (completion_only_loss). The run was 1 epoch of 277 steps at lr 2e-4 on a cosine schedule with 5% warmup, batch 1 with 16 steps of gradient accumulation, and max length 2048. It took 29 minutes on one RTX 5090 at 12.5 GiB peak. Train loss was 0.061 and held-out eval loss 0.00013; the first adapter's 0.0116 was the random labels. I made the GGUF with llama.cpp's convert_lora_to_gguf.py --outtype f16.

Results

This is listen/speak decision accuracy on all 2,591 held-out windows, base versus LoRA, both on the Q4_K_M GGUF on llama.cpp b11146.

Window type	n	Base	LoRA
Mid-utterance	1315	37%	100%
Silent	452	100%	100%
Silent while speaking	394	99%	100%
Utterance end	110	88%	100%
Utterance end, raw	107	91%	100%
User talking over the voice	108	64%	100%
Right after its own answer	99	1%	100%
Stop	6	67%	100%
All	2591	62%	100%

On the 196 windows where both the label and the model speak:

Speak-window metric	Base	LoRA
Delegation present when expected, absent when not	93%	100%
Delegation close to the label	32%	70%
Claims of unfinished work	6%	1%
Echoes of the user's words	1%	0%
Long lines	3%	0%

The relay eval covers the front's other JSON turns, with 17 fixtures run 3 times each. Those turns never name the adapter and the server holds it at scale 0, so with the adapter scoped the two arms run the same base model and the gap between them is sampling:

Setup	Passed
Base	46/51
This adapter on micro-turn requests only	45/51
First adapter on every request	39/51

Both scoped arms fail the same two fixtures, a combined report and a single result announced twice, with no invented actions and no question asked twice.

Limits

The held-out calls come from the same template generator as the training calls, so 100% is the ceiling on this generator and says nothing yet about a live session. The adapter is English only, and I have tested it only on the Q4_K_M GGUF of the base.

Files

timing-lora.gguf is the adapter as an f16 GGUF LoRA for llama.cpp.
peft/adapter_config.json and peft/adapter_model.safetensors are the PEFT adapter.
training.json holds the prompt fingerprint, plugin version, training source, base model revision, and the GGUF and llama.cpp build it was tested with.