FIS-Jev

A LoRA adapter for Qwen3-Omni-30B-A3B-Instruct that scores a therapist's spoken response on the 8 items of the Facilitative Interpersonal Skills (FIS) task, from the response video (audio + webcam frames) and its transcript.

It does not generate text. One forward pass reads the logits of the tokens 1–5 at 8 slots, giving a 1–5 distribution per item. The expected value is the score and the distribution is its confidence.

Model details

Base model	Qwen3-Omni-30B-A3B-Instruct, thinker only (no talker)
Adapter	LoRA r=16, α=32 on q_proj, k_proj, v_proj, o_proj of all 48 decoder layers; 13.4M parameters
Frozen	Base weights, audio and vision encoders, LM head
Input	Audio (16 kHz) and webcam frames (0.5 fps, 320×180, at most 64 frames) interleaved in 2 s chunks, followed by the transcript, after the client context
Output	Per item: 5-way distribution over 1–5 and its expected value; FIS total = mean of the 8 expected values
Items (slot order)	verbal fluency, hope, persuasiveness, emotional expression, warmth, empathy, alliance bond, rupture repair

Files

File	Content
adapter.safetensors	LoRA weights. Keys model.layers.{i}.self_attn.{q,k,v,o}_proj.{A,B}; the update is (α/r)·B·A added to the frozen projection
adapter_config.json	Base model, LoRA settings, input settings and slot order

The adapter uses its own LoRA layer, not PEFT. Load it with fis_jev/infer.py from the code repository.

Usage

Code: github.com/jbnhandsome/fis-jev

pip install -r requirements.txt
python fis_jev/infer.py \
  --model /path/to/Qwen3-Omni-30B-A3B-Instruct \
  --adapter adapter.safetensors \
  --video response.mp4 --transcript-file response.txt \
  --client-prompt "<background of the client clip>" --client-says "<what the client says>"

Output (JSON): for each item score (1–5) and probs (distribution over 1–5), plus fis_total, input_tokens and forward_ms.

Hardware: one 80 GB GPU (about 64 GB in bf16). Tested with torch 2.14.1 and transformers 5.19.0.

Training

• Data: 996 FIS responses with video, transcript and rater scores, split by participant into train 554 / dev 212 / test 230. Scores are very concentrated: 3 and 4 make up 86–97% of every item.
• Labels: soft labels built from the individual raters' half-point scores, whose expected value equals the 3-rater mean. Loss: cross-entropy.
• Epoch count chosen on dev (5), then retrained on train + dev (766 responses). AdamW, lr 2e-4, cosine schedule, 4×H100.
• The training data is not included and is not public.

Results (held-out test set)

FIS total vs. the mean of 3 raters, Pearson r (230 responses)	0.71 (zero-shot base model: 0.54)
Agreement with the other two raters' mean, mean of 8 items (178 responses)	model 0.64, single human rater 0.62
Share of the noise ceiling reached (FIS total)	77%
Calibration (ECE per item, no post-hoc calibration; same setup trained on train only)	0.03–0.15
Latency of one pass, single H100, bf16	~130–300 ms after warm-up

Per item, vs. the 3-rater mean:

Item	r
Verbal fluency	0.70
Hope	0.75
Persuasiveness	0.64
Emotional expression	0.58
Warmth	0.55
Empathy	0.58
Alliance bond	0.73
Rupture repair	0.62

The model is stronger on content items (hope, empathy, alliance bond) than on delivery items (verbal fluency, emotional expression). Full results and ablations: RESULTS.md in the code repository.

Intended use and limitations

• For research and for practice feedback in training settings. Not for clinical, hiring or certification decisions.
• Trained and tested only on the FIS task: a single response to a recorded client clip. Multi-turn sessions are not validated.
• Trained on human-cleaned transcripts; accuracy with automatic speech recognition transcripts is untested.
• Small evaluation set (230 test responses; 178 with 3 raters, from 12 participants).
• Scores can differ by a few hundredths of a point between runs (bf16 / MoE numerics).