houssemzaier/jeff-gestures

Jeff gestures

Jeff v1.3 taught to pick a talking avatar's gesture, in one 1.08 GB file for llama.cpp, run on the phone by the Iris app. While the persona listens, it reads what the user just said and picks one of seven reactions: smile, touch_chin, tilt_head_right, lean_slow, touch_hair, move_subtle, or the nod, listen. It works in English, French, Arabic and Arabizi, and in the 96 other languages of the app's language picker. It is Jeff v1.3 (a 0.8B decision model built on Qwen3.5-0.8B) with a LoRA adapter merged into it, trained on invented messages only.

Files

File	What it is	Size, storage	sha256
jeff-gestures-r5-hard-merged-q8_0.gguf	Jeff v1.3 with the adapter merged, every number on 8 bits (Q8_0): the file the app runs	1.08 GB	ad45771ffe634b784199af3ae11a171f6a98b3f6af7b4ada56b9f0b18baaf4ba
jeff-gestures-r5-hard.jeff.json	what an app needs next to the model: the answer codes and their token ids, the two questions, the temperature of each way to run it, the margin
lora/jeff-gestures-r5-hard.lora.gguf	the adapter alone, for llama.cpp's --lora, on Jeff v1.3's own GGUF (mstrasser/jeff-base-gguf at a5d0e595)	0.17 GB	07a983b7b8f681df7a5c8be88915c03708ded62622c3342d68a4dabbab938322
adapter/	the adapter for PyTorch, as Jeff's kit saved it: the LoRA (PEFT) and Jeff's readout; its base path is the training Mac's, so point it at a local copy of mstrasser/jeff-base v1.3	0.04 GB
training/train-r5-hard.log	the training run's log: the evaluations at steps 0, 82, 164, 246, 328 and 410

"Storage" is the room a file takes on the phone's disk, and what it downloads. The RAM it takes while it runs is another number, given under "Results".

How it decides

the user's message
   |
   +-- a clear sign (a greeting, thanks, love, a goodbye... in English, French or Arabic words)?
   |      yes --> that sign's gesture
   |
   +-- no --> Jeff reads the app's prompt and gives a score to each of the 7 answer codes (A to G)
                 --> probabilities: the scores divided by T, then a softmax
                 --> does the best gesture beat `listen` by more than m?
                        yes --> the gesture
                        no  --> `listen` (the nod)

p(i) = exp(s(i) / T) / [ exp(s(A) / T) + ... + exp(s(G) / T) ]     s(i): the model's score for code i
g    = the option other than listen with the highest p
play g if p(g) > p(listen) + m, otherwise play listen               T = 1.3478, m = 0.8 for this file

• The prompt is the app's own (JeffReactionPrompt.prompt), in Jeff v1.3's request format: the state (The user says: <message>), the question, then the options under the codes A to G. Jeff answers with one code, so llama.cpp needs one pass: the app reads the log-probabilities of the 7 code tokens (their ids are in the .jeff.json).
• T and m belong to this file. Rounding the numbers to 8 bits shifts the scores a little, so each export has its own T (all four are in the .jeff.json), and m was chosen with this file's probabilities. A new export needs both again.

Results

Measured with llama.cpp b10453 on a Mac mini (Metal), the clear signs on. Right / real gestures / wrong:

Set	Qwen3 0.6B (the app's first model)	Jeff v1.3 alone	This file, margin 0.8	The bar
Unseen short (120 messages: English, French, Arabic, Arabizi)	59 / 21 / 0	79 / 61 / 12	93 / 71 / 0	holds
Unseen long (120)	58 / 14 / 0	79 / 69 / 28	99 / 79 / 1	holds
World short (2,880: the same messages in 96 other languages)	32% / 2% / 0%	53% / 37% / 12%	62% / 39.5% / 4.5%	misses narrowly
World long (1,440)	39% / 1% / 0%	53% / 47% / 37%	75% / 56% / 2.2%	holds

• Right is a reaction that fits the message (a nod can be right), real gestures are the right ones that are not a nod, wrong is a gesture that does not fit (a smile at bad news). Qwen and Jeff v1.3 ran on a OnePlus 12 for the two unseen sets, on the Mac for the world sets.
• On an iPhone 16 Plus, inside the app, this file made the Mac's pick for every message of the four sets (4,462 picks, probabilities within 0.001), in 0.26 s per message (median, Unseen short).
• RAM: on the iPhone 16 Plus the app's footprint does not grow (0.69 GB with a Q8_0 file of this size and Qwen, against 0.79 GB with Jeff v1.3 Q4_K_M; measured with round 3's file), because the GPU reads the file in place. On a OnePlus 12 (Android, CPU), llama.cpp copies the numbers into its own memory: about 0.6 GB more than Jeff v1.3 Q4_K_M (estimated), so about 2.8 GB for the whole app with both models.

How it was made

The architecture.

• Qwen3.5-0.8B (Qwen team, Apache-2.0): 24 layers with a hidden state of 1,024 numbers; 18 layers use linear attention (Gated DeltaNet) and every fourth uses full attention.
• Jeff v1.3 (mstrasser/jeff-base, revision v1.3; its kit is github.com/firelex/jeff): Qwen3.5-0.8B trained to decide. It reads a state, a question and up to 254 options, and answers with one code in a single pass. Its "readout" is the output rows of those codes.
• The adapter: a LoRA of rank 16 (alpha 32) on 150 weight matrices of the 24 layers (q_proj, k_proj, v_proj, o_proj in the full-attention layers; in_proj_qkv, in_proj_z, out_proj in the linear ones; gate_proj, up_proj, down_proj in every layer's MLP), plus Jeff's readout rows: 10.5 million trained numbers, about 1% of the file's 1.0 billion. Everything else stays frozen.
• The file: the adapter added into Jeff's weights (merged), then every number rounded to 8 bits, in blocks of 32 that share one scale (Q8_0), with llama.cpp b10453.

The data. Invented messages only, written by AI writers from briefs that never showed them a test set: 2,680 rows (1,880 for the reaction while listening, 800 for the reply) in English, French, Arabic in four varieties and Arabizi, short and long; 640 more rows for training only, where an earlier round was weak; and 5,760 rows translated into the app's 96 other languages. Every row's prompt equals the app's byte for byte. The rows are split by family: 70% to train, 10% for choices (development), 10% for the temperature (calibration), 10% for our own final grade (test).

The training. jeff-train (kit commit 3720e7c, with two patches: the Mac's GPU, and Qwen3.5's tokenizer cutting text as llama.cpp does), one pass over 6,560 rows, 16 rows per step (410 steps), learning rate 4.5e-4, on the plain labels, on a Mac mini M4 Pro's GPU: 2 h 50 min. The learning rate won a sweep: 5e-5, 1.5e-4 and 4.5e-4, where 4.5e-4 had the lowest development loss; 1.35e-3 diverged.

Why LoRA

Way	What it would mean here	Verdict
Jeff v1.3 as it is	the app's prompt is fixed, and Jeff alone plays a wrong gesture on 10% of Unseen short and 23% of Unseen long	far from the 3% bar
Rules only (word lists)	the app's clear signs already catch obvious messages, but they fire on few: 19 of 120 on Unseen short, 9 of 576 development messages in the 96 new languages	a complement, not the answer
Full fine-tuning	retrain all 1 billion numbers: more memory and time for each try, a higher risk of losing what Jeff knows, a whole new model for each try	too heavy for a few thousand examples
A separate small classifier	another model, runtime and download in the app, for one choice	more app work, nothing gained
LoRA	train 1% of the numbers in small side matrices, Jeff frozen; Jeff's kit is built for it (Jeff's own task adapters are LoRA)	chosen: under 4 hours on a Mac mini, a 0.04 GB result, merged into one file the app already runs

How each number was chosen

The method. Every choice is made on our own invented rows, never on the app's test sets: the development pile chooses (the learning rate, the margin), the calibration pile sets the temperature, the test pile gives our own grade. The app's test sets only judge a final candidate, once, and every look at them is logged (DECISIONS.md). The rules were written before the results they judge, so that a lucky number on a few messages cannot steer a choice.

The temperature T is the value that makes the calibration pile's probabilities most honest: it minimizes their average negative log-likelihood, one number for the whole file (Jeff's own fit):

T = the value that minimizes  -(1/N) x [ log p_T(right answer of row 1) + ... + log p_T(right answer of row N) ]

T above 1 means the trained model was a little too sure of itself, and T spreads its probabilities out: 1.3478 for this file.

The margin m follows a rule written before round 2's training: on the development pile's listening rows, the clear signs first, try m = 0.1, 0.2, ..., 0.8; keep the smallest m with at most 1.5% wrong picks; if none reaches it, the m with the fewest wrong picks, the smaller one on a tie.

• 1.5% is half the bar's 3%. The development rows are our own messages, closer to the training rows than the app's unseen sets, so the target at home is stricter, to keep room.
• The grid stops at 0.8. Past it, a gesture would need more than 90% probability, and the avatar would hardly move: at 0.8, round 5 already plays only 41% real gestures on the development pile, against the bar's 40% floor.
• The fewest wrong picks, then the smaller m: a wrong gesture is the mistake a user sees; at equal mistakes, the smaller m plays more gestures.

Round 3 (four languages, 186 development rows, its merged Q8_0; its adapter file gives the same choice):

m	0.1	0.2	0.3	0.4	0.5	0.6	0.7	0.8
Wrong	20 (11%)	15 (8%)	11 (6%)	11 (6%)	10 (5%)	8 (4%)	7 (4%)	7 (4%)
Real gestures	63%	62%	62%	61%	60%	58%	57%	54%

None reaches 1.5%; 0.7 and 0.8 tie at 7 wrong picks, and the smaller one plays more gestures: m = 0.7.

Round 5, this file (762 development rows: the same 186 plus 576 in the 96 new languages):

m	0.1	0.2	0.3	0.4	0.5	0.6	0.7	0.8
Wrong	168 (22%)	156 (20%)	145 (19%)	117 (15%)	90 (12%)	63 (8%)	48 (6%)	31 (4%)
Real gestures	65%	64%	62%	60%	56%	52%	47%	41%

None reaches 1.5%, and 0.8 has the fewest wrong picks: m = 0.8. The model is less sure in the new languages, so the same rule asks for a bigger gap.

The bar (issue #1335) judges the app's test sets, with the clear signs on:

• At most 3% wrong (3 of 120): a wrong gesture is the mistake a user sees; a nod is never shocking.
• At least 40% real gestures: below that, the avatar mostly nods and the feature adds little.
• In each language, at least as many real gestures as Qwen: no language gets worse than with the app's model today.
• Step 5, the phone file decides like the trained model: at least 98% of PyTorch's picks on the 838 test rows. This file gives 99.3%; the same model in Q4_K_M only 92.7%, which is why the app gets the Q8_0.

Six rounds. Each round changed one thing and was judged by these rules.

Round	What changed	Verdict
1	the learning rate (a sweep of four)	4.5e-4
2	learning to nod when unsure, against a margin applied in the app	the margin, read from the probabilities, does better
3	640 more rows where round 2 was weak	the bar holds in English, French, Arabic and Arabizi, m = 0.7
4	the app's 96 other languages, by translated rows	a tokenizer bug in the kit: the file followed PyTorch on only 96.4%
5	the same, with the tokenizer fixed	this file: the bar holds except short messages in the 96 languages, m = 0.8
6	4,800 more short rows in the 96 languages	worse than round 5, which stays

Limits

• Short messages in the 96 new languages miss the bar narrowly: 39.5% real gestures and 4.5% wrong. Round 6 did not fix it.
• The gesture at the end of the persona's reply (English, French, Arabic, Arabizi) is trained too, but the app has no prompt for it yet: it must send exactly the prompt the adapter learned (prompts.app_prompt(reply, "reply") in the training tools).
• Invented data only: no real user conversation was used.

License

Apache-2.0, as its sources: Jeff v1.3 (mstrasser/jeff-base, Apache-2.0; its kit, github.com/firelex/jeff, is MIT) and Qwen3.5-0.8B (the Qwen team, Apache-2.0). The changes made here: a LoRA adapter trained by Iris (irisona) on invented data, merged into Jeff v1.3's weights and rounded to Q8_0. Made and measured in the Iris app's repository (tools/jeff-adapter/, issue #1335, pull request #1340), where every choice and dead end is written down (tools/jeff-adapter/DECISIONS.md).