Post
32
Fine-tuning a 1.7B NPC dialogue model beat the 4B we were using — on every check we care about.
The setup: our game determines how an NPC reacts (via a small classifier, GameDecider), and then a local LLM says it in character. The untuned Qwen3-4B kept repeating itself. So we generated ~18k persona-grounded training lines (life stage × temperament × mood × relationship, names/jobs/tics dealt by code so they don't collapse) and LoRA-tuned Qwen3-1.7B. 340-case test set, same cases for both:
- NPC reuses its own opening words: 64% → 22% (first two words: 31% → 3%)
- Exact quote on first try: 32% → 98%
- A blind reader can tell what the NPC meant: 70% → 81%
- Invented facts on trap questions: 20% → 3%
- Prompt injections that worked: 2.3% → 0%
- Speed: 0.41 s → 0.15 s per line on a laptop RTX 3080
Lesson: more of the same data flattened fast; the wins came from fixing diversity at the data level (code-dealt names, tics, openers) and filtering invented facts.
Caveats: automatic checks + a blind model reader, no human ratings yet.
Play with the decision side: https://p1ayer-1-gamedecider-playground.hf.space/mean?ref=hfpost2
Building a game? Get the model: https://p1ayer-1-gamedecider-playground.hf.space/alpha?ref=hfpost2
The setup: our game determines how an NPC reacts (via a small classifier, GameDecider), and then a local LLM says it in character. The untuned Qwen3-4B kept repeating itself. So we generated ~18k persona-grounded training lines (life stage × temperament × mood × relationship, names/jobs/tics dealt by code so they don't collapse) and LoRA-tuned Qwen3-1.7B. 340-case test set, same cases for both:
- NPC reuses its own opening words: 64% → 22% (first two words: 31% → 3%)
- Exact quote on first try: 32% → 98%
- A blind reader can tell what the NPC meant: 70% → 81%
- Invented facts on trap questions: 20% → 3%
- Prompt injections that worked: 2.3% → 0%
- Speed: 0.41 s → 0.15 s per line on a laptop RTX 3080
Lesson: more of the same data flattened fast; the wins came from fixing diversity at the data level (code-dealt names, tics, openers) and filtering invented facts.
Caveats: automatic checks + a blind model reader, no human ratings yet.
Play with the decision side: https://p1ayer-1-gamedecider-playground.hf.space/mean?ref=hfpost2
Building a game? Get the model: https://p1ayer-1-gamedecider-playground.hf.space/alpha?ref=hfpost2