Hugging Face's logo

Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
Back to feed
P1ayer-1 
posted an update 5 days ago
Post
103
GameDecider: a 400M model that answers your game's "what happens next?" questions, open weights (Apache-2.0

GameDecider is an encoder (Ettin-400M base) fine-tuned to take *a scene + a question + options* and return a
probability for each option, or say "not sure". It runs locally on CPU or GPU (ONNX: about 0.1 s per check on an RTX 3060 Ti, about 1 s on a 6-core desktop CPU; up to 5 questions about the same scene
cost the same as one), and it ships with compiled adapters for common NPC questions.

What's in the repo:
- ONNX weights (fp16/fp32) + tokenizer, calibrated thresholds: default (answers ~97% of the time) and strict
(abstains more; for setups that hand "not sure" to an LLM)
- 39 compiled adapters for common NPC questions (threat, apology, persuasion, quest accept, ...), plus SkyrimNet-style memory matching and NPC mood, and a character-reaction adapter (14 reaction animations)
- Model card with evals, data sources and licences, a minor-safety section, and known limits; 200 sample rows

Honest numbers (private held-out sets, CIs in the card): 0.600 on task families it never saw (heldout_v3), 0.707 on real
game transcripts. Planning: 0.939 on our procedural "what should I do next to reach X?" eval, but the training data has
rows from the same generator, so treat that as in-family, not general planning; use an LLM or a planner for real plans.

Training data: ~334k decision rows from permissive datasets (MIT / Apache / CC-BY / ODC-By, all credited) plus
procedurally generated game scenarios written by open models. Removed before release: anything without a clear licence
and anything generated by models whose terms forbid training.

Try it: P1ayer-1/gamedecider-playground · Weights: holdoutlabs/gamedecider
Feedback very welcome — especially question types it gets wrong in your game.
dipankarsarkar

heldout_v3 is the number I'd ship on, and the card gives two halves of it: 0.600 accuracy over 2,993 questions, and 66.2% coverage at default. The missing half is how they combine.

Accuracy on the 66.2% it answers vs the 33.8% it declines.
If misses pile up in the declined third, a threshold fitted on dev (97.4% coverage there) still ranks risk on families it never saw.
If both buckets score near 0.600, coverage dropped but "not sure" isn't tracking error.

Is 0.600 argmax over all 2,993, like the cookbook raw row, and do you have the split?

·
P1ayer-1

Yes. The 0.600 is argmax over all 2,993 with no abstention, averaged over both option orders (avg2), same as the cookbook raw row. The split below is single-order (0.596 overall), since that's what the threshold report computes:
• answered (66.2%, n=1,980): 0.672
• declined (33.8%, n=1,013): 0.448
• strict profile: answers 50.7% at 0.722, declines 49.3% at 0.467
So the misses do pile up in the declined third, and "not sure" is tracking error, not just cutting coverage. But your first point holds too. On dev, the default thresholds give 97.4% coverage with ~5% error on what they answer. On heldout_v3, coverage falls to 66% and the answered error is still ~33%. It ranks risk on unseen families, but it's overconfident there. Out of distribution, read "answered" as "more likely right", not as calibrated, and if you need it, use strict or set your own threshold on a few dozen labelled examples from your game.
These CIs aren't clustered by family. The family-clustered ones would be wider; I can add them to the card.
dipankarsarkar

The band strict gives up scores about 0.51: the 463 questions default answers and strict declines, 1,330 minus 1,095 correct.
That sits nearer your declined third (0.448) than the answered two-thirds (0.672), so strict is cutting the right questions.

What the split can't show yet is where the 22.4 points live.
choice-2 and choice-11+ never abstain, so they only ever land in "answered". Only the thresholded buckets can decline, each with its own base rate, and that mix could push the gap either way.

Does it hold inside one bucket, say choice 3-5, answered vs declined?
(And thanks for putting the split on the card with family CIs.)

·
P1ayer-1

Yes, it holds inside each bucket, and it's wider there than pooled. Default profile, heldout_v3, family-clustered CIs:

  • choice 3–5 (n=1,002): answered .684 (771) vs declined .303 (231), gap +38.1 [25.9, 49.3]
  • choice 6–10 (n=393): .640 (308) vs .200 (85), gap +44.0 [29.7, 57.9]
  • yes/no (n=1,032): .740 (438) vs .561 (594), gap +17.9 [10.0, 25.0]
  • score (n=548): .609 (445) vs .330 (103), gap +27.9 [18.9, 37.7]

You're right that the mix moves it. Weighted by declines per bucket, the within-bucket gap is +25.7 [19.2, 31.3] vs 22.4 pooled. The pooled number is pulled down because yes/no questions are 59% of the declines, and a declined yes/no answer is still right 56% of the time (chance is 50%). Declined answers sit near chance in every bucket: .30 vs ~.23 for 3–5 options, .20 vs ~.15 for 6–10.

One note on the never-abstain buckets: heldout_v3 has no 2-option choice questions and only 18 with 11+ options, so they barely touch this split. And your 0.51 band is right (I get 236/463); by bucket it's .46 for choice 3–5, .42 for choice 6–10, .65 for yes/no.
dipankarsarkar

On yes/no, default declines more right answers than wrong ones.
594 declined at .561 is about 333 right and 261 wrong.

Every other bucket goes the other way:
choice 3-5 declines about 70 right to stop 161 wrong.
choice 6-10, 17 to stop 68. score, 34 to stop 69.

And yes/no is where it declines most: 58% of the bucket, vs 19-23% everywhere else.
So the default spends most of its abstention budget where a decline is worth least, about 0.8 wrong answers stopped per right one lost, against 2.3 on choice 3-5.

Was the yes/no cutoff fitted on dev to a coverage target, and what did declined yes/no score there?
In this post