Model card: point to PythiaFinance/8ball, inline criteria + sampling, drop private-repo links
76c4106 verified | license: apache-2.0 | |
| language: | |
| - en | |
| base_model: convaiinnovations/laya-typed-decisions | |
| library_name: laya | |
| pipeline_tag: text-classification | |
| tags: | |
| - system-one | |
| - decision-model | |
| - rlcd | |
| - typed-decisions | |
| - 8ball | |
| # 8ball | |
| *Published by [PythiaFinance](https://huggingface.co/PythiaFinance).* | |
| A System One decision model fine-tuned to **be a Magic 8-Ball**: it answers one | |
| 20-way `choice` question — "the asker posed the question in the state; answer it | |
| with the classic Magic 8-Ball response that fits best" — with a calibrated | |
| probability distribution over the canonical 20 answers (10 affirmative / | |
| 5 non-committal / 5 negative), each mapped to a signed alignment value in [-1, 1]. | |
| 421M parameters (ModernBERT-large encoder + decision head), Apache-2.0, runs on | |
| CPU (~330–390 ms/question) or a single consumer GPU (~5–10 ms). Loads like any | |
| Laya checkpoint: | |
| ```python | |
| import laya | |
| CRITERIA = { # the canonical 20, verbatim — the model was trained on exactly these | |
| "It is certain": "Strong affirmative: no doubt whatsoever", | |
| "It is decidedly so": "Strong affirmative: beyond question", | |
| "Without a doubt": "Strong affirmative", | |
| "Yes definitely": "Strong affirmative", | |
| "You may rely on it": "Strong affirmative: dependable yes", | |
| "As I see it, yes": "Mild affirmative: yes with minor reservations", | |
| "likely": "Mild affirmative: probable yes", | |
| "Outlook good": "Mild affirmative: favorable prospects", | |
| "Yes": "Plain affirmative", | |
| "Signs point to yes": "Mild affirmative: indicators favor yes", | |
| "Reply hazy, try again": "Non-committal: rephrase and ask again", | |
| "Ask again later": "Non-committal: no answer available right now", | |
| "Better not tell you now": "Non-committal: answer withheld for now", | |
| "Cannot predict now": "Non-committal: prediction unavailable at this time", | |
| "Concentrate and ask again": "Non-committal: asker should focus and retry", | |
| "Don't count on it": "Mild negative: unlikely", | |
| "My reply is no": "Strong negative: the ball says no", | |
| "My sources say no": "Strong negative: authoritative no", | |
| "Outlook not so good": "Mild negative: unfavorable prospects", | |
| "doubtful": "Mild negative: doubtful", | |
| } | |
| agent = laya.load("PythiaFinance/8ball") | |
| result = agent.predict("Should I deploy on Friday?", { | |
| "answer": { | |
| "type": "choice", | |
| "instructions": "The asker posed the question found in the state. Answer it with the classic Magic 8-Ball response that fits best.", | |
| "criteria": CRITERIA, | |
| } | |
| }) | |
| probs = result["answers"]["answer"]["probabilities"] # {answer: p}, sums to 1 | |
| ``` | |
| ### Sampling & deflection | |
| The distribution is the point, so sample from it rather than taking the argmax. | |
| When the model is unsure (`p_max < 0.09`, calibrated to this checkpoint), restrict | |
| the draw to the five non-committal answers: | |
| ```python | |
| import random | |
| NON_COMMITTAL = ["Reply hazy, try again", "Ask again later", "Better not tell you now", | |
| "Cannot predict now", "Concentrate and ask again"] | |
| def shake(probs, temperature=0.9, deflect_below=0.09): | |
| pool = NON_COMMITTAL if max(probs.values()) < deflect_below else list(probs) | |
| weights = [probs[a] ** (1 / temperature) for a in pool] | |
| return random.choices(pool, weights=weights)[0] | |
| ``` | |
| The signed alignment score is `value(answer) * probs[answer]`, where `value` runs | |
| +1.0 → +0.1 down the affirmative ladder (in 0.1 steps: It is certain, It is | |
| decidedly so, Without a doubt, Yes definitely, You may rely on it, Yes, As I see | |
| it yes, likely, Outlook good, Signs point to yes), is 0 for all non-committal | |
| answers, and −1.0 → −0.2 down the negative ladder (in 0.2 steps: My reply is no, | |
| My sources say no, Don't count on it, Outlook not so good, doubtful). | |
| ## Results | |
| ### The 8-ball persona (held-out eval, n=620) | |
| | metric | value | | |
| |---|---| | |
| | Sampled bucket mix | **49.7 / 25.1 / 25.2** (canonical: 50 / 25 / 25) | | |
| | Brier vs persona gold | **0.0031** (teacher: 0.0095, uniform: 0.0175) | | |
| | Mean p_max | 0.106 (p5 0.084 · p50 0.100 · p95 0.145) — deliberately humble | | |
| | Deflection rate @0.09 | 18.7% of shakes fall back to the non-committal answers | | |
| ### General decision skill (S1MB english-v1, 137 benchmarks, 26,269 judgments) | |
| Evaluated with the official [S1MB evaluator](https://github.com/hotchpotch/S1MB) | |
| (baseline-adjusted scores 0–100; extended-input condition `--max-len 9216`). | |
| | model | noul | choice | score | **Task Avg** | | |
| |---|---|---|---|---| | |
| | TypeSafe Jev 1.13 | 64.77 | 67.31 | 46.69 | 59.59 | | |
| | laya-typed-decisions (base, published) | 20.06 | 18.93 | 5.99 | 15.00 | | |
| | 8ball v1 (persona-only fine-tune) | 18.77 | 17.14 | 2.14 | 12.69 | | |
| | 8ball v2 (MTL r1, 47k items) | 28.87 | 21.47 | 19.18 | 23.17 | | |
| | **8ball v3 (this model, 143k items, 6 epochs)** | **33.85** | **24.82** | **21.43** | **26.70** | | |
| The multi-task training traded ~1.7 Task Avg vs the base teacher for the persona | |
| (mix, alignment scale, humility) — and recovered general skill 12.69 → 26.70 along | |
| the way. Methodology is summarized under Training below. | |
| ## Training | |
| - **Recipe**: RLCD (upstream Laya loop) — proper-scoring-rule policy gradient | |
| (spherical 0.75 + ranked probability 1.0) over noisy logit projections, plus soft | |
| cross-entropy against gold distributions. 6 epochs total, fp32, per-type | |
| temperature calibration after the final epoch. | |
| - **Corpus** (143,514 items, 0 skipped): | |
| - Open-Jev `release-v2-redistributable` train (79k, CC0) — all 12 sources, uncapped | |
| - 14 public NLP train splits converted to S1MB decision formats (30.1k): SNLI, | |
| HANS, PAWS, CREAK, SciTail, ETHOS, Civil Comments, DBpedia, AG News, | |
| poem_sentiment, ARC, OpenBookQA, AQuA-RAT, Banking77 | |
| - 8-ball persona gold (11.6k effective): laya-typed-decisions teacher | |
| distributions over 6,200 whimsical questions, bucket-rescaled to the canonical | |
| 50/25/25 mix, blended 70/30 with uniform (humility baked into targets) | |
| - Qwen3.8-27B-generated coding/logic/finance decision cases (4.4k) | |
| - **Never trained on S1MB test rows** — NLP conversion transfers decision *formats* | |
| (instructions/criteria/state shape read from local test schemas) applied to | |
| train-split rows only. | |
| ## Limitations | |
| - **No world knowledge**: it judges the state you hand it. Without evidence in the | |
| state it is deliberately near-uniform — an honest 8-ball, not an oracle. | |
| - The persona is trained-in: this checkpoint is near-uniform on almost everything. | |
| For general typed decisions use `convaiinnovations/laya-typed-decisions`. | |
| - Score-primitive skill is weak (21.4) relative to frontier decision models. | |
| - Trained for the fixed 20-option 8-ball choice; other option sets work | |
| (request-time declaration) but are not calibrated for. | |
| ## Intended use | |
| A calibrated, whimsical decision layer: "should I…" questions in → an 8-ball | |
| answer with a signed alignment score between −1 and +1 (direction from the | |
| sampled answer, magnitude from the probability mass). See *Sampling & deflection* above | |
| (deflection threshold calibrated to this checkpoint: 0.09). | |