Hugging Face's logo

Buckets:
Mercity
/
Desicion-Model-Custom

Files
xet
Mercity/Desicion-Model-Custom / run1 /logs /check_report.log
Pranav2748's picture
Pranav2748
download
raw
3.8 kB
# Reasoning decision model: results
## Hypotheses
| # | Result | Numbers | Confirmed |
|---|---|---|---|
| H1 | joint readout vs gold-only, in-distribution accuracy | +0.0 pts [+0.0, +0.0] | no |
| H2 | disagreement AUROC, joint vs others (need +0.03) | vs gold_only +0.000 [+0.000, +0.000]; vs base +0.000 [+0.000, +0.000] | no |
| H3 | joint reasoner vs base reasoner, all eval | +0.0 pts [+0.0, +0.0] | yes |
| H4 | gap recovered at f = 0.3 (need 80%) | 33.3% of 52.6 → 56.4; learned router 66.7% | no |
## Readout
Accuracy (%) after per-type temperatures fitted on the calibration set; ECE uses 15 bins.
| Readout | Procedural | tasksource | OpenDecisions | LocalLLaMA | LocalLLaMA (panel agrees) | OD Capability | OD Capability (raw) | In-dist ECE | In-dist NLL |
|---|---|---|---|---|---|---|---|---|---|
| base | 38.9 | 68.8 | 75.0 | 40.0 | 100.0 | – | – | 0.189 | 1.206 |
| gold_only | 38.9 | 68.8 | 75.0 | 40.0 | 100.0 | – | – | 0.189 | 1.206 |
| votes_only | 38.9 | 68.8 | 75.0 | 40.0 | 100.0 | – | – | 0.189 | 1.206 |
| joint | 38.9 | 68.8 | 75.0 | 40.0 | 100.0 | – | – | 0.189 | 1.206 |
Temperatures (choice / yes-no / score): base 2.49 / 20.00 / 3.16; gold_only 2.49 / 20.00 / 3.16; votes_only 2.49 / 20.00 / 3.16; joint 2.49 / 20.00 / 3.16
## Reasoning (one sample per item)
| Reasoner | Procedural | tasksource | OpenDecisions | LocalLLaMA | All | Median tokens | p90 tokens | Looped | Letter = readout after chain |
|---|---|---|---|---|---|---|---|---|---|
| base | 2.8 | 0.0 | 56.2 | 0.0 | 12.8 | 32326 | 32571 | 85.9% | 14.1% |
| joint | 2.8 | 0.0 | 56.2 | 0.0 | 12.8 | 32326 | 32571 | 85.9% | 14.1% |
## Does the readout agree with its reasoning?
Agreement = the readout's answer equals the same model's thinking-mode answer (the base reasoner for the readout-only ablations). The thinking call never sees the readout's answer, so this is also the faithfulness check of explanation on demand.
| Readout | Agreement | Procedural | tasksource | OpenDecisions | LocalLLaMA | Disagreement AUROC | Fixable AUROC |
|---|---|---|---|---|---|---|---|
| base | 72.7 | 0.0 | 100.0 | 77.8 | nan | 0.792 | 0.333 |
| gold_only | 72.7 | 0.0 | 100.0 | 77.8 | nan | 0.792 | 0.333 |
| votes_only | 72.7 | 0.0 | 100.0 | 77.8 | nan | 0.792 | 0.333 |
| joint | 72.7 | 0.0 | 100.0 | 77.8 | nan | 0.792 | 0.333 |
## Routing
Route the lowest-margin share f of items to thinking mode (all eval sets pooled).
| Readout | Router | f=0 | 0.1 | 0.2 | 0.3 | 0.5 | 1.0 | tokens at 0.3 | random at 0.3 |
|---|---|---|---|---|---|---|---|---|---|
| base | margin | 52.6 | 53.8 | 53.8 | 53.8 | 55.1 | 56.4 | 9093 | 53.7 |
| base | learned | 52.6 | 53.8 | 53.8 | 55.1 | 55.1 | 56.4 | 8714 | 53.7 |
| gold_only | margin | 52.6 | 53.8 | 53.8 | 53.8 | 55.1 | 56.4 | 9093 | 53.7 |
| gold_only | learned | 52.6 | 53.8 | 53.8 | 55.1 | 55.1 | 56.4 | 8714 | 53.7 |
| votes_only | margin | 52.6 | 53.8 | 53.8 | 53.8 | 55.1 | 56.4 | 9093 | 53.7 |
| votes_only | learned | 52.6 | 53.8 | 53.8 | 55.1 | 55.1 | 56.4 | 8714 | 53.7 |
| joint | margin | 52.6 | 53.8 | 53.8 | 53.8 | 55.1 | 56.4 | 9093 | 53.7 |
| joint | learned | 52.6 | 53.8 | 53.8 | 55.1 | 55.1 | 56.4 | 8714 | 53.7 |
## Run
- Throughput probe: 1880 generated tokens/s; chain tokens median 32211, p90 32334, max 32334; parse rate 18.8%.
- Generation: 16 items, 30 chains, 0.9M tokens; 1 items with a correct chain; 29 chains looped to the context limit.
- Training joint: 1 reasoner + 16 readout examples, 0.0M tokens, 1/1 steps, 3 min (79 tokens/s).
- Training gold_only: 0 reasoner + 16 readout examples, 0.0M tokens, 1/1 steps, 0 min (391 tokens/s).
- Training votes_only: 0 reasoner + 16 readout examples, 0.0M tokens, 1/1 steps, 0 min (1930 tokens/s).
- Single seed (17); see results/DEVIATIONS.md for every change to the plan.

Xet Storage Details

Size:
3.8 kB
·
Xet hash:
b5ecc9242cd444dbd63ab4b92e9da89867c99b77cf4a192b431680a19dc216ed

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.