Scotty

Scotty is a compact, 421-million-parameter decision model trained with Beam-generated data and Typed Decisions examples. It reads a situation, a question, and the supplied answer options together, then scores all options in a single forward pass. It supports boolean, categorical, and ordered decisions without generating free-form text.

• Inference code: run_scotty.py
• Paper: paper/scotty-paper.pdf
• Demo: https://spark-29fd.taile0f6e8.ts.net (access-controlled)

Contact

For research questions or demo access, contact Oliver Kramer at oliver.kramer@uni-oldenburg.de.

How it works

Scotty combines a pretrained answerdotai/ModernBERT-large encoder with a newly initialized two-layer Laya-format decision head. Each option receives a [MASK] marker. A shared scalar head scores those marker states, and softmax ranks the options. Candidates are encoded jointly rather than in separate chunks.

The output values are comparative scores, not calibrated confidence. Scotty does not produce explanations or hidden reasoning.

Training data

Reflection Beam 1.0 501B-A23B generated 19,878 typed decisions across 18 task families. Beam supplied supervision, not weights. Fitting uses 11,948 Beam decisions plus 4,320 Typed decisions from LocalLLaMA/typed-decisions. The earlier 13,292/4,800 counts described source pools before reserving 1,344/480 audit decisions. The released weights match the audited finalist and its fitting configuration; see provenance verification.

Every effective batch is split 50:50 between Beam-generated and Typed examples, with uniform sampling with replacement inside each source. This does not guarantee every fitting item is seen. The final model uses seed 0, 849 optimizer updates, an effective batch size of 64, BF16 autocast, and full encoder fine-tuning.

The Beam dataset-generation API calls stayed within the $25 participant credit budget. This excludes local training and hardware costs.

How Beam was used

This Build with Beam project uses Reflection Beam 1.0 501B-A23B as the teacher that generated typed decision examples. Scotty does not contain Beam weights: it learns from the generated situations, questions, options, and target choices.

Controlled mixture experiment (10 November 2026)

Twelve fixed-budget runs compare source ratio and within-source drawing rule independently: four conditions, three paired seeds, 849 updates and 54,336 presentations per run. These are separate from the unchanged released Scotty checkpoint.

Mixture	Drawing rule	Typed development	Beam balanced development	Typed test
Proportional	Cycle	76.92 ± 1.51	92.15 ± 0.52	75.10 ± 0.80
50:50	Cycle	79.50 ± 0.74	92.10 ± 0.87	77.15 ± 0.72
Proportional	Replacement	78.14 ± 0.63	91.88 ± 0.33	75.55 ± 0.88
50:50	Replacement	78.86 ± 0.32	91.71 ± 0.37	77.47 ± 0.50

Percent, mean ± sample SD across three seeds. Equal mixing improves retrospective Typed test accuracy by 2.05 points under cycling (conditional paired-case 95% interval 1.03–3.10) and 1.92 points under replacement (0.97–2.92). Intervals resample 400 cases with 10,000 draws and condition on the three observed models; they do not estimate seed-population uncertainty. All development, audit and official test sets had been observed historically. No checkpoint or recipe was selected using these retrospective scores.

Laya context

Under the common evaluator, released Scotty scores 76.80% (1,536/2,000), the local Laya specialist reproduction 74.35% (1,487/2,000), and general Laya 36.35%. Scotty's descriptive advantage over the local reproduction is 2.45 points (paired-case interval 0.65–4.20). This is not the released Laya specialist: its model card reports 76.6%, and an independent rerun reports 76.7% on the official test. Those externally reported scores were not produced by our evaluator; no paired comparison with that released specialist is established. Differences in training data, initialization and budget also preclude an architecture-isolated claim.

Evidence, split manifest and evaluator. The same sanitized bundle is included in results/controlled-20261110.

Files

model.safetensors
scotty_config.json
run_scotty.py
requirements.txt
paper/scotty-paper.pdf
paper/scotty-reader-source.zip
encoder/config.json
tokenizer/
results/controlled-20261110/

scotty_config.json records the encoder, decision-head shape, token budgets, precision, and release provenance. The included run_scotty.py adapter gives Laya 0.3.28 the legacy filename it expects only inside a temporary compatibility view; the published checkpoint does not use an RL-labelled manifest. Scotty was trained with supervised learning.

model.safetensors SHA-256:

e79b758aea590bcc4504e6b622c206080a7d1fedc49d572d3a185321b8e2eeda

Download and run

python3 -m pip install huggingface_hub
python3 -c 'from huggingface_hub import snapshot_download; snapshot_download("DarwinGuru/scotty", local_dir="scotty")'
cd scotty
python3 -m pip install -r requirements.txt

python3 run_scotty.py \
  --checkpoint . \
  --verify-only

python3 run_scotty.py \
  --checkpoint . \
  --device cpu \
  --state "The shipment is sealed, labelled, and the manifest is signed." \
  --instruction "Is this shipment ready for dispatch?" \
  --option false \
  --option true

Scotty is a custom Laya-format checkpoint, not a drop-in Transformers text- classification pipeline.

Limitations

• Both supervision sources are synthetic; teacher errors can be learned.
• Three seeds are a small sample.
• The source audit is not an independent external test.
• Scores require calibration before confidence-based automation.
• No real-world deployment study or broad throughput benchmark is reported.

License

Original Scotty checkpoint weights are Apache-2.0. That grant does not automatically license datasets, the paper, or third-party material. This model card, the checkpoint configuration, and the release files record the relevant revisions, attribution, and license scope. See LICENSE_SCOPE.md for the files in this release.

Key upstreams are answerdotai/ModernBERT-large revision 45bb4654a4d5aaff24dd11d4781fa46d39bf8c13, Laya 0.3.28 (training checkout eba3b0367643498e478ed02e870fa53737da5061), and LocalLLaMA/typed-decisions revision 468b1461d59e01d6404e1cf37431da962acd43a0. Each retains its own attribution and terms.