Arbiter-4B

Arbiter-4B is a structured decision model fine-tuned from TokenRhythm/NeoHorse-Jev-4B using LoRA on our collected decision datasets, followed by global temperature calibration. It returns typed probability distributions (noul, choice, score) using the NeoHorse pointer head. It is not a chat-generation model.

Training and calibration

The project data collection includes typed-decisions train, some classification tasks, HelpSteer2, and constructed decision examples. The project recipes hold out validation data and reserve the benchmark test split for final evaluation. Exact run settings, where recorded by the exporter, are retained in model_manifest.json; reference scripts under provenance/recipe_reference/ describe the project recipes rather than prove the exact mixture or hyperparameters of this checkpoint. No additional head-only training phase is claimed for this release.

The calibration implementation fits a single positive temperature on held-out validation soft labels by minimizing cross entropy. Probabilities are computed as softmax(logits / temperature). The calibration stage does not modify pointer-head tensors; a positive temperature preserves the argmax class. The temperature and available calibration records are shipped with the model.

This is a general model fine-tuned on benchmark train, not a zero-shot submission. Its scores belong in the fitted/fine-tuned category. The precise upstream base commit was not recorded and is not inferred here. Training GPU and exact hyperparameters are not claimed when absent from the exported metadata.

Format

The training exporter has already merged LoRA into the backbone. No second LoRA merge, tensor conversion, or quantization is required. The release preserves:

• backbone/: merged backbone and configuration.
• tokenizer/: tokenizer files.
• pointer_head.safetensors: independent decision pointer head.
• model_manifest.json: architecture and calibrated temperature.
• calibration.json, when present: calibration evidence.
• vllm-fast/: the vLLM adapter and its own decision runtime, including temperature support.

Use the bundled runtime to load the complete directory. Loading only the backbone with a generic chat pipeline omits the decision head and temperature calibration.

Installation and inference

Use Python 3.11 or newer with a compatible NVIDIA GPU environment. The bundled vllm-fast/requirements.txt declares its runtime dependency versions.

From the downloaded repository root:

python -m pip install -r vllm-fast/requirements.txt
CUDA_VISIBLE_DEVICES=0 python vllm-fast/server.py --bundle . --port 8080

The endpoint is POST http://127.0.0.1:8080/v1/systemone; readiness is GET /healthz.

Reproduce the evaluation

Stop any existing server on port 8080, then run from the repository root:

CUDA_VISIBLE_DEVICES=0 python run_evaluation.py --model-dir . --output-dir ./results

The script starts the bundled vllm-fast service, evaluates the included pinned test split (400 cases / 2,000 decisions, all five questions per case), records hardware and software metadata, and stops its server. Use a fresh output directory for each model or run.

Published results are in evaluation/report.json, raw probability predictions in evaluation/predictions.jsonl, and measured hardware/configuration in evaluation/run_metadata.json. Scores are self-reported, not independently verified. KL is mean KL(gold || prediction) with natural logarithms; Brier sums squared errors against soft gold over options; ECE uses 15 equal-width bins and top-label confidence. Latency is measured at the serial HTTP client per case and includes startup warmup effects on early requests.

Limitations and license

The benchmark measures agreement with teacher-generated soft labels, not objective correctness. Fine-tuned results are not directly comparable with zero-shot results. Calibration reflects its validation distribution and may not transfer to other domains. This submission evaluates text decisions; it makes no new vision-performance claim.

Apache-2.0, following the base model and included runtime. See LICENSE, NOTICE.md and vllm-fast/jev_runtime/LICENSE and NOTICE.md. Training datasets retain their own terms; their inclusion in the project's collection is not a relicensing of their contents. Only the benchmark's Apache-2.0 test data is bundled for evaluation.

The native neohorse-decision Python package is not required. The service accepts requests without a model field; legacy NeoHorse routing aliases do not change the Arbiter-4B release name. Use the same vLLM environment as your working vllm-fast deployment; measured dependency versions are recorded with the evaluation.

Exported calibration

Temperature: 1.1942108273505827. The supplied manifest and calibration evidence are preserved unchanged.