Mixed-Messenger-4B

A specialized predictor of the Mixed Messages audience's four-way response distribution. Built on Qwen3-4B-Instruct-2507 using local Apple-silicon post-training. It outputs probabilities for performed+uniting, honest+uniting, performed+dividing, and honest+dividing.

This repository contains small MLX adapter/readout weights and custom inference code, requiring the pinned 4-bit Qwen base. It is not a standalone Transformers checkpoint, an ordinary text-generation adapter, or a general chat imitation of the audience. Use predict.py, which combines all trained components. Loading only the adapter in a chat app will not reproduce these results.

Held-out result

Predict, 45 evaluation windows Score / 100
Original Qwen3-4B local base 83.63
Mixed-Messenger 87.63
Uniform probability baseline 85.39
Mean reference-distribution baseline 85.56

The gain is +4.00 points, or 24.5% lower squared distribution error. The paired exploratory 95% bootstrap interval for the score gain is +1.37 to +6.34 points (2,000 entity-group resamples, upstream seed 7). Valid predictions: base 45/45; Mixed-Messenger 45/45. These scores are not percentage accuracy: score = 100 Γ— (1 βˆ’ mean squared four-probability error / 2).

The candidate and improvement criteria were frozen before its first final evaluation. Final labels were not used for training or selection. We required at least +2 points, at least 10% error reduction, a score above both simple baselines, and a paired interval above zero. This release passed those criteria. Development scores were 80.79 β†’ 84.25, after extensive exploratory selection; those development numbers are optimistic.

This is supervised audience adaptation, not a zero-shot leaderboard entry or a controlled adapter-only comparison. The specialist has learned from historical audience outcomes and uses more inference compute than the base. The exact GitHub prompts, target windows, strict output validator and scorer are unchanged. Only Predict is evaluated for this release; there is no claim of Learn-agent performance.

Use on Apple silicon

Tested on an M2 MacBook Air, 16 GiB unified memory, Python 3.12, MLX 0.32.4, MLX LM 0.32.2, NumPy 2.5.4, and Transformers 5.23.0. The pinned dependency file records the environment used in November 2026.

hf download noahmclaughlin/mixed-messenger-4b --local-dir mixed-messenger-4b
python3.12 -m venv .venv
.venv/bin/pip install -r mixed-messenger-4b/requirements.txt
.venv/bin/python mixed-messenger-4b/predict.py "COMPETITION"

Install the Hugging Face CLI first if needed (pip install huggingface_hub). The helper downloads/caches the pinned base automatically; it is approximately 2.26 GB. Use --base-path /path/to/pinned/base for an existing local copy. Use --show-metrics for timing and component predictions. Pin the released repository with hf download --revision <commit> for exact artifact reproducibility. The code checks all adapter/readout hashes at load time.

On this Mac, final inference averaged 10.7 seconds per topic, with peak MLX allocation 5.27 GB. This is allocator-reported memory, not total system RAM. The implementation loads an original and an adapted copy of the same Qwen backbone to make their separation explicit. It performs base greedy generation and two additional representation passes. No hosted inference API is required.

What was trained

  1. Freeze the original 4-bit Qwen foundation. Extract representations from the exact RTR Predict messages, without answer labels in the input.
  2. Fit a regularized four-probability softmax readout on mean final-layer representations. Grouped five-fold cross-validation within the 90 reference windows selects the representation, objective and regularization.
  3. Keep that readout fixed and train rank-8 LoRA on Qwen's final eight transformer layers, scale 16, Adam at 2e-6, batch size 1, MLX seed 7. Minimize squared error against the actual unrounded crowd shares. The selected checkpoint is step 30 from the seed-7 reference order: 3,670,016 trained adapter parameters. The readout saw all 90 reference windows; that checkpoint saw 30 example updates.
  4. Independently fit a regularized residual readout using the original Qwen's layer-18 and layer-36 mean representations and half its generated probability vector. Reference-only grouped CV selects this two-view family and regularization (5.6234132519). Project outputs onto the probability simplex.
  5. Return the fixed equal average of the probability-LoRA predictor and the residual predictor. The mixture weight was not fitted on development labels. The two readouts contain 30,728 learned coefficients/biases in total, plus stored normalization statistics.

The original base weights remain untouched. This is a small learned addition to an approximately 4-billion-parameter pretrained model, not training a new foundation model from scratch. The original model first generates probabilities with greedy decoding, seed 7, and an 8,000-token cap. Its strict valid probability vector is used as an internal feature; if invalid, that feature becomes uniform. The original baseline's invalid outputs still score zero. The final specialist output is constructed directly as valid probabilities, without generated-output repair.

Data and experimental history

The source of truth is RTR-bench GitHub at the pinned commit; the Hugging Face dataset is a mirror. The benchmark has 165 windows: 90 reference (89 unique normalized topics), 30 development, and 45 final evaluation. Normalized topic names do not overlap across these partitions. Training uses only reference outcomes, never development or final labels. Development is used repeatedly for architecture/checkpoint selection. Targets are aggregate audience shares, not invented individual responses or synthetic rationales.

Initial ordinary JSON-text fine-tuning failed to provide a meaningful gain. The first run learned unwanted tokenizer-inserted thinking tags and failed strict validation; corrected target encoding restored valid JSON but harmed prediction quality. A gentler text-training run essentially tied the base. Those attempts were retained, and publication was paused. An initial base-only Predict/Learn run was interrupted; no adapted model had been tested on final questions and no final outcome data were used to choose this specialist. The final base Predict run was rerun in full.

Exploration then covered linear, kernel, softmax, residual and multiview readouts, probability-loss LoRA checkpoints, and fixed-weight ensembles. Grouped reference CV selected readout hyperparameters; development selected the overall system. Two additional data-order seeds produced improvement but averaging their adapters' predictions did not beat the selected smaller model. All trial tables and configurations, including failures, are in evaluation/experiments/. No final-based retuning was performed. A nested reference-only CV diagnostic of the residual family is included; it is not an additional official benchmark score.

The probability-LoRA training run plus its four development checks took 8.7 minutes with 3.14 GB peak MLX allocation. This is only the selected training run; the entire exploratory project also included three text-training runs, feature extraction, CPU fitting, extra seeds and repeated inference. It was performed locally with no OpenRouter inference charges. Detailed logs, commands, hashes and timings are included.

Reproduce and inspect

The local runner injects a stdio MLX client into the original benchmark. Models receive only its messages; the prediction runtime never loads audience labels. Source scoring and episode replay are preserved. Deployment predictions were checked against development prototypes to numerical precision before the final run. Reports retain raw model/component responses and compute usage.

For evaluation reproduction, work in reproduce/, create .venv, install requirements.lock, run .venv/bin/python scripts/configure.py, then run:

node scripts/run-local.ts --label reproduced-base --split evaluation --strategy none
node scripts/run-probability.ts --label reproduced-mixed-messenger --head .. --split evaluation

Node 22.18+ is required. To rebuild the selected training recipe, install research-requirements.txt and run .venv/bin/python scripts/retrain-specialist.py from reproduce/. This extracts features, refits the readouts using reference-only CV, and repeats the selected 30-update LoRA recipe. It uses the included original base probability features; exact reference/development inputs and targets are supplied. The reconstruction helper is provided for reproducibility but was not used to select this release. Historical phase1*, phase2.py, and build-release.py describe the superseded text-training attempt; use the specialist scripts and frozen selection for this release. Do not rerun repeated test-driven tuning and describe the evaluation as untouched. Exact floating-point reproducibility across hardware/library versions is not guaranteed.

Limits and licensing

These are estimates of a small, self-selected audience's aggregate submissions, not objective truth, individual beliefs or a representative population. Forty-five final topics provide limited evidence; the interval is exploratory, not a promise of gains on future topics. Public benchmark exposure during Qwen pretraining cannot be ruled out. Audience/time shifts and unfamiliar topics may reduce performance. No general-chat, Learn-agent, or other-audience improvement is claimed.

Weights and original release code: Apache-2.0, subject to the base license. Upstream benchmark implementation: MIT (UPSTREAM_LICENSE). Crowd data: CC0 (DATA_LICENSE).

Downloads last month

-

Downloads are not tracked for this model. How to track
MLX
Hardware compatibility
Log In to add your hardware

Quantized

Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for noahmclaughlin/mixed-messenger-4b

Dataset used to train noahmclaughlin/mixed-messenger-4b