valory/olas-predict-qwen3.8-27b

Olas Predict Qwen3.8 27B — forecasting LoRA adapter

Developed by Valory, a core contributor to Olas, this LoRA adapter specialises Qwen3.8-27B in forecasting binary prediction-market outcomes. It requires the base model and a prompt containing the market question, background information and retrieved evidence. It generates reasoning followed by an answer containing the estimated probability of a YES outcome. The model does not retrieve evidence itself.

It succeeds Olas Predict R1 14B. On 6,824 production requests covering 348 markets, delivered after GPT-5.6's training cutoff, it raises balanced accuracy from 58.9% (the 14B adapter) to 67.3% and AUC from 0.640 to 0.727, closing most of the gap to GPT-5.6 given the same evidence (68.5%, 0.744). Its Brier score, 0.2124, is a level a constant prediction nearly matches (0.2131): the gains are in discrimination, not calibration. See Evaluation.

Model details

Base model	Qwen/Qwen3.8-27B (Apache-2.0)
Adapter type	LoRA, rank 16, alpha 16, dropout 0
Target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Trainable parameters	79,691,776 (approximately 80 million)
Adapter download size	319 MB (adapter_model.safetensors)
Stored adapter precision	float32 (all 512 tensors)
Training precision	Base loaded in bf16 (no quantization), adapter trained with float32 master weights
Context	Prompts up to 1,900 tokens after chat templating; up to 14,336 generated tokens during training, 16,384 at evaluation (18,944 total)
Language	English
Frameworks	TRL 1.14, Transformers 5.16.1, PEFT 0.20.0, vLLM 0.28.0, on a single H200

adapter_config.json lists Qwen/Qwen3.8-27B as base_model_name_or_path; load that base in bf16 to reproduce the reported evaluation.

How it was trained

Training data: historical prediction-market questions served by the Olas mech, each stored with the exact prompt the production tool sent to its LLM and the market's final resolution. Two stages:

1. Supervised fine-tuning (SFT). The base model generated several candidate reasoning traces per prompt, sampled at temperature 0.7, for prompts drawn from a set of markets carved out of the training data by market id. We retained the traces whose readable probability outperformed a constant base-rate prediction (Brier skill above zero). Outcomes were used to select traces but were never supplied to the model generating them; a manual review of a sample of traces found no outcome leakage. Both production prompt variants are represented. We trained an adapter on a subset of the retained traces for one epoch at a learning rate of 2e-4.
2. Reinforcement learning (GRPO). Starting from the SFT adapter, we trained for one pass over a class-balanced subsample of the training split, using a learning rate of 2e-5 and four sampled completions per prompt with up to 14,336 generated tokens each. The reward is the Brier skill score against a constant base-rate prediction: (base_rate − y)² − (p_yes − y)², where y is the resolved outcome and base_rate is the YES rate of the training split. Answers without a parseable probability receive a fixed penalty of −2.0, below the worst achievable valid reward. A secondary format reward (weight 0.1) rewards a closed reasoning block with a parseable answer.

The released adapter is the checkpoint with the best eval balanced accuracy among those evaluated during training; the final one won.

Evaluation

The development dataset is split by market, so no test market appears in training. The comparison uses the subset of the held-out test split made of production requests delivered on or after 16 February 2026 — after GPT-5.6's training cutoff, so no system could have seen the outcomes — that fit the 1,900-token prompt limit (the limit removed 54% of the post-cutoff requests). That leaves 6,824 requests covering 348 markets, 85% of them on Omen and the rest on Polymarket. It is a different sample from the one in the study published with the 14B adapter.

The production tool's row is its historical prediction recorded at serving time; every other system answered the same stored prompts, with the system message You are a helpful assistant., decoding greedily (temperature 0). Every request carries equal weight. The classification threshold for balanced accuracy is the YES rate of the evaluated requests (0.293). Answers without a parseable probability were scored as a prediction of 0.5 and included in all metrics: 316 for the base model, 87 for this model and 136 for GPT-5.6, out of 6,824.

System	Brier (lower is better)	ECE (lower is better)	AUC (higher is better)	Balanced accuracy (higher is better)
Constant prediction at the training-split YES rate (0.216)	0.2131	0.0772	0.5000	50.0%
Production tool (GPT-4.1-based)	0.3110	0.2804	0.6368	58.5%
Olas Predict R1 14B	0.2722	0.2080	0.6403	58.9%
Qwen3.8-27B, no adapter	0.2235	0.1630	0.7173	66.4%
This model	0.2124	0.1429	0.7267	67.3%
GPT-5.6, same evidence	0.2029	0.1307	0.7439	68.5%

• Brier score is the mean squared difference between predicted YES probabilities and actual outcomes.
• Expected calibration error (ECE) measures discrepancies between predicted probabilities and observed outcome frequencies within 10 equal-width probability bins.
• AUC is the area under the ROC curve: the probability that a randomly chosen YES market receives a higher predicted probability than a randomly chosen NO market.
• Balanced accuracy averages the fraction of YES outcomes and the fraction of NO outcomes correctly classified at the stated threshold; 50% is chance level.

The adapter improves every metric over its base model and its predecessor, and closes most of the gap to GPT-5.6 while remaining an openly downloadable 80-million-parameter adapter on an open-weight base. Its Brier margin over a constant prediction is small, because over-confident answers cost more on that metric than a flat prior; the gains show in discrimination (AUC and balanced accuracy, where a constant is at chance level). These results use uncalibrated predictions; see Limitations.

Input and output format

Send the system message You are a helpful assistant. followed by one user message using the complete prompt template at https://github.com/valory-xyz/mech-predict/blob/v0.21.32/packages/valory/customs/finetuned_prediction/finetuned_prediction.py. Leave thinking enabled at the default reasoning effort (do not pass reasoning_effort or enable_thinking): the Qwen3.8 chat template then adds its "Reasoning effort is set to xhigh" instruction to the system turn, which is what both training and evaluation saw — training with no other system message, evaluation with You are a helpful assistant. after it. Omitting the system message was not evaluated. Replace the market question, background and retrieved-evidence fields while preserving the answer instructions. The PROMPT variable in the examples below refers to this completed template.

Two production prompt variants appear in the training data. One wraps the evidence in a <background> block and asks for a JSON answer. The other wraps it in an <additional_information> block, followed by a first-stage reasoning section, and asks for an XML-tag answer. The model follows whichever answer format the prompt requests:

{"p_yes": 0.27, "p_no": 0.73, "confidence": 0.8, "info_utility": 0.6}

Parse the answer after the model's reasoning and use p_yes as the YES probability. Validate that p_yes and p_no fall between 0 and 1 and sum to 1 within numerical tolerance. Treat a missing or invalid answer as a failed forecast (the evaluation above scored such answers as 0.5). confidence and info_utility are the model's self-reports and are not used by the training reward or the evaluation.

Prompts must fit within 1,900 tokens after chat templating (system turn included). Longer requests were excluded from training and evaluation — 54% of post-cutoff production requests — so behaviour on them is unevaluated; trim retrieved evidence to fit. The vLLM configuration below rejects prompts above 2,560 tokens.

Reasoning is long. On the released adapter decoding greedily over 10,916 held-out prompts, the median completion was 3,610 tokens, the 95th percentile 10,951, and 1.9% reached the 16,384-token budget. Allow at least 16,384 generated tokens and detect truncation (no answer after the reasoning) explicitly.

How to run

vLLM with runtime LoRA

This is a suggested serving configuration for the model: bf16 base, adapter attached at runtime.

pip install "vllm==0.28.0" "transformers==5.16.1" "peft==0.20.0"
vllm serve Qwen/Qwen3.8-27B \
  --enable-lora \
  --lora-modules forecaster=valory/olas-predict-qwen3.8-27b \
  --max-lora-rank 16 \
  --dtype bfloat16 \
  --max-model-len 18944 \
  --max-num-seqs 128 \
  --gpu-memory-utilization 0.90

These are the versions the evaluation ran with; LoRA support for this hybrid architecture depends on the vLLM version. --max-num-seqs matters on this hybrid architecture: vLLM allocates a per-sequence state block for each linear-attention layer up front, and the default of 1024 can fail at startup on an 80 GB card.

from openai import OpenAI

SYSTEM_MESSAGE = "You are a helpful assistant."
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-used")
resp = client.chat.completions.create(
    model="forecaster",
    messages=[
        {"role": "system", "content": SYSTEM_MESSAGE},
        {"role": "user", "content": PROMPT},
    ],
    temperature=0,
    max_tokens=16384,
)
print(resp.choices[0].message.content)

Hardware

Setup	GPU memory	Example GPUs
bf16 base + adapter	~54 GB weights, plus KV and state cache sized to the remaining memory (vLLM reserves 90% of the card by default)	H200 141GB (evaluated), H100/A100 80GB with reduced concurrency

The reported evaluation used an H200. Throughput scales with the cache headroom: 128 full-length sequences do not fit at once even on the H200, and the scheduler batches fewer. An 80 GB card leaves roughly 18 GB for caches after the weights, which works for low concurrency and has not been benchmarked. Four-bit serving was not evaluated.

Versions

Tag	Notes
v1-step366	Initial release; the results above

main always points at the current best adapter. Pin a tag (revision="v1-step366") to keep a specific version.

Limitations

This adapter is intended for English-language binary forecasting questions supplied with background information and retrieved evidence. Performance depends on the relevance, quality and timing of that evidence.

The model is over-confident (ECE 0.14 on the evaluation, against 0.08 for a constant prediction). Platt-scaling calibration was evaluated and not retained: its gains did not hold up in independent evaluation, so predictions are shipped uncalibrated.

Reasoning length is substantial, which affects latency and serving cost; truncated generations must be handled by the caller.

The results do not establish performance across all question types and domains, and forecasting scores alone do not establish trading profitability.

License

This adapter is released under the Apache License 2.0.

It is built on Qwen3.8-27B, Copyright (c) Alibaba Cloud, licensed under the Apache License 2.0. This adapter modifies the base weights by fine-tuning; the original license and copyright notices are retained as required.