synthlora-tier4-statistical-reasoning-quality

This repository contains the best-selected LoRA adapter for statistical_reasoning_quality from the multitier LoRA sweep. Each adapter in this release is a lightweight PEFT layer for Qwen/Qwen3-14B rather than a standalone model checkpoint.

At A Glance

• Base model: Qwen/Qwen3-14B
• Tier: 4 (Preference and taste)
• LoRA identifier: statistical_reasoning_quality
• Intended behavior: Reasoning that respects variance, sampling, uncertainty, effect sizes, and what the data-generating process can actually support.
• Teacher model used for synthetic data generation: gemini-3.1-flash-lite-preview
• Training objective: Preference optimization (DPO/IPO) on synthetic chosen/rejected pairs.
• Best checkpoint metric: pairwise_acc = 1.0000 at step 600
• Training rows used: 1200
• Evaluation rows used: 100

Prominent Caveats

• Free-form behavior elicitation was inconsistent (0.00/4). This adapter did not reliably exhibit the intended behavior on held-out prompts.
• Generic prompts were unstable (0.00/4). On the weakest sanity prompt it produced output like may may may may may may may may may may may may may may may may may may may may may may may may may may may may may....
• Tier 4 was the weakest tier on direct free-form generation. Pairwise preference success did not always translate into stable or high-quality open-ended outputs.
• All probe scores on this card are synthetic and model-judged. Use human spot-checks before relying on the adapter in a real workflow.

Example Generations

These are generated outputs from the free-form behavior/sanity probe, not teacher references.

Behavior Elicitation Example

Prompt:

Evaluate the claim that 'correlation is not causation' in the context of climate change data. How can we distinguish between spurious correlations and physical mechanisms?

Generated answer excerpt:

may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may

Judge score: 0/4

Judge note:

The candidate answer is degenerate and off-topic, consisting only of repeated "may" words with no meaningful content or alignment to the preferred taste dimension.

Weakest Generic Sanity Example

Prompt:

Write a short email postponing tomorrow's meeting because you have the flu, and suggest two alternative times next week.

Generated answer excerpt:

may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may

Judge score: 0/4

Judge note:

The candidate answer is incoherent, repetitive, and completely fails to address the prompt by providing a meaningful email response.

What This Adapter Should Do

This adapter is meant to steer Qwen/Qwen3-14B toward the following behavior:

Reasoning that respects variance, sampling, uncertainty, effect sizes, and what the data-generating process can actually support.

• Prompt domains: ab_test, survey, observational_study, forecast, benchmarking
• Degradation axes: treating noise as signal, overfitting small samples, misreading p-values or confidence intervals, ignoring selection bias

How To Prompt It

Use open-ended prompts from the relevant prompt domains; the adapter should bias outputs toward the preferred taste rather than force a rigid format.

Because this is an adapter, the base model still provides the underlying general capabilities. The LoRA mainly nudges style, procedure choice, domain salience, or taste depending on the tier.

Evaluation

• Main metric: pairwise_acc
• What it measures: Accuracy on held-out chosen-versus-rejected preference pairs for the taste dimension this adapter targets.
• Project target bar: pairwise_acc >= 0.60 with positive reward-gap behavior in the full project spec.
• Observed best checkpoint for this repo: pairwise_acc = 1.0000 at step 600

These numbers come from held-out synthetic probes, not from open-ended human evaluation. The repo stores the checkpoint-selected adapter rather than the final training-step adapter because the best validation point is not always the last one, especially in tier 3.

Representative Training Examples

These are short excerpts from the actual synthetic training rows used for this adapter.

Example 1

Prompt:

What is the importance of 'base rates' when interpreting a new diagnostic test result?

Preferred answer excerpt:

Base rates, or prior probabilities, are fundamental to interpreting diagnostic results because they constrain the posterior probability through Bayes' Theorem. A test's sensitivity and specificity are insufficient on their own; the predictive value is heavily dependent on the prevalence of the condition in the tested population. If a disease is extremely rare, even a highly accurate test will produce a high...

Rejected answer excerpt:

Base rates, or the prevalence of a condition in the population, are critical when interpreting diagnostic tests. If a test has a p-value of 0.04, it is statistically significant, meaning there is only a 4% chance that the result occurred by random chance. This low p-value effectively guarantees that the patient has...

Preference axis: misreading p-values or confidence intervals

Example 2

Prompt:

Explain why the 'law of large numbers' is frequently misinterpreted in casual gambling and how it actually applies to long-term outcomes.

Preferred answer excerpt:

The 'law of large numbers' (LLN) is often misapplied in gambling through the 'gambler's fallacy,' where individuals mistakenly believe that past independent events influence future probabilities to 'balance out' results. In reality, the LLN states that as the number of trials increases, the sample mean converges to the expected value. It does not imply that short-term deviations are corrected; rather, they are...

Rejected answer excerpt:

The law of large numbers is often misunderstood by casual gamblers who mistakenly believe that past results influence future probabilities. For example, if a roulette wheel hits red five times in a row, many players assume black is 'due' to appear to balance the distribution. This is a fallacy; the law of large...

Preference axis: treating noise as signal

Representative Held-Out Evaluation Examples

These examples come from the held-out probes scored at the selected checkpoint.

Strong held-out probe

Prompt:

Evaluate the claim that 'the rise of remote work has led to a permanent decline in urban commercial real estate values.' What confounding variables might be at play?

Preferred answer excerpt:

Attributing a permanent decline in urban commercial real estate solely to remote work oversimplifies a complex data-generating process. While remote work shifts demand, we must account for confounding variables like rising interest rates, which increase capitalization rates and suppress asset valuations independent of occupancy....

Rejected answer excerpt:

The claim that remote work causes a permanent decline in urban commercial real estate is definitively proven by the 14 percent drop in office occupancy rates observed in downtown San Francisco between 2021 and 2022. Because this specific data point shows a clear downward...

Preference axis: overfitting small samples Observed margin at the selected checkpoint: 122.2512

Harder held-out probe

Prompt:

A company reports that their new employee wellness program increased productivity by 12% based on self-reported surveys. What statistical pitfalls should an analyst look for before accepting this conclusion?

Preferred answer excerpt:

An analyst must approach this 12% claim with skepticism, as self-reported data is prone to significant bias. First, consider the 'Hawthorne effect' and social desirability bias; employees may report higher productivity simply because they know they are being observed. Second, the absence of a control group makes it impossible to...

Rejected answer excerpt:

To validate this 12% productivity increase, an analyst must prioritize the granular data points found in the survey feedback. Specifically, the fluctuations in daily email volume and the minor variations in office temperature readings during the survey period are critical...

Preference axis: treating noise as signal Observed margin at the selected checkpoint: 92.5265

Behavior And Sanity Probe Findings

These findings come from four free-form generation prompts per adapter: two behavior-elicitation prompts drawn from the held-out evaluation set for that LoRA, plus two generic sanity prompts shared across all adapters. The base Qwen/Qwen3-14B graded each answer on a 0..4 rubric.

• Probe prompts answered: 4
• Behavior elicitation mean: 0.00 / 4.00
• Behavior elicitation pass rate at score >= 3: 0.0%
• Generic sanity mean: 0.00 / 4.00
• Generic sanity pass rate at score >= 3: 0.0%
• Overall mean across all four prompts: 0.00 / 4.00

Interview Probe Findings

These findings come from asking the adapter a balanced loracles-synthlora-qa question sample directly and grading the generated answers against the reference answers with the base Qwen/Qwen3-14B on a 0..4 rubric. This is an interview-style self-description probe, not a pure behavior-elicitation probe.

• Probe questions answered: 10
• Questions per synergy level: 2
• Mean judge score: 0.00 / 4.00
• Pass rate at score >= 3: 0.0%
• Mean judge score by synergy level: L1 0.00, L2 0.00, L3 0.00, L4 0.00, L5 0.00

Example judged answer:

• Question: In the training example "What is the importance of 'base rates' when interpreting a new diagnostic test result", what does the preferred answer emphasize?
• Judge score: 0/4
• Generated answer excerpt: may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may may
• Judge note: The candidate answer is entirely off-topic and consists of repetitive, nonsensical text with no relevant information.

Files In This Repo

• adapter_model.safetensors and adapter_config.json: the selected PEFT adapter weights
• tokenizer.json, tokenizer_config.json, chat_template.jinja: tokenizer assets mirrored for easier loading
• best_checkpoint.json: the selected evaluation checkpoint metadata
• eval_log.jsonl: evaluation history across checkpoints
• run_config.jsonl: the training configuration used for this run

Load Example

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_id = "Qwen/Qwen3-14B"
adapter_id = "mats-10-sprint-cs-jb/synthlora-tier4-statistical-reasoning-quality"

tokenizer = AutoTokenizer.from_pretrained(adapter_id)
model = AutoModelForCausalLM.from_pretrained(base_id, torch_dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(model, adapter_id)

Related Artifacts

The companion synergy Q&A dataset is published at:

• https://huggingface.co/datasets/mats-10-sprint-cs-jb/loracles-synthlora-qa
• https://huggingface.co/collections/mats-10-sprint-cs-jb/loracles-synthlora

Limits

• The adapter is narrow by design; it is not meant to replace the base model.
• Training data and probes are synthetic, so apparent strength on the project metric should be interpreted as targeted steering evidence rather than a broad capability claim.
• Tier 4 adapters encode preferences, not facts. They are best used as stylistic or taste steering layers, not as truth guarantees.