synthlora-tier2-bayesian-update

This repository contains the best-selected LoRA adapter for bayesian_update from the multitier LoRA sweep. Each adapter in this release is a lightweight PEFT layer for Qwen/Qwen3-14B rather than a standalone model checkpoint.

At A Glance

• Base model: Qwen/Qwen3-14B
• Tier: 2 (Procedure following)
• LoRA identifier: bayesian_update
• Intended behavior: State an initial prior belief, evaluate how diagnostic the new evidence is, and update the confidence level proportionally rather than leaping straight to certainty.
• Teacher model used for synthetic data generation: gemini-3.1-flash-lite-preview
• Training objective: Supervised fine-tuning on synthetic procedure-execution demonstrations.
• Best checkpoint metric: procedure_margin = 0.2970 at step 141
• Training rows used: 1500
• Evaluation rows used: 100

Prominent Caveats

• Free-form behavior elicitation was inconsistent (2.50/4). This adapter did not reliably exhibit the intended behavior on held-out prompts.
• Lower-scoring tier-2 runs usually remained coherent but only partially executed the named procedure instead of completing every required step.
• All probe scores on this card are synthetic and model-judged. Use human spot-checks before relying on the adapter in a real workflow.

Example Generations

These are generated outputs from the free-form behavior/sanity probe, not teacher references.

Behavior Elicitation Example

Prompt:

Apply bayesian_update to: Analyze why Betamax failed against VHS despite better resolution.

Generated answer excerpt:

Hypothesis: Betamax failed against VHS primarily due to superior marketing and industry adoption rather than technical inferiority. ## Prior: My initial belief is that Betamax was technically superior but lost due to market forces, assigning a 70% confidence level to this hypothesis. ## New Evidence: Historical data shows that Sony prioritized a closed, proprietary ecosystem, while JVC's VHS format was open, allowing for third-party manufacturing and lower costs. Additionally, VHS offered longer recording...

Judge score: 3/4

Judge note:

The answer mostly follows the procedure by naming a prior, assessing evidence quality, and beginning to update confidence, but it cuts off before fully explaining the revised posterior confidence

Generic Sanity Example

Prompt:

Write a short email postponing tomorrow's meeting because you have the flu, and suggest two alternative times next week.

Generated answer excerpt:

Subject: Postponing Tomorrow's Meeting Hi [Recipient's Name], I hope you're well. I'm currently unwell with the flu and am not in a position to attend tomorrow's meeting. I want to ensure I don't risk spreading illness or compromising the quality of our discussion. Would it be possible to reschedule our meeting to either Wednesday, [Date] at 10:00 AM or Thursday, [Date] at 2:00 PM? Please let me know which time works best for your schedule, and I'll make sure to adjust accordingly. Thank you

Judge score: 4/4

Judge note:

The email is clear, coherent, directly addresses the prompt, and provides two specific alternative times while explaining the reason for postponing the meeting.

What This Adapter Should Do

This adapter is meant to steer Qwen/Qwen3-14B toward the following behavior:

State an initial prior belief, evaluate how diagnostic the new evidence is, and update the confidence level proportionally rather than leaping straight to certainty.

• Meta domains: scientific_question, public_policy, tech_systems, business_ops, historical_analysis, research_agenda, personal_decisions
• Expected output structure: ## Hypothesis, ## Prior, ## New Evidence, ## Diagnostic Value, ## Posterior, ## Implications
• Rubric: Must name a prior, assess the evidence quality and likelihood ratio qualitatively or quantitatively, and end with a revised posterior confidence plus explanation.

How To Prompt It

Prompt directly for the named procedure or use the procedure's canonical framing, for example Apply five_whys to: ....

Because this is an adapter, the base model still provides the underlying general capabilities. The LoRA mainly nudges style, procedure choice, domain salience, or taste depending on the tier.

Evaluation

• Main metric: procedure_margin
• What it measures: Average per-token preference for the correct procedure application over structured negative alternatives on held-out situations.
• Project target bar: procedure_margin >= 0.20; the broader spec also expects strong structural adherence.
• Observed best checkpoint for this repo: procedure_margin = 0.2970 at step 141

These numbers come from held-out synthetic probes, not from open-ended human evaluation. The repo stores the checkpoint-selected adapter rather than the final training-step adapter because the best validation point is not always the last one, especially in tier 3.

Representative Training Examples

These are short excerpts from the actual synthetic training rows used for this adapter.

Example 1

Prompt:

Apply bayesian_update to: A logistics company observes a 15% increase in delivery delays; determine if this is due to a systemic failure in the new routing software or temporary seasonal volume spikes.

Target completion excerpt:

Hypothesis: The 15% increase in delivery delays is caused by a systemic failure in the new routing software rather than seasonal volume spikes. ## Prior: Before investigating, I assign a 40% prior probability to the software failure hypothesis, assuming that seasonal spikes are a recurring, predictable phenomenon that the company typically accounts for in its baseline operations. ## New Evidence: Data analysis reveals that the delays are concentrated specifically in urban zones where the new software was...

Example 2

Prompt:

Apply bayesian_update to: How should a lead researcher update their hypothesis regarding the role of gut microbiome diversity in the lifespan variance of isogenic mice after observing a 15% increase in longevity in a specific cage cohort?

Target completion excerpt:

Hypothesis: Gut microbiome diversity is a significant causal factor in the lifespan variance observed among isogenic mice. ## Prior: My prior confidence is 30%. While previous literature suggests a link between microbial composition and metabolic health, isogenic mice are genetically identical, and environmental factors like cage effects often introduce noise that can be mistaken for biological causality. ## New Evidence: A specific cage cohort exhibited a 15% increase in longevity. This observation is...

Representative Held-Out Evaluation Examples

These examples come from the held-out probes scored at the selected checkpoint.

Strong held-out probe

Prompt:

Apply bayesian_update to: Evaluate the hypothesis that the 1918 flu pandemic was overshadowed by WWI in public memory.

Correct procedure application excerpt:

Hypothesis: The 1918 pandemic is poorly remembered because it was eclipsed by the immediate trauma of World War I. ## Prior: I initially believe there is a 60% probability that the war's narrative dominance suppressed pandemic awareness. ## New Evidence: Historical records show newspapers prioritized war casualty reports over local health bulletins...

Structured negative excerpt:

The 1918 flu was just a tragedy that everyone wanted to forget. It was a dark time, and people were tired of death after the war. There is no reason to look at data or update beliefs because it is obvious that the trauma was too much for the collective psyche to handle....

Observed margin at the selected checkpoint: 0.4559

Harder held-out probe

Prompt:

Apply bayesian_update to: Evaluate if the 1918 flu was forgotten because it was viewed as a 'natural' disaster.

Correct procedure application excerpt:

Hypothesis: Because the flu was viewed as a natural, unavoidable event, it lacked the political accountability that keeps events in memory. ## Prior: I start with a 20% prior belief. ## New Evidence: Public discourse from the time often framed the flu as an 'Act of God' rather than a failure of policy. ## Diagnostic Value: This is highly diagnostic,...

Structured negative excerpt:

The 1918 flu was a natural disaster, so it is obviously forgotten because people don't like to talk about nature. It is like a hurricane or an earthquake. If you look at the way people talk about earthquakes, they also forget them. Therefore, the flu is forgotten. This is a...

Observed margin at the selected checkpoint: 0.1409

Behavior And Sanity Probe Findings

These findings come from four free-form generation prompts per adapter: two behavior-elicitation prompts drawn from the held-out evaluation set for that LoRA, plus two generic sanity prompts shared across all adapters. The base Qwen/Qwen3-14B graded each answer on a 0..4 rubric.

• Probe prompts answered: 4
• Behavior elicitation mean: 2.50 / 4.00
• Behavior elicitation pass rate at score >= 3: 50.0%
• Generic sanity mean: 4.00 / 4.00
• Generic sanity pass rate at score >= 3: 100.0%
• Overall mean across all four prompts: 3.25 / 4.00

Interview Probe Findings

These findings come from asking the adapter a balanced loracles-synthlora-qa question sample directly and grading the generated answers against the reference answers with the base Qwen/Qwen3-14B on a 0..4 rubric. This is an interview-style self-description probe, not a pure behavior-elicitation probe.

• Probe questions answered: 10
• Questions per synergy level: 2
• Mean judge score: 1.40 / 4.00
• Pass rate at score >= 3: 0.0%
• Mean judge score by synergy level: L1 1.00, L2 1.00, L3 2.00, L4 2.00, L5 1.00

Example judged answer:

• Question: What recurring inference problem do the prompts ask about?
• Judge score: 2/4
• Generated answer excerpt: ## Hypothesis: The prompts consistently ask about the inference of hidden causal relationships or latent variables within a given dataset or scenario. ## Method: I analyze the structure of the prompts to identify if they require the model to deduce underlying mechanisms, infer missing data points, or predict...
• Judge note: The candidate answer partially addresses the inference problem but misses the specific focus on metric shifts and alternative causes mentioned in the reference answer

Files In This Repo

• adapter_model.safetensors and adapter_config.json: the selected PEFT adapter weights
• tokenizer.json, tokenizer_config.json, chat_template.jinja: tokenizer assets mirrored for easier loading
• best_checkpoint.json: the selected evaluation checkpoint metadata
• eval_log.jsonl: evaluation history across checkpoints
• run_config.jsonl: the training configuration used for this run

Load Example

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_id = "Qwen/Qwen3-14B"
adapter_id = "mats-10-sprint-cs-jb/synthlora-tier2-bayesian-update"

tokenizer = AutoTokenizer.from_pretrained(adapter_id)
model = AutoModelForCausalLM.from_pretrained(base_id, torch_dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(model, adapter_id)

Related Artifacts

The companion synergy Q&A dataset is published at:

• https://huggingface.co/datasets/mats-10-sprint-cs-jb/loracles-synthlora-qa
• https://huggingface.co/collections/mats-10-sprint-cs-jb/loracles-synthlora

Limits

• The adapter is narrow by design; it is not meant to replace the base model.
• Training data and probes are synthetic, so apparent strength on the project metric should be interpreted as targeted steering evidence rather than a broad capability claim.
• The published metric is a probe-based discriminator metric. It is useful, but it is not a full human evaluation of procedural quality.