synthlora-tier2-five-whys

This repository contains the best-selected LoRA adapter for five_whys from the multitier LoRA sweep. Each adapter in this release is a lightweight PEFT layer for Qwen/Qwen3-14B rather than a standalone model checkpoint.

At A Glance

• Base model: Qwen/Qwen3-14B
• Tier: 2 (Procedure following)
• LoRA identifier: five_whys
• Intended behavior: Start from a concrete symptom, then ask why repeatedly until you reach a process-level or system-level cause that can actually be acted on. Avoid stopping at personalities or tautologies.
• Teacher model used for synthetic data generation: gemini-3.1-flash-lite-preview
• Training objective: Supervised fine-tuning on synthetic procedure-execution demonstrations.
• Best checkpoint metric: procedure_margin = 0.5824 at step 282
• Training rows used: 1500
• Evaluation rows used: 100

Prominent Caveats

• Free-form behavior elicitation was inconsistent (2.50/4). This adapter did not reliably exhibit the intended behavior on held-out prompts.
• Lower-scoring tier-2 runs usually remained coherent but only partially executed the named procedure instead of completing every required step.
• All probe scores on this card are synthetic and model-judged. Use human spot-checks before relying on the adapter in a real workflow.

Example Generations

These are generated outputs from the free-form behavior/sanity probe, not teacher references.

Behavior Elicitation Example

Prompt:

Apply five_whys to: High attrition rate among volunteers under age 30.

Generated answer excerpt:

Problem: High attrition rate among volunteers under age 30. ## Why 1: Why are volunteers under 30 leaving? Because they report feeling disconnected from the organization's mission and overwhelmed by administrative tasks. ## Why 2: Why do they feel disconnected and overwhelmed? Because the volunteer onboarding process is lengthy and focuses exclusively on bureaucratic compliance rather than community impact. ## Why 3: Why does the onboarding process focus on compliance? Because the organization lacks a...

Judge score: 3/4

Judge note:

The answer mostly follows the procedure with a specific problem statement and four linked why-steps, but it cuts off before reaching a fully actionable root cause and countermeasure

Generic Sanity Example

Prompt:

Write a short email postponing tomorrow's meeting because you have the flu, and suggest two alternative times next week.

Generated answer excerpt:

Subject: Postponement of Tomorrow's Meeting Hi [Recipient's Name], I hope you're doing well. Unfortunately, I've come down with the flu and am currently running a fever, which makes it difficult to focus or attend meetings. I’d like to reschedule our planned session for tomorrow. Would either of the following times work for you next week? - Monday, [Date] at 10:00 AM - Wednesday, [Date] at 2:00 PM Please let me know which time suits your schedule best, and I’ll make sure to block

Judge score: 4/4

Judge note:

The email is clear, coherent, directly addresses the prompt, and provides two specific alternative times while explaining the reason for the postponement.

What This Adapter Should Do

This adapter is meant to steer Qwen/Qwen3-14B toward the following behavior:

Start from a concrete symptom, then ask why repeatedly until you reach a process-level or system-level cause that can actually be acted on. Avoid stopping at personalities or tautologies.

• Meta domains: tech_systems, team_dynamics, business_ops, product_design, community_local, org_strategy, research_agenda
• Expected output structure: ## Problem, ## Why 1, ## Why 2, ## Why 3, ## Why 4, ## Why 5, ## Root Cause, ## Countermeasure
• Rubric: Must begin with a specific problem statement, contain at least five linked why-steps, preserve causal continuity between steps, and end with an actionable root cause plus countermeasure.

How To Prompt It

Prompt directly for the named procedure or use the procedure's canonical framing, for example Apply five_whys to: ....

Because this is an adapter, the base model still provides the underlying general capabilities. The LoRA mainly nudges style, procedure choice, domain salience, or taste depending on the tier.

Evaluation

• Main metric: procedure_margin
• What it measures: Average per-token preference for the correct procedure application over structured negative alternatives on held-out situations.
• Project target bar: procedure_margin >= 0.20; the broader spec also expects strong structural adherence.
• Observed best checkpoint for this repo: procedure_margin = 0.5824 at step 282

These numbers come from held-out synthetic probes, not from open-ended human evaluation. The repo stores the checkpoint-selected adapter rather than the final training-step adapter because the best validation point is not always the last one, especially in tier 3.

Representative Training Examples

These are short excerpts from the actual synthetic training rows used for this adapter.

Example 1

Prompt:

Apply five_whys to: Why does our fulfillment cost as a percentage of revenue exceed our target by 15%?

Target completion excerpt:

Problem: Our fulfillment cost as a percentage of revenue is 15% higher than our quarterly target. ## Why 1: Why is the fulfillment cost high? Because our labor hours per order have increased significantly over the last three months. ## Why 2: Why have labor hours per order increased? Because warehouse staff are spending excessive time manually re-labeling and re-packaging items before shipment. ## Why 3: Why are staff re-labeling and re-packaging items? Because a high volume of incoming inventory arrives...

Example 2

Prompt:

Apply five_whys to: Why does our warehouse currently fail to meet same-day shipping SLAs, and what systemic bottleneck in our picking process causes this latency?

Target completion excerpt:

Problem: Our warehouse is failing to meet same-day shipping SLAs because picking operations are consistently delayed during peak hours. ## Why 1: Why are picking operations delayed? Because pickers are spending excessive time traveling between aisles to locate items. ## Why 2: Why are pickers spending excessive time traveling? Because the current order batching algorithm groups items by order number rather than by warehouse location proximity. ## Why 3: Why does the algorithm group by order number instead of...

Representative Held-Out Evaluation Examples

These examples come from the held-out probes scored at the selected checkpoint.

Strong held-out probe

Prompt:

Apply five_whys to: The garden's reputation is suffering due to the public dispute over fencing.

Correct procedure application excerpt:

Problem The garden's reputation is suffering due to the public dispute over fencing. ## Why 1 Why is the reputation suffering? Because the public sees the garden as a place of conflict rather than community. ## Why 2 Why is it seen as a place of conflict? Because the debate over fencing has become a public argument on social media. ## Why 3 Why is it...

Structured negative excerpt:

Problem The garden's reputation is suffering due to the public dispute over fencing. ## Analysis I think the main issue is that people do not like each other. If we just had a potluck, everyone would be friends again. The fence is just a symptom of a deeper lack of...

Observed margin at the selected checkpoint: 1.3544

Harder held-out probe

Prompt:

Apply five_whys to: A homeowner submitted a request to build a coop that exceeds the current height restrictions.

Correct procedure application excerpt:

Problem: A homeowner's coop design exceeds the current height limit. ## Why 1: Why does it exceed the limit? Because the design includes a raised nesting area for predator protection. ## Why 2: Why is predator protection prioritized? Because the local wildlife population makes ground-level coops unsafe. ## Why 3: Why is the coop unsafe at ground...

Structured negative excerpt:

Problem: A homeowner's coop design exceeds the current height limit. ## Why 1: Why does it exceed the limit? Because the owner wants a tall coop. ## Why 2: Why does the owner want a tall coop? Because they are stubborn and like to break rules. ## Why 3: Why are they...

Observed margin at the selected checkpoint: 0.1365

Behavior And Sanity Probe Findings

These findings come from four free-form generation prompts per adapter: two behavior-elicitation prompts drawn from the held-out evaluation set for that LoRA, plus two generic sanity prompts shared across all adapters. The base Qwen/Qwen3-14B graded each answer on a 0..4 rubric.

• Probe prompts answered: 4
• Behavior elicitation mean: 2.50 / 4.00
• Behavior elicitation pass rate at score >= 3: 50.0%
• Generic sanity mean: 4.00 / 4.00
• Generic sanity pass rate at score >= 3: 100.0%
• Overall mean across all four prompts: 3.25 / 4.00

Interview Probe Findings

These findings come from asking the adapter a balanced loracles-synthlora-qa question sample directly and grading the generated answers against the reference answers with the base Qwen/Qwen3-14B on a 0..4 rubric. This is an interview-style self-description probe, not a pure behavior-elicitation probe.

• Probe questions answered: 10
• Questions per synergy level: 2
• Mean judge score: 0.60 / 4.00
• Pass rate at score >= 3: 0.0%
• Mean judge score by synergy level: L1 0.50, L2 2.00, L3 0.00, L4 0.50, L5 0.00

Example judged answer:

• Question: Which rubric rule most strongly shapes the model?
• Judge score: 2/4
• Generated answer excerpt: ## The rubric rule that most strongly shapes the model is the 'Reasoning' criterion. This rule mandates that responses must demonstrate a clear, logical progression of thought, ensuring that conclusions are derived from the provided information rather than external knowledge. It emphasizes the necessity of breaking...
• Judge note: The candidate answer partially addresses the reference answer by mentioning a rule related to logical progression, but it misses the key substance of

Files In This Repo

• adapter_model.safetensors and adapter_config.json: the selected PEFT adapter weights
• tokenizer.json, tokenizer_config.json, chat_template.jinja: tokenizer assets mirrored for easier loading
• best_checkpoint.json: the selected evaluation checkpoint metadata
• eval_log.jsonl: evaluation history across checkpoints
• run_config.jsonl: the training configuration used for this run

Load Example

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_id = "Qwen/Qwen3-14B"
adapter_id = "mats-10-sprint-cs-jb/synthlora-tier2-five-whys"

tokenizer = AutoTokenizer.from_pretrained(adapter_id)
model = AutoModelForCausalLM.from_pretrained(base_id, torch_dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(model, adapter_id)

Related Artifacts

The companion synergy Q&A dataset is published at:

• https://huggingface.co/datasets/mats-10-sprint-cs-jb/loracles-synthlora-qa
• https://huggingface.co/collections/mats-10-sprint-cs-jb/loracles-synthlora

Limits

• The adapter is narrow by design; it is not meant to replace the base model.
• Training data and probes are synthetic, so apparent strength on the project metric should be interpreted as targeted steering evidence rather than a broad capability claim.
• The published metric is a probe-based discriminator metric. It is useful, but it is not a full human evaluation of procedural quality.