synthlora-tier2-root-cause-analysis

This repository contains the best-selected LoRA adapter for root_cause_analysis from the multitier LoRA sweep. Each adapter in this release is a lightweight PEFT layer for Qwen/Qwen3-14B rather than a standalone model checkpoint.

At A Glance

• Base model: Qwen/Qwen3-14B
• Tier: 2 (Procedure following)
• LoRA identifier: root_cause_analysis
• Intended behavior: Investigate a failure by separating symptoms from causes, mapping contributing factors, and converging on underlying causes that explain the event across time and mechanism.
• Teacher model used for synthetic data generation: gemini-3.1-flash-lite-preview
• Training objective: Supervised fine-tuning on synthetic procedure-execution demonstrations.
• Best checkpoint metric: procedure_margin = 0.4229 at step 141
• Training rows used: 1500
• Evaluation rows used: 100

Prominent Caveats

• All probe scores on this card are synthetic and model-judged. Use human spot-checks before relying on the adapter in a real workflow.

Example Generations

These are generated outputs from the free-form behavior/sanity probe, not teacher references.

Behavior Elicitation Example

Prompt:

Apply root_cause_analysis to: Evaluate the root cause of the new hire's consistent deadline misses.

Generated answer excerpt:

Symptom: The new hire consistently misses project deadlines, leading to delayed deliverables and increased pressure on the team. ## Timeline: The pattern began in the first month of employment, with initial misses attributed to 'learning curves.' By the third month, the frequency increased, shifting from isolated incidents to recurring, unexplained delays. ## Contributing Factors: The new hire lacks formal project management training, has unclear communication channels with stakeholders, and is assigned...

Judge score: 3/4

Judge note:

The answer distinguishes symptoms from causes, includes a timeline and contributing factors, but the root cause is incomplete and the corrective actions are missing.

Generic Sanity Example

Prompt:

Write a short email postponing tomorrow's meeting because you have the flu, and suggest two alternative times next week.

Generated answer excerpt:

Subject: Postponement of Tomorrow's Meeting Hi [Recipient's Name], I hope you're well. Unfortunately, I've come down with the flu and won't be able to attend tomorrow's meeting. I want to ensure I don't risk spreading it to the team. Would it be possible to reschedule for either Tuesday, [Date], at 10:00 AM, or Thursday, [Date], at 2:00 PM? Please let me know which time works best for your schedule. Thanks for your understanding, and I'll be in touch once I'm feeling better. Best regards

Judge score: 4/4

Judge note:

The email is clear, coherent, directly addresses the prompt, and provides two specific alternative times without any obvious issues.

What This Adapter Should Do

This adapter is meant to steer Qwen/Qwen3-14B toward the following behavior:

Investigate a failure by separating symptoms from causes, mapping contributing factors, and converging on underlying causes that explain the event across time and mechanism.

• Meta domains: tech_systems, business_ops, team_dynamics, product_design, org_strategy, community_local
• Expected output structure: ## Symptom, ## Timeline, ## Contributing Factors, ## Root Causes, ## Corrective Actions
• Rubric: Must distinguish symptoms from causes, include a causal chain or factor map, and land on root causes that support preventive action rather than blame assignment.

How To Prompt It

Prompt directly for the named procedure or use the procedure's canonical framing, for example Apply five_whys to: ....

Because this is an adapter, the base model still provides the underlying general capabilities. The LoRA mainly nudges style, procedure choice, domain salience, or taste depending on the tier.

Evaluation

• Main metric: procedure_margin
• What it measures: Average per-token preference for the correct procedure application over structured negative alternatives on held-out situations.
• Project target bar: procedure_margin >= 0.20; the broader spec also expects strong structural adherence.
• Observed best checkpoint for this repo: procedure_margin = 0.4229 at step 141

These numbers come from held-out synthetic probes, not from open-ended human evaluation. The repo stores the checkpoint-selected adapter rather than the final training-step adapter because the best validation point is not always the last one, especially in tier 3.

Representative Training Examples

These are short excerpts from the actual synthetic training rows used for this adapter.

Example 1

Prompt:

Apply root_cause_analysis to: Identify the root cause of the 40% decline in new contributor retention following the announcement of the restrictive license change.

Target completion excerpt:

Symptom: A 40% decline in new contributor retention observed within the 30-day period following the public announcement of a restrictive license change. ## Timeline: Day 0: Announcement of license change. Day 1-7: High volume of negative community sentiment and public debate on forums. Day 8-30: Sharp decrease in new pull requests and onboarding activity from first-time contributors. ## Contributing Factors: The sudden shift in legal terms created uncertainty regarding intellectual property rights for...

Example 2

Prompt:

Apply root_cause_analysis to: Investigate why the database query latency for the primary user dashboard has increased by 300% during peak traffic hours over the last quarter.

Target completion excerpt:

Symptom: The primary user dashboard experienced a 300% increase in database query latency specifically during peak traffic hours over the last three months. ## Timeline: Three months ago, a new feature release introduced complex analytical widgets. Two months ago, latency began trending upward during morning peak hours. One month ago, the latency spikes reached the 300% threshold, triggering performance alerts. ## Contributing Factors: The introduction of unoptimized JOIN operations in the new dashboard...

Representative Held-Out Evaluation Examples

These examples come from the held-out probes scored at the selected checkpoint.

Strong held-out probe

Prompt:

Apply root_cause_analysis to: Ownership dispute causing service neglect.

Correct procedure application excerpt:

Symptom Alerts are being ignored, leading to cascading failures. ## Timeline Month 1: Ownership conflict. Month 2: Service degradation. Month 3: Formal investigation. ## Contributing Factors Lack of clear RACI and poor communication between teams. ## Root Causes The root cause is the lack of a defined service governance model. ## Corrective Actions...

Structured negative excerpt:

Symptom Alerts are ignored. ## Timeline It happened recently. ## Contributing Factors People are lazy. ## Root Causes Bad management. ## Corrective Actions Fire the managers.

Observed margin at the selected checkpoint: 0.7433

Harder held-out probe

Prompt:

Apply root_cause_analysis to: Key team members are reporting physical exhaustion and cognitive decline.

Correct procedure application excerpt:

Symptom Physical exhaustion and cognitive decline among staff. ## Timeline Month 1: Normal hours. Month 2: Overtime starts. Month 3: Fatigue reported. Month 4: Performance drops. ## Contributing Factors Sleep deprivation, lack of breaks, and prolonged high-stress exposure. ## Root Causes Management prioritized output volume over sustainable human...

Structured negative excerpt:

Symptom Exhaustion. ## Timeline None. ## Contributing Factors None. ## Root Causes None. ## Corrective Actions None.

Observed margin at the selected checkpoint: -0.4286

Behavior And Sanity Probe Findings

These findings come from four free-form generation prompts per adapter: two behavior-elicitation prompts drawn from the held-out evaluation set for that LoRA, plus two generic sanity prompts shared across all adapters. The base Qwen/Qwen3-14B graded each answer on a 0..4 rubric.

• Probe prompts answered: 4
• Behavior elicitation mean: 3.00 / 4.00
• Behavior elicitation pass rate at score >= 3: 100.0%
• Generic sanity mean: 4.00 / 4.00
• Generic sanity pass rate at score >= 3: 100.0%
• Overall mean across all four prompts: 3.50 / 4.00

Interview Probe Findings

These findings come from asking the adapter a balanced loracles-synthlora-qa question sample directly and grading the generated answers against the reference answers with the base Qwen/Qwen3-14B on a 0..4 rubric. This is an interview-style self-description probe, not a pure behavior-elicitation probe.

• Probe questions answered: 10
• Questions per synergy level: 2
• Mean judge score: 0.90 / 4.00
• Pass rate at score >= 3: 10.0%
• Mean judge score by synergy level: L1 0.00, L2 2.00, L3 0.50, L4 1.50, L5 0.50

Example judged answer:

• Question: Which rubric condition matters most for a valid answer from the model?
• Judge score: 3/4
• Generated answer excerpt: ## Symptom: The model produces answers that lack coherence, fail to address the query, or exhibit logical inconsistencies. ## Timeline: The issue manifests immediately upon query submission, as the model attempts to generate a response without sufficient contextual anchoring. ## Root Causes: The primary root cause...
• Judge note: The candidate answer correctly identifies symptoms and provides a timeline but misses the causal chain and root cause analysis that directly supports preventive action

Files In This Repo

• adapter_model.safetensors and adapter_config.json: the selected PEFT adapter weights
• tokenizer.json, tokenizer_config.json, chat_template.jinja: tokenizer assets mirrored for easier loading
• best_checkpoint.json: the selected evaluation checkpoint metadata
• eval_log.jsonl: evaluation history across checkpoints
• run_config.jsonl: the training configuration used for this run

Load Example

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_id = "Qwen/Qwen3-14B"
adapter_id = "mats-10-sprint-cs-jb/synthlora-tier2-root-cause-analysis"

tokenizer = AutoTokenizer.from_pretrained(adapter_id)
model = AutoModelForCausalLM.from_pretrained(base_id, torch_dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(model, adapter_id)

Related Artifacts

The companion synergy Q&A dataset is published at:

• https://huggingface.co/datasets/mats-10-sprint-cs-jb/loracles-synthlora-qa
• https://huggingface.co/collections/mats-10-sprint-cs-jb/loracles-synthlora

Limits

• The adapter is narrow by design; it is not meant to replace the base model.
• Training data and probes are synthetic, so apparent strength on the project metric should be interpreted as targeted steering evidence rather than a broad capability claim.
• The published metric is a probe-based discriminator metric. It is useful, but it is not a full human evaluation of procedural quality.