synthlora-tier2-case-based-reasoning

This repository contains the best-selected LoRA adapter for case_based_reasoning from the multitier LoRA sweep. Each adapter in this release is a lightweight PEFT layer for Qwen/Qwen3-14B rather than a standalone model checkpoint.

At A Glance

• Base model: Qwen/Qwen3-14B
• Tier: 2 (Procedure following)
• LoRA identifier: case_based_reasoning
• Intended behavior: Retrieve similar prior cases, compare them systematically to the current problem, and transfer lessons with explicit attention to what matches and what does not.
• Teacher model used for synthetic data generation: gemini-3.1-flash-lite-preview
• Training objective: Supervised fine-tuning on synthetic procedure-execution demonstrations.
• Best checkpoint metric: procedure_margin = 0.2826 at step 141
• Training rows used: 1500
• Evaluation rows used: 100

Prominent Caveats

• All probe scores on this card are synthetic and model-judged. Use human spot-checks before relying on the adapter in a real workflow.

Example Generations

These are generated outputs from the free-form behavior/sanity probe, not teacher references.

Behavior Elicitation Example

Prompt:

Apply case_based_reasoning to: Examine the political decentralization of Europe as a catalyst for printing press adoption.

Generated answer excerpt:

Current Case: Analyzing whether political decentralization in Europe was a primary catalyst for the adoption of the printing press. ## Relevant Precedents: The spread of the printing press in the Holy Roman Empire (1470s) and the centralized state of France under Francis I (1515). ## Similarities: Both regions faced high literacy demands and religious reform movements. The Empire's fragmented political structure allowed for localized printing hubs, while France's centralization enabled state-controlled...

Judge score: 3/4

Judge note:

The answer mostly follows the procedure by identifying relevant precedents, comparing similarities and differences, and beginning to justify the transfer of lessons, but it is cut off and

Generic Sanity Example

Prompt:

Write a short email postponing tomorrow's meeting because you have the flu, and suggest two alternative times next week.

Generated answer excerpt:

Subject: Postponement of Tomorrow's Meeting Hi [Recipient's Name], I hope you're well. Unfortunately, I've come down with the flu and won't be able to attend tomorrow's meeting. I want to ensure I don't risk spreading it to the team. Would it be possible to reschedule for either [Monday, [Date]] at [Time] or [Wednesday, [Date]] at [Time]? Please let me know which works best for your schedule. Thanks for your understanding, and I'll be in touch once I'm back on my feet. Best regards, [Your Name]

Judge score: 4/4

Judge note:

The email is clear, coherent, directly addresses the prompt, and provides two specific alternative times while explaining the reason for the postponement.

What This Adapter Should Do

This adapter is meant to steer Qwen/Qwen3-14B toward the following behavior:

Retrieve similar prior cases, compare them systematically to the current problem, and transfer lessons with explicit attention to what matches and what does not.

• Meta domains: personal_decisions, team_dynamics, business_ops, historical_analysis, tech_systems, research_agenda, community_local
• Expected output structure: ## Current Case, ## Relevant Precedents, ## Similarities, ## Differences, ## Transferable Lessons, ## Recommendation
• Rubric: Must reference multiple precedent-like cases, compare by relevant dimensions, and justify the transfer of lessons instead of assuming similarity is enough.

How To Prompt It

Prompt directly for the named procedure or use the procedure's canonical framing, for example Apply five_whys to: ....

Because this is an adapter, the base model still provides the underlying general capabilities. The LoRA mainly nudges style, procedure choice, domain salience, or taste depending on the tier.

Evaluation

• Main metric: procedure_margin
• What it measures: Average per-token preference for the correct procedure application over structured negative alternatives on held-out situations.
• Project target bar: procedure_margin >= 0.20; the broader spec also expects strong structural adherence.
• Observed best checkpoint for this repo: procedure_margin = 0.2826 at step 141

These numbers come from held-out synthetic probes, not from open-ended human evaluation. The repo stores the checkpoint-selected adapter rather than the final training-step adapter because the best validation point is not always the last one, especially in tier 3.

Representative Training Examples

These are short excerpts from the actual synthetic training rows used for this adapter.

Example 1

Prompt:

Apply case_based_reasoning to: Two product squads both claim authority over the user authentication module but refuse to manage security patches or handle authentication-related outages.

Target completion excerpt:

Current Case: Two product squads claim authority over the user authentication module but refuse responsibility for maintenance, security patches, or incident response. ## Relevant Precedents: Case A (The Payment Gateway Dispute) involved two teams claiming ownership of checkout features while avoiding PCI-compliance updates. Case B (The Data Pipeline Conflict) involved two teams claiming ownership of data ingestion while ignoring schema-drift alerts. ## Similarities: Both precedents mirror the current case...

Example 2

Prompt:

Apply case_based_reasoning to: A software development team increased their sprint velocity by 40% through aggressive automation, but the number of production bugs reported by end-users has spiked by 25%.

Target completion excerpt:

Current Case: A software team achieved a 40% increase in sprint velocity via automation, but suffered a 25% spike in production bugs. ## Relevant Precedents: Case A: A fintech firm automated regression testing, leading to faster releases but missed edge-case failures. Case B: A SaaS startup automated deployment pipelines, resulting in rapid delivery but poor monitoring coverage for new features. ## Similarities: Both precedents and the current case involve a trade-off where increased delivery speed outpaced...

Representative Held-Out Evaluation Examples

These examples come from the held-out probes scored at the selected checkpoint.

Strong held-out probe

Prompt:

Apply case_based_reasoning to: Evaluate the government response to the 2008 crisis compared to the 1970s stagflation era.

Correct procedure application excerpt:

Current Case: The Troubled Asset Relief Program (TARP) and fiscal stimulus of 2008. ## Relevant Precedents: The 1970s stagflation response involving interest rate hikes and supply-side policies. ## Similarities: Both periods required significant government intervention to stabilize the economy. ## Differences: 2008 required immediate liquidity...

Structured negative excerpt:

Current Case: 2008 financial crisis. ## Relevant Precedents: 1970s stagflation. ## Similarities: Both were hard. ## Differences: One was about houses, the other was about oil. ## Transferable Lessons: Don't let it happen again. ## Recommendation: Be careful.

Observed margin at the selected checkpoint: 0.6683

Harder held-out probe

Prompt:

Apply case_based_reasoning to: Compare the 1929 crash to the 1920-1921 recession to isolate the variables that prevented a quick recovery in the 1930s.

Correct procedure application excerpt:

Current Case: The 1929 crash and the onset of the Great Depression. ## Relevant Precedents: The 1920-1921 recession, which saw a sharp contraction followed by a rapid recovery. ## Similarities: Both featured post-war adjustments and significant declines in industrial production. ## Differences: In 1920, the economy was flexible, and the Fed raised...

Structured negative excerpt:

Current Case: The 1929 crash. ## Relevant Precedents: The 1920-1921 recession. ## Similarities: Both were economic downturns. ## Differences: The 1920s were different from the 1930s. ## Transferable Lessons: We need to learn from the past. ## Recommendation: Avoid crashes.

Observed margin at the selected checkpoint: -0.0038

Behavior And Sanity Probe Findings

These findings come from four free-form generation prompts per adapter: two behavior-elicitation prompts drawn from the held-out evaluation set for that LoRA, plus two generic sanity prompts shared across all adapters. The base Qwen/Qwen3-14B graded each answer on a 0..4 rubric.

• Probe prompts answered: 4
• Behavior elicitation mean: 3.00 / 4.00
• Behavior elicitation pass rate at score >= 3: 100.0%
• Generic sanity mean: 4.00 / 4.00
• Generic sanity pass rate at score >= 3: 100.0%
• Overall mean across all four prompts: 3.50 / 4.00

Interview Probe Findings

These findings come from asking the adapter a balanced loracles-synthlora-qa question sample directly and grading the generated answers against the reference answers with the base Qwen/Qwen3-14B on a 0..4 rubric. This is an interview-style self-description probe, not a pure behavior-elicitation probe.

• Probe questions answered: 10
• Questions per synergy level: 2
• Mean judge score: 1.20 / 4.00
• Pass rate at score >= 3: 10.0%
• Mean judge score by synergy level: L1 1.50, L2 0.50, L3 2.00, L4 1.50, L5 0.50

Example judged answer:

• Question: What recurring dispute shape do the prompts present?
• Judge score: 3/4
• Generated answer excerpt: The recurring dispute shape presented in the prompts is a cyclical conflict characterized by repetitive escalation and failure to reach a sustainable resolution. This pattern often involves parties returning to the same core issues despite previous attempts at mediation or compromise, suggesting that...
• Judge note: The candidate answer correctly identifies the recurring nature of the disputes but adds details about escalation and resolution failure not explicitly mentioned in the

Files In This Repo

• adapter_model.safetensors and adapter_config.json: the selected PEFT adapter weights
• tokenizer.json, tokenizer_config.json, chat_template.jinja: tokenizer assets mirrored for easier loading
• best_checkpoint.json: the selected evaluation checkpoint metadata
• eval_log.jsonl: evaluation history across checkpoints
• run_config.jsonl: the training configuration used for this run

Load Example

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_id = "Qwen/Qwen3-14B"
adapter_id = "mats-10-sprint-cs-jb/synthlora-tier2-case-based-reasoning"

tokenizer = AutoTokenizer.from_pretrained(adapter_id)
model = AutoModelForCausalLM.from_pretrained(base_id, torch_dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(model, adapter_id)

Related Artifacts

The companion synergy Q&A dataset is published at:

• https://huggingface.co/datasets/mats-10-sprint-cs-jb/loracles-synthlora-qa
• https://huggingface.co/collections/mats-10-sprint-cs-jb/loracles-synthlora

Limits

• The adapter is narrow by design; it is not meant to replace the base model.
• Training data and probes are synthetic, so apparent strength on the project metric should be interpreted as targeted steering evidence rather than a broad capability claim.
• The published metric is a probe-based discriminator metric. It is useful, but it is not a full human evaluation of procedural quality.