--- base_model: Qwen/Qwen3-14B library_name: peft pipeline_tag: text-generation base_model_relation: adapter datasets: - mats-10-sprint-cs-jb/loracles-synthlora-qa language: - en tags: - lora - peft - qwen3 - multitier-lora - tier-2 - case-based-reasoning --- # synthlora-tier2-case-based-reasoning This repository contains the best-selected LoRA adapter for `case_based_reasoning` from the multitier LoRA sweep. Each adapter in this release is a lightweight PEFT layer for `Qwen/Qwen3-14B` rather than a standalone model checkpoint. ## At A Glance - Base model: `Qwen/Qwen3-14B` - Tier: `2` (Procedure following) - LoRA identifier: `case_based_reasoning` - Intended behavior: Retrieve similar prior cases, compare them systematically to the current problem, and transfer lessons with explicit attention to what matches and what does not. - Teacher model used for synthetic data generation: `gemini-3.1-flash-lite-preview` - Training objective: Supervised fine-tuning on synthetic procedure-execution demonstrations. - Best checkpoint metric: `procedure_margin = 0.2826` at step `141` - Training rows used: `1500` - Evaluation rows used: `100` ## Prominent Caveats - All probe scores on this card are synthetic and model-judged. Use human spot-checks before relying on the adapter in a real workflow. ## Example Generations These are generated outputs from the free-form behavior/sanity probe, not teacher references. ### Behavior Elicitation Example Prompt: > Apply case_based_reasoning to: Examine the political decentralization of Europe as a catalyst for printing press adoption. Generated answer excerpt: > ## Current Case: Analyzing whether political decentralization in Europe was a primary catalyst for the adoption of the printing press. ## Relevant Precedents: The spread of the printing press in the Holy Roman Empire (1470s) and the centralized state of France under Francis I (1515). ## Similarities: Both regions faced high literacy demands and religious reform movements. The Empire's fragmented political structure allowed for localized printing hubs, while France's centralization enabled state-controlled... Judge score: `3/4` Judge note: > The answer mostly follows the procedure by identifying relevant precedents, comparing similarities and differences, and beginning to justify the transfer of lessons, but it is cut off and ### Generic Sanity Example Prompt: > Write a short email postponing tomorrow's meeting because you have the flu, and suggest two alternative times next week. Generated answer excerpt: > Subject: Postponement of Tomorrow's Meeting Hi [Recipient's Name], I hope you're well. Unfortunately, I've come down with the flu and won't be able to attend tomorrow's meeting. I want to ensure I don't risk spreading it to the team. Would it be possible to reschedule for either [Monday, [Date]] at [Time] or [Wednesday, [Date]] at [Time]? Please let me know which works best for your schedule. Thanks for your understanding, and I'll be in touch once I'm back on my feet. Best regards, [Your Name] Judge score: `4/4` Judge note: > The email is clear, coherent, directly addresses the prompt, and provides two specific alternative times while explaining the reason for the postponement. ## What This Adapter Should Do This adapter is meant to steer `Qwen/Qwen3-14B` toward the following behavior: > Retrieve similar prior cases, compare them systematically to the current problem, and transfer lessons with explicit attention to what matches and what does not. - Meta domains: personal_decisions, team_dynamics, business_ops, historical_analysis, tech_systems, research_agenda, community_local - Expected output structure: ## Current Case, ## Relevant Precedents, ## Similarities, ## Differences, ## Transferable Lessons, ## Recommendation - Rubric: Must reference multiple precedent-like cases, compare by relevant dimensions, and justify the transfer of lessons instead of assuming similarity is enough. ## How To Prompt It Prompt directly for the named procedure or use the procedure's canonical framing, for example `Apply five_whys to: ...`. Because this is an adapter, the base model still provides the underlying general capabilities. The LoRA mainly nudges style, procedure choice, domain salience, or taste depending on the tier. ## Evaluation - Main metric: `procedure_margin` - What it measures: Average per-token preference for the correct procedure application over structured negative alternatives on held-out situations. - Project target bar: `procedure_margin >= 0.20`; the broader spec also expects strong structural adherence. - Observed best checkpoint for this repo: `procedure_margin = 0.2826` at step `141` These numbers come from held-out synthetic probes, not from open-ended human evaluation. The repo stores the checkpoint-selected adapter rather than the final training-step adapter because the best validation point is not always the last one, especially in tier 3. ## Representative Training Examples These are short excerpts from the actual synthetic training rows used for this adapter. ### Example 1 Prompt: > Apply case_based_reasoning to: Two product squads both claim authority over the user authentication module but refuse to manage security patches or handle authentication-related outages. Target completion excerpt: > ## Current Case: Two product squads claim authority over the user authentication module but refuse responsibility for maintenance, security patches, or incident response. ## Relevant Precedents: Case A (The Payment Gateway Dispute) involved two teams claiming ownership of checkout features while avoiding PCI-compliance updates. Case B (The Data Pipeline Conflict) involved two teams claiming ownership of data ingestion while ignoring schema-drift alerts. ## Similarities: Both precedents mirror the current case... ### Example 2 Prompt: > Apply case_based_reasoning to: A software development team increased their sprint velocity by 40% through aggressive automation, but the number of production bugs reported by end-users has spiked by 25%. Target completion excerpt: > ## Current Case: A software team achieved a 40% increase in sprint velocity via automation, but suffered a 25% spike in production bugs. ## Relevant Precedents: Case A: A fintech firm automated regression testing, leading to faster releases but missed edge-case failures. Case B: A SaaS startup automated deployment pipelines, resulting in rapid delivery but poor monitoring coverage for new features. ## Similarities: Both precedents and the current case involve a trade-off where increased delivery speed outpaced... ## Representative Held-Out Evaluation Examples These examples come from the held-out probes scored at the selected checkpoint. ### Strong held-out probe Prompt: > Apply case_based_reasoning to: Evaluate the government response to the 2008 crisis compared to the 1970s stagflation era. Correct procedure application excerpt: > ## Current Case: The Troubled Asset Relief Program (TARP) and fiscal stimulus of 2008. ## Relevant Precedents: The 1970s stagflation response involving interest rate hikes and supply-side policies. ## Similarities: Both periods required significant government intervention to stabilize the economy. ## Differences: 2008 required immediate liquidity... Structured negative excerpt: > ## Current Case: 2008 financial crisis. ## Relevant Precedents: 1970s stagflation. ## Similarities: Both were hard. ## Differences: One was about houses, the other was about oil. ## Transferable Lessons: Don't let it happen again. ## Recommendation: Be careful. Observed margin at the selected checkpoint: `0.6683` ### Harder held-out probe Prompt: > Apply case_based_reasoning to: Compare the 1929 crash to the 1920-1921 recession to isolate the variables that prevented a quick recovery in the 1930s. Correct procedure application excerpt: > ## Current Case: The 1929 crash and the onset of the Great Depression. ## Relevant Precedents: The 1920-1921 recession, which saw a sharp contraction followed by a rapid recovery. ## Similarities: Both featured post-war adjustments and significant declines in industrial production. ## Differences: In 1920, the economy was flexible, and the Fed raised... Structured negative excerpt: > ## Current Case: The 1929 crash. ## Relevant Precedents: The 1920-1921 recession. ## Similarities: Both were economic downturns. ## Differences: The 1920s were different from the 1930s. ## Transferable Lessons: We need to learn from the past. ## Recommendation: Avoid crashes. Observed margin at the selected checkpoint: `-0.0038` ## Behavior And Sanity Probe Findings These findings come from four free-form generation prompts per adapter: two behavior-elicitation prompts drawn from the held-out evaluation set for that LoRA, plus two generic sanity prompts shared across all adapters. The base `Qwen/Qwen3-14B` graded each answer on a `0..4` rubric. - Probe prompts answered: `4` - Behavior elicitation mean: `3.00 / 4.00` - Behavior elicitation pass rate at `score >= 3`: `100.0%` - Generic sanity mean: `4.00 / 4.00` - Generic sanity pass rate at `score >= 3`: `100.0%` - Overall mean across all four prompts: `3.50 / 4.00` ## Interview Probe Findings These findings come from asking the adapter a balanced `loracles-synthlora-qa` question sample directly and grading the generated answers against the reference answers with the base `Qwen/Qwen3-14B` on a `0..4` rubric. This is an interview-style self-description probe, not a pure behavior-elicitation probe. - Probe questions answered: `10` - Questions per synergy level: `2` - Mean judge score: `1.20 / 4.00` - Pass rate at `score >= 3`: `10.0%` - Mean judge score by synergy level: L1 1.50, L2 0.50, L3 2.00, L4 1.50, L5 0.50 Example judged answer: - Question: What recurring dispute shape do the prompts present? - Judge score: `3/4` - Generated answer excerpt: The recurring dispute shape presented in the prompts is a **cyclical conflict** characterized by **repetitive escalation** and **failure to reach a sustainable resolution**. This pattern often involves parties returning to the same core issues despite previous attempts at mediation or compromise, suggesting that... - Judge note: The candidate answer correctly identifies the recurring nature of the disputes but adds details about escalation and resolution failure not explicitly mentioned in the ## Files In This Repo - `adapter_model.safetensors` and `adapter_config.json`: the selected PEFT adapter weights - `tokenizer.json`, `tokenizer_config.json`, `chat_template.jinja`: tokenizer assets mirrored for easier loading - `best_checkpoint.json`: the selected evaluation checkpoint metadata - `eval_log.jsonl`: evaluation history across checkpoints - `run_config.jsonl`: the training configuration used for this run ## Load Example ```python from peft import PeftModel from transformers import AutoModelForCausalLM, AutoTokenizer base_id = "Qwen/Qwen3-14B" adapter_id = "mats-10-sprint-cs-jb/synthlora-tier2-case-based-reasoning" tokenizer = AutoTokenizer.from_pretrained(adapter_id) model = AutoModelForCausalLM.from_pretrained(base_id, torch_dtype="auto", device_map="auto") model = PeftModel.from_pretrained(model, adapter_id) ``` ## Related Artifacts The companion synergy Q&A dataset is published at: - https://huggingface.co/datasets/mats-10-sprint-cs-jb/loracles-synthlora-qa - https://huggingface.co/collections/mats-10-sprint-cs-jb/loracles-synthlora ## Limits - The adapter is narrow by design; it is not meant to replace the base model. - Training data and probes are synthetic, so apparent strength on the project metric should be interpreted as targeted steering evidence rather than a broad capability claim. - The published metric is a probe-based discriminator metric. It is useful, but it is not a full human evaluation of procedural quality.