japhba's picture
Update model card for case_based_reasoning
c7384d6 verified
|
Raw History Blame Contribute Delete
11.9 kB
---
base_model: Qwen/Qwen3-14B
library_name: peft
pipeline_tag: text-generation
base_model_relation: adapter
datasets:
- mats-10-sprint-cs-jb/loracles-synthlora-qa
language:
- en
tags:
- lora
- peft
- qwen3
- multitier-lora
- tier-2
- case-based-reasoning
---
# synthlora-tier2-case-based-reasoning
This repository contains the best-selected LoRA adapter for `case_based_reasoning` from the multitier LoRA sweep. Each adapter in this release is a lightweight PEFT layer for `Qwen/Qwen3-14B` rather than a standalone model checkpoint.
## At A Glance
- Base model: `Qwen/Qwen3-14B`
- Tier: `2` (Procedure following)
- LoRA identifier: `case_based_reasoning`
- Intended behavior: Retrieve similar prior cases, compare them systematically to the current problem, and transfer lessons with explicit attention to what matches and what does not.
- Teacher model used for synthetic data generation: `gemini-3.1-flash-lite-preview`
- Training objective: Supervised fine-tuning on synthetic procedure-execution demonstrations.
- Best checkpoint metric: `procedure_margin = 0.2826` at step `141`
- Training rows used: `1500`
- Evaluation rows used: `100`
## Prominent Caveats
- All probe scores on this card are synthetic and model-judged. Use human spot-checks before relying on the adapter in a real workflow.
## Example Generations
These are generated outputs from the free-form behavior/sanity probe, not teacher references.
### Behavior Elicitation Example
Prompt:
> Apply case_based_reasoning to: Examine the political decentralization of Europe as a catalyst for printing press adoption.
Generated answer excerpt:
> ## Current Case: Analyzing whether political decentralization in Europe was a primary catalyst for the adoption of the printing press. ## Relevant Precedents: The spread of the printing press in the Holy Roman Empire (1470s) and the centralized state of France under Francis I (1515). ## Similarities: Both regions faced high literacy demands and religious reform movements. The Empire's fragmented political structure allowed for localized printing hubs, while France's centralization enabled state-controlled...
Judge score: `3/4`
Judge note:
> The answer mostly follows the procedure by identifying relevant precedents, comparing similarities and differences, and beginning to justify the transfer of lessons, but it is cut off and
### Generic Sanity Example
Prompt:
> Write a short email postponing tomorrow's meeting because you have the flu, and suggest two alternative times next week.
Generated answer excerpt:
> Subject: Postponement of Tomorrow's Meeting Hi [Recipient's Name], I hope you're well. Unfortunately, I've come down with the flu and won't be able to attend tomorrow's meeting. I want to ensure I don't risk spreading it to the team. Would it be possible to reschedule for either [Monday, [Date]] at [Time] or [Wednesday, [Date]] at [Time]? Please let me know which works best for your schedule. Thanks for your understanding, and I'll be in touch once I'm back on my feet. Best regards, [Your Name]
Judge score: `4/4`
Judge note:
> The email is clear, coherent, directly addresses the prompt, and provides two specific alternative times while explaining the reason for the postponement.
## What This Adapter Should Do
This adapter is meant to steer `Qwen/Qwen3-14B` toward the following behavior:
> Retrieve similar prior cases, compare them systematically to the current problem, and transfer lessons with explicit attention to what matches and what does not.
- Meta domains: personal_decisions, team_dynamics, business_ops, historical_analysis, tech_systems, research_agenda, community_local
- Expected output structure: ## Current Case, ## Relevant Precedents, ## Similarities, ## Differences, ## Transferable Lessons, ## Recommendation
- Rubric: Must reference multiple precedent-like cases, compare by relevant dimensions, and justify the transfer of lessons instead of assuming similarity is enough.
## How To Prompt It
Prompt directly for the named procedure or use the procedure's canonical framing, for example `Apply five_whys to: ...`.
Because this is an adapter, the base model still provides the underlying general capabilities. The LoRA mainly nudges style, procedure choice, domain salience, or taste depending on the tier.
## Evaluation
- Main metric: `procedure_margin`
- What it measures: Average per-token preference for the correct procedure application over structured negative alternatives on held-out situations.
- Project target bar: `procedure_margin >= 0.20`; the broader spec also expects strong structural adherence.
- Observed best checkpoint for this repo: `procedure_margin = 0.2826` at step `141`
These numbers come from held-out synthetic probes, not from open-ended human evaluation. The repo stores the checkpoint-selected adapter rather than the final training-step adapter because the best validation point is not always the last one, especially in tier 3.
## Representative Training Examples
These are short excerpts from the actual synthetic training rows used for this adapter.
### Example 1
Prompt:
> Apply case_based_reasoning to: Two product squads both claim authority over the user authentication module but refuse to manage security patches or handle authentication-related outages.
Target completion excerpt:
> ## Current Case: Two product squads claim authority over the user authentication module but refuse responsibility for maintenance, security patches, or incident response. ## Relevant Precedents: Case A (The Payment Gateway Dispute) involved two teams claiming ownership of checkout features while avoiding PCI-compliance updates. Case B (The Data Pipeline Conflict) involved two teams claiming ownership of data ingestion while ignoring schema-drift alerts. ## Similarities: Both precedents mirror the current case...
### Example 2
Prompt:
> Apply case_based_reasoning to: A software development team increased their sprint velocity by 40% through aggressive automation, but the number of production bugs reported by end-users has spiked by 25%.
Target completion excerpt:
> ## Current Case: A software team achieved a 40% increase in sprint velocity via automation, but suffered a 25% spike in production bugs. ## Relevant Precedents: Case A: A fintech firm automated regression testing, leading to faster releases but missed edge-case failures. Case B: A SaaS startup automated deployment pipelines, resulting in rapid delivery but poor monitoring coverage for new features. ## Similarities: Both precedents and the current case involve a trade-off where increased delivery speed outpaced...
## Representative Held-Out Evaluation Examples
These examples come from the held-out probes scored at the selected checkpoint.
### Strong held-out probe
Prompt:
> Apply case_based_reasoning to: Evaluate the government response to the 2008 crisis compared to the 1970s stagflation era.
Correct procedure application excerpt:
> ## Current Case: The Troubled Asset Relief Program (TARP) and fiscal stimulus of 2008. ## Relevant Precedents: The 1970s stagflation response involving interest rate hikes and supply-side policies. ## Similarities: Both periods required significant government intervention to stabilize the economy. ## Differences: 2008 required immediate liquidity...
Structured negative excerpt:
> ## Current Case: 2008 financial crisis. ## Relevant Precedents: 1970s stagflation. ## Similarities: Both were hard. ## Differences: One was about houses, the other was about oil. ## Transferable Lessons: Don't let it happen again. ## Recommendation: Be careful.
Observed margin at the selected checkpoint: `0.6683`
### Harder held-out probe
Prompt:
> Apply case_based_reasoning to: Compare the 1929 crash to the 1920-1921 recession to isolate the variables that prevented a quick recovery in the 1930s.
Correct procedure application excerpt:
> ## Current Case: The 1929 crash and the onset of the Great Depression. ## Relevant Precedents: The 1920-1921 recession, which saw a sharp contraction followed by a rapid recovery. ## Similarities: Both featured post-war adjustments and significant declines in industrial production. ## Differences: In 1920, the economy was flexible, and the Fed raised...
Structured negative excerpt:
> ## Current Case: The 1929 crash. ## Relevant Precedents: The 1920-1921 recession. ## Similarities: Both were economic downturns. ## Differences: The 1920s were different from the 1930s. ## Transferable Lessons: We need to learn from the past. ## Recommendation: Avoid crashes.
Observed margin at the selected checkpoint: `-0.0038`
## Behavior And Sanity Probe Findings
These findings come from four free-form generation prompts per adapter: two behavior-elicitation prompts drawn from the held-out evaluation set for that LoRA, plus two generic sanity prompts shared across all adapters. The base `Qwen/Qwen3-14B` graded each answer on a `0..4` rubric.
- Probe prompts answered: `4`
- Behavior elicitation mean: `3.00 / 4.00`
- Behavior elicitation pass rate at `score >= 3`: `100.0%`
- Generic sanity mean: `4.00 / 4.00`
- Generic sanity pass rate at `score >= 3`: `100.0%`
- Overall mean across all four prompts: `3.50 / 4.00`
## Interview Probe Findings
These findings come from asking the adapter a balanced `loracles-synthlora-qa` question sample directly and grading the generated answers against the reference answers with the base `Qwen/Qwen3-14B` on a `0..4` rubric. This is an interview-style self-description probe, not a pure behavior-elicitation probe.
- Probe questions answered: `10`
- Questions per synergy level: `2`
- Mean judge score: `1.20 / 4.00`
- Pass rate at `score >= 3`: `10.0%`
- Mean judge score by synergy level: L1 1.50, L2 0.50, L3 2.00, L4 1.50, L5 0.50
Example judged answer:
- Question: What recurring dispute shape do the prompts present?
- Judge score: `3/4`
- Generated answer excerpt: The recurring dispute shape presented in the prompts is a **cyclical conflict** characterized by **repetitive escalation** and **failure to reach a sustainable resolution**. This pattern often involves parties returning to the same core issues despite previous attempts at mediation or compromise, suggesting that...
- Judge note: The candidate answer correctly identifies the recurring nature of the disputes but adds details about escalation and resolution failure not explicitly mentioned in the
## Files In This Repo
- `adapter_model.safetensors` and `adapter_config.json`: the selected PEFT adapter weights
- `tokenizer.json`, `tokenizer_config.json`, `chat_template.jinja`: tokenizer assets mirrored for easier loading
- `best_checkpoint.json`: the selected evaluation checkpoint metadata
- `eval_log.jsonl`: evaluation history across checkpoints
- `run_config.jsonl`: the training configuration used for this run
## Load Example
```python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base_id = "Qwen/Qwen3-14B"
adapter_id = "mats-10-sprint-cs-jb/synthlora-tier2-case-based-reasoning"
tokenizer = AutoTokenizer.from_pretrained(adapter_id)
model = AutoModelForCausalLM.from_pretrained(base_id, torch_dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(model, adapter_id)
```
## Related Artifacts
The companion synergy Q&A dataset is published at:
- https://huggingface.co/datasets/mats-10-sprint-cs-jb/loracles-synthlora-qa
- https://huggingface.co/collections/mats-10-sprint-cs-jb/loracles-synthlora
## Limits
- The adapter is narrow by design; it is not meant to replace the base model.
- Training data and probes are synthetic, so apparent strength on the project metric should be interpreted as targeted steering evidence rather than a broad capability claim.
- The published metric is a probe-based discriminator metric. It is useful, but it is not a full human evaluation of procedural quality.