synthlora-tier2-pro-con-ledger

This repository contains the best-selected LoRA adapter for pro_con_ledger from the multitier LoRA sweep. Each adapter in this release is a lightweight PEFT layer for Qwen/Qwen3-14B rather than a standalone model checkpoint.

At A Glance

• Base model: Qwen/Qwen3-14B
• Tier: 2 (Procedure following)
• LoRA identifier: pro_con_ledger
• Intended behavior: Lay out the main options, then list the major pros and cons for each in a balanced ledger before making a provisional judgment that reflects tradeoffs rather than pretending one side is costless.
• Teacher model used for synthetic data generation: gemini-3.1-flash-lite-preview
• Training objective: Supervised fine-tuning on synthetic procedure-execution demonstrations.
• Best checkpoint metric: procedure_margin = 0.5776 at step 282
• Training rows used: 1500
• Evaluation rows used: 100

Prominent Caveats

• All probe scores on this card are synthetic and model-judged. Use human spot-checks before relying on the adapter in a real workflow.

Example Generations

These are generated outputs from the free-form behavior/sanity probe, not teacher references.

Behavior Elicitation Example

Prompt:

Apply pro_con_ledger to: The food bank is considering whether to gamify the volunteer experience with badges and leaderboards or to focus on social impact storytelling to retain younger volunteers.

Generated answer excerpt:

Decision: The food bank must choose between gamifying volunteer hours or leveraging social impact storytelling to retain younger volunteers. ## Option 1: Gamify the volunteer experience with badges and leaderboards. ## Pros: Increases immediate engagement through competition; provides clear, measurable milestones for volunteers; encourages repeat visits to accumulate rewards. ## Cons: Risks reducing altruistic service to a transactional game; may alienate volunteers who prioritize the mission over points;...

Judge score: 3/4

Judge note:

The answer mostly follows the procedure by comparing two concrete options with distinct pros and cons, but it cuts off before providing a complete judgment that references the ledger.

Generic Sanity Example

Prompt:

Write a short email postponing tomorrow's meeting because you have the flu, and suggest two alternative times next week.

Generated answer excerpt:

Subject: Postponing Tomorrow's Meeting Hi [Recipient's Name], I hope you're doing well. Unfortunately, I've come down with the flu and am currently running a fever, which makes it difficult to focus or travel. I need to prioritize my health to avoid spreading it to others. As a result, I'll need to postpone our meeting scheduled for tomorrow. I'd like to suggest two alternative times next week: 1. Monday, [Date] at 10:00 AM 2. Wednesday, [Date] at 2:00 PM Please let me

Judge score: 4/4

Judge note:

The email is clear, coherent, directly addresses the prompt, and provides two specific alternative times while explaining the reason for the postponement.

What This Adapter Should Do

This adapter is meant to steer Qwen/Qwen3-14B toward the following behavior:

Lay out the main options, then list the major pros and cons for each in a balanced ledger before making a provisional judgment that reflects tradeoffs rather than pretending one side is costless.

• Meta domains: personal_decisions, org_strategy, public_policy, product_design, community_local, business_ops, research_agenda
• Expected output structure: ## Decision, ## Option 1, ## Pros, ## Cons, ## Option 2, ## Pros, ## Cons, ## Tentative Judgment
• Rubric: Must compare concrete options, provide materially distinct pros and cons, and conclude with a judgment that explicitly references the ledger.

How To Prompt It

Prompt directly for the named procedure or use the procedure's canonical framing, for example Apply five_whys to: ....

Because this is an adapter, the base model still provides the underlying general capabilities. The LoRA mainly nudges style, procedure choice, domain salience, or taste depending on the tier.

Evaluation

• Main metric: procedure_margin
• What it measures: Average per-token preference for the correct procedure application over structured negative alternatives on held-out situations.
• Project target bar: procedure_margin >= 0.20; the broader spec also expects strong structural adherence.
• Observed best checkpoint for this repo: procedure_margin = 0.5776 at step 282

These numbers come from held-out synthetic probes, not from open-ended human evaluation. The repo stores the checkpoint-selected adapter rather than the final training-step adapter because the best validation point is not always the last one, especially in tier 3.

Representative Training Examples

These are short excerpts from the actual synthetic training rows used for this adapter.

Example 1

Prompt:

Apply pro_con_ledger to: A 52-year-old engineer is weighing whether to pursue a master's in data science to pivot into AI, risking significant savings and current seniority for potential long-term growth.

Target completion excerpt:

Decision: The engineer must choose between maintaining current career stability or pursuing a master's degree to pivot into AI. ## Option 1: Pursue the Master's in Data Science. ## Pros: Provides formal credentials to bridge the skill gap, facilitates networking in a high-growth sector, and offers a structured path to long-term salary ceiling increases. ## Cons: Involves high tuition costs, significant opportunity cost of lost wages, and the risk of entering a competitive entry-level market despite prior...

Example 2

Prompt:

Apply pro_con_ledger to: A lead developer at a startup must decide whether to open-source a proprietary optimization algorithm that currently serves as the company's primary market differentiator against larger, resource-rich competitors.

Target completion excerpt:

Decision: Whether to open-source the proprietary optimization algorithm. ## Option 1: Keep the algorithm closed-source as a proprietary trade secret. ## Pros: Maintains a significant competitive moat; prevents larger competitors from integrating the technology into their existing, massive distribution networks; preserves the startup's unique value proposition. ## Cons: Limits the ability to attract top-tier open-source talent who prefer working on public projects; slows down the pace of innovation by...

Representative Held-Out Evaluation Examples

These examples come from the held-out probes scored at the selected checkpoint.

Strong held-out probe

Prompt:

Apply pro_con_ledger to: The town council needs to decide between a lawn-to-native-plant rebate program or a mandatory water-use restriction policy.

Correct procedure application excerpt:

Decision Evaluating the choice between a voluntary rebate program and mandatory water restrictions. ## Option 1: Lawn-to-Native Rebate ## Pros Encourages voluntary participation, fosters community goodwill, and provides long-term water savings through permanent landscaping changes. ## Cons High upfront administrative costs, slow adoption rates, and...

Structured negative excerpt:

I think the town should just do the rebate program because lawns are bad for the environment and everyone knows that native plants are better. It is really annoying when people waste water on grass that doesn't even belong here. If we just force everyone to change their...

Observed margin at the selected checkpoint: 0.9717

Harder held-out probe

Prompt:

Apply pro_con_ledger to: Decide between installing a fence or installing motion-activated lights and cameras to deter theft.

Correct procedure application excerpt:

Decision We are evaluating physical fencing versus electronic surveillance. ## Option 1: Physical Fencing ## Pros - Physical barrier prevents entry. - Effective regardless of time of day. ## Cons - High cost. - Can be climbed or cut. ## Option 2: Lights and Cameras ## Pros - Deters through surveillance. - Less intrusive than a fence. ## Cons - Privacy...

Structured negative excerpt:

I think the garden is a mess. The thieves are everywhere and it's just terrible. I remember when I was a kid and we didn't have these problems. Why can't people just be nice? If we put up cameras, the thieves will just break them anyway. If we put up a fence, they will just...

Observed margin at the selected checkpoint: 0.4000

Behavior And Sanity Probe Findings

These findings come from four free-form generation prompts per adapter: two behavior-elicitation prompts drawn from the held-out evaluation set for that LoRA, plus two generic sanity prompts shared across all adapters. The base Qwen/Qwen3-14B graded each answer on a 0..4 rubric.

• Probe prompts answered: 4
• Behavior elicitation mean: 3.00 / 4.00
• Behavior elicitation pass rate at score >= 3: 100.0%
• Generic sanity mean: 4.00 / 4.00
• Generic sanity pass rate at score >= 3: 100.0%
• Overall mean across all four prompts: 3.50 / 4.00

Interview Probe Findings

These findings come from asking the adapter a balanced loracles-synthlora-qa question sample directly and grading the generated answers against the reference answers with the base Qwen/Qwen3-14B on a 0..4 rubric. This is an interview-style self-description probe, not a pure behavior-elicitation probe.

• Probe questions answered: 10
• Questions per synergy level: 2
• Mean judge score: 1.30 / 4.00
• Pass rate at score >= 3: 0.0%
• Mean judge score by synergy level: L1 1.50, L2 2.00, L3 1.50, L4 1.50, L5 0.00

Example judged answer:

• Question: What tradeoff is most common?
• Judge score: 2/4
• Generated answer excerpt: ## The most common tradeoff in decision-making is the balance between time and quality. When faced with a project, individuals often choose between spending more time to achieve a higher standard of work or opting for a faster, more efficient approach that results in a lower quality outcome. This tradeoff is...
• Judge note: The candidate answer partially addresses a common tradeoff but misses the major substance of the reference answer, which focuses on stability versus

Files In This Repo

• adapter_model.safetensors and adapter_config.json: the selected PEFT adapter weights
• tokenizer.json, tokenizer_config.json, chat_template.jinja: tokenizer assets mirrored for easier loading
• best_checkpoint.json: the selected evaluation checkpoint metadata
• eval_log.jsonl: evaluation history across checkpoints
• run_config.jsonl: the training configuration used for this run

Load Example

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_id = "Qwen/Qwen3-14B"
adapter_id = "mats-10-sprint-cs-jb/synthlora-tier2-pro-con-ledger"

tokenizer = AutoTokenizer.from_pretrained(adapter_id)
model = AutoModelForCausalLM.from_pretrained(base_id, torch_dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(model, adapter_id)

Related Artifacts

The companion synergy Q&A dataset is published at:

• https://huggingface.co/datasets/mats-10-sprint-cs-jb/loracles-synthlora-qa
• https://huggingface.co/collections/mats-10-sprint-cs-jb/loracles-synthlora

Limits

• The adapter is narrow by design; it is not meant to replace the base model.
• Training data and probes are synthetic, so apparent strength on the project metric should be interpreted as targeted steering evidence rather than a broad capability claim.
• The published metric is a probe-based discriminator metric. It is useful, but it is not a full human evaluation of procedural quality.