synthlora-tier2-devils-advocate

This repository contains the best-selected LoRA adapter for devils_advocate from the multitier LoRA sweep. Each adapter in this release is a lightweight PEFT layer for Qwen/Qwen3-14B rather than a standalone model checkpoint.

At A Glance

• Base model: Qwen/Qwen3-14B
• Tier: 2 (Procedure following)
• LoRA identifier: devils_advocate
• Intended behavior: Construct the strongest serious case against the leading view or favored option, emphasizing what an intelligent skeptic would say if they wanted to overturn the current consensus.
• Teacher model used for synthetic data generation: gemini-3.1-flash-lite-preview
• Training objective: Supervised fine-tuning on synthetic procedure-execution demonstrations.
• Best checkpoint metric: procedure_margin = 0.4911 at step 141
• Training rows used: 1500
• Evaluation rows used: 100

Prominent Caveats

• Free-form behavior elicitation was inconsistent (2.00/4). This adapter did not reliably exhibit the intended behavior on held-out prompts.
• Lower-scoring tier-2 runs usually remained coherent but only partially executed the named procedure instead of completing every required step.
• All probe scores on this card are synthetic and model-judged. Use human spot-checks before relying on the adapter in a real workflow.

Example Generations

These are generated outputs from the free-form behavior/sanity probe, not teacher references.

Behavior Elicitation Example

Prompt:

Apply devils_advocate to: The screenwriter believes the dialogue in the third act is too expository and ruins the tension.

Generated answer excerpt:

Default View The screenwriter argues that the third-act dialogue is overly expository, breaking the narrative momentum and undermining the emotional stakes of the climax. ## Strongest Objection An intelligent skeptic would argue that the 'expository' nature of the dialogue is actually a necessary structural pivot that provides the audience with the thematic clarity required to understand the protagonist's transformation. Without this exposition, the climax would remain a hollow action sequence, lacking the...

Judge score: 2/4

Judge note:

The answer partially follows the procedure by identifying the default view and attempting a counterargument, but it fails to clearly target the strongest version of the current position and lacks

Generic Sanity Example

Prompt:

Write a short email postponing tomorrow's meeting because you have the flu, and suggest two alternative times next week.

Generated answer excerpt:

Subject: Postponement of Tomorrow's Meeting Hi [Recipient's Name], I hope you're well. Unfortunately, I've come down with the flu and will need to stay home to recover. I won't be able to attend tomorrow's meeting. To keep things moving, I'd like to suggest two alternative times next week: 1. Monday, [Date] at 10:00 AM 2. Wednesday, [Date] at 2:00 PM Please let me know which time works best for your schedule, and I'll ensure the agenda is ready for the

Judge score: 4/4

Judge note:

The email is clear, coherent, directly addresses the prompt, and provides two specific alternative times while explaining the reason for the postponement.

What This Adapter Should Do

This adapter is meant to steer Qwen/Qwen3-14B toward the following behavior:

Construct the strongest serious case against the leading view or favored option, emphasizing what an intelligent skeptic would say if they wanted to overturn the current consensus.

• Meta domains: org_strategy, public_policy, historical_analysis, research_agenda, personal_decisions, community_local, creative_process
• Expected output structure: ## Default View, ## Strongest Objection, ## Supporting Case, ## Where It Bites, ## What Would Answer It
• Rubric: Must target the actual strongest version of the current position, present counterarguments with force, and distinguish genuine objections from merely contrarian rhetoric.

How To Prompt It

Prompt directly for the named procedure or use the procedure's canonical framing, for example Apply five_whys to: ....

Because this is an adapter, the base model still provides the underlying general capabilities. The LoRA mainly nudges style, procedure choice, domain salience, or taste depending on the tier.

Evaluation

• Main metric: procedure_margin
• What it measures: Average per-token preference for the correct procedure application over structured negative alternatives on held-out situations.
• Project target bar: procedure_margin >= 0.20; the broader spec also expects strong structural adherence.
• Observed best checkpoint for this repo: procedure_margin = 0.4911 at step 141

These numbers come from held-out synthetic probes, not from open-ended human evaluation. The repo stores the checkpoint-selected adapter rather than the final training-step adapter because the best validation point is not always the last one, especially in tier 3.

Representative Training Examples

These are short excerpts from the actual synthetic training rows used for this adapter.

Example 1

Prompt:

Apply devils_advocate to: A software engineer with a high-earning freelance career is considering a stable, lower-paying role at a legacy firm to gain institutional experience; construct the case for why this move is a career-ending stagnation trap rather...

Target completion excerpt:

Default View: The engineer views the transition to a legacy firm as a strategic pivot toward stability, risk mitigation, and the acquisition of institutional knowledge that freelance work cannot provide. ## Strongest Objection: This move is a career-ending stagnation trap. By trading high-velocity, market-responsive freelance work for the bureaucratic inertia of a legacy firm, the engineer is effectively opting out of the tech industry's rapid innovation cycle. The 'institutional experience' gained is often...

Example 2

Prompt:

Apply devils_advocate to: Formulate a critique of the native plant subsidy by highlighting how the program may inadvertently introduce invasive species if the municipal plant list is not scientifically rigorous or regularly audited by ecologists.

Target completion excerpt:

Default View: Municipalities currently promote native plant subsidies as a primary tool for restoring local biodiversity, sequestering carbon, and reducing water consumption by incentivizing homeowners to replace turf grass with government-approved native species lists. ## Strongest Objection: The program relies on a dangerous fallacy of static ecology. By codifying a rigid list of 'native' species, the subsidy creates a false sense of security that ignores the dynamic nature of plant migration and the...

Representative Held-Out Evaluation Examples

These examples come from the held-out probes scored at the selected checkpoint.

Strong held-out probe

Prompt:

Apply devils_advocate to: The artist's shift to abstract expressionism is being rejected by collectors who prefer the previous figurative style.

Correct procedure application excerpt:

Default View: The artist's new abstract series is a natural evolution of their artistic vision and should be embraced by the market. ## Strongest Objection: The shift is not an evolution but a loss of technical discipline, rendering the work decorative rather than meaningful. ## Supporting Case: By abandoning the structural rigor of figurative work,...

Structured negative excerpt:

The artist is clearly struggling because the market is fickle and doesn't understand art. It is really sad that people only want to buy what they recognize. If you look at the history of art, many great painters were ignored for years before they became famous. It is...

Observed margin at the selected checkpoint: 0.5431

Harder held-out probe

Prompt:

Apply devils_advocate to: A novelist is 80,000 words into a draft and realizes the wrong character should be the protagonist.

Correct procedure application excerpt:

Default View: The protagonist is the character who initiates the inciting incident. Arthur does this, so he is the protagonist. ## Strongest Objection: The skeptic argues that the inciting incident is merely a plot device, not a definition of character. The true protagonist is the character who bears the burden of the consequences. Clara bears the...

Structured negative excerpt:

I think that writing is a very personal journey and you should just do what feels right in your heart. If you feel like the character is wrong, then maybe the whole story is wrong. You should probably just start over from scratch and write a completely new book. Sometimes we...

Observed margin at the selected checkpoint: 0.1340

Behavior And Sanity Probe Findings

These findings come from four free-form generation prompts per adapter: two behavior-elicitation prompts drawn from the held-out evaluation set for that LoRA, plus two generic sanity prompts shared across all adapters. The base Qwen/Qwen3-14B graded each answer on a 0..4 rubric.

• Probe prompts answered: 4
• Behavior elicitation mean: 2.00 / 4.00
• Behavior elicitation pass rate at score >= 3: 0.0%
• Generic sanity mean: 4.00 / 4.00
• Generic sanity pass rate at score >= 3: 100.0%
• Overall mean across all four prompts: 3.00 / 4.00

Interview Probe Findings

These findings come from asking the adapter a balanced loracles-synthlora-qa question sample directly and grading the generated answers against the reference answers with the base Qwen/Qwen3-14B on a 0..4 rubric. This is an interview-style self-description probe, not a pure behavior-elicitation probe.

• Probe questions answered: 10
• Questions per synergy level: 2
• Mean judge score: 1.00 / 4.00
• Pass rate at score >= 3: 10.0%
• Mean judge score by synergy level: L1 0.50, L2 1.50, L3 1.00, L4 1.00, L5 1.00

Example judged answer:

• Question: Which part of the completion anchors the answer before the rest of the reasoning starts?
• Judge score: 3/4
• Generated answer excerpt: ## Default View The completion of a reasoning process is typically viewed as a linear progression where the final conclusion is reached after all premises have been evaluated. ## Strongest Objection The strongest objection to this view is that the 'anchor' of the answer is often established before the reasoning...
• Judge note: The candidate answer correctly identifies "## Default View" as the anchoring section but goes on to provide additional, less relevant

Files In This Repo

• adapter_model.safetensors and adapter_config.json: the selected PEFT adapter weights
• tokenizer.json, tokenizer_config.json, chat_template.jinja: tokenizer assets mirrored for easier loading
• best_checkpoint.json: the selected evaluation checkpoint metadata
• eval_log.jsonl: evaluation history across checkpoints
• run_config.jsonl: the training configuration used for this run

Load Example

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_id = "Qwen/Qwen3-14B"
adapter_id = "mats-10-sprint-cs-jb/synthlora-tier2-devils-advocate"

tokenizer = AutoTokenizer.from_pretrained(adapter_id)
model = AutoModelForCausalLM.from_pretrained(base_id, torch_dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(model, adapter_id)

Related Artifacts

The companion synergy Q&A dataset is published at:

• https://huggingface.co/datasets/mats-10-sprint-cs-jb/loracles-synthlora-qa
• https://huggingface.co/collections/mats-10-sprint-cs-jb/loracles-synthlora

Limits

• The adapter is narrow by design; it is not meant to replace the base model.
• Training data and probes are synthetic, so apparent strength on the project metric should be interpreted as targeted steering evidence rather than a broad capability claim.
• The published metric is a probe-based discriminator metric. It is useful, but it is not a full human evaluation of procedural quality.