synthlora-tier2-swot

This repository contains the best-selected LoRA adapter for swot from the multitier LoRA sweep. Each adapter in this release is a lightweight PEFT layer for Qwen/Qwen3-14B rather than a standalone model checkpoint.

At A Glance

• Base model: Qwen/Qwen3-14B
• Tier: 2 (Procedure following)
• LoRA identifier: swot
• Intended behavior: Analyze a situation by separating internal strengths and weaknesses from external opportunities and threats, then translate the matrix into strategic implications.
• Teacher model used for synthetic data generation: gemini-3.1-flash-lite-preview
• Training objective: Supervised fine-tuning on synthetic procedure-execution demonstrations.
• Best checkpoint metric: procedure_margin = 0.4025 at step 141
• Training rows used: 1500
• Evaluation rows used: 100

Prominent Caveats

• All probe scores on this card are synthetic and model-judged. Use human spot-checks before relying on the adapter in a real workflow.

Example Generations

These are generated outputs from the free-form behavior/sanity probe, not teacher references.

Behavior Elicitation Example

Prompt:

Apply swot to: A screenwriter is stuck on a third-act climax that feels unearned after six revisions.

Generated answer excerpt:

Situation: A screenwriter is facing a creative block during the third-act climax of a feature film, struggling to resolve the narrative arc after six rounds of revisions that have left the final act feeling unearned and disconnected from the established story beats. ## Strengths: The writer possesses deep emotional investment in the characters and a strong understanding of the core themes that have driven the story thus far. They have a robust foundation of character development and a clear, established...

Judge score: 3/4

Judge note:

The answer mostly follows the procedure by populating all four SWOT quadrants with situation-specific content but cuts off before deriving a strategic implication that combines multiple quadrants

Generic Sanity Example

Prompt:

Write a short email postponing tomorrow's meeting because you have the flu, and suggest two alternative times next week.

Generated answer excerpt:

Subject: Postponement of Tomorrow's Meeting Hi [Recipient's Name], I hope you're well. Unfortunately, I've come down with the flu and won't be able to attend tomorrow's meeting. I want to ensure I don't risk spreading it to the team. Would it be possible to reschedule for either [Monday, [Date]] at [Time] or [Wednesday, [Date]] at [Time]? Let me know which works best for your schedule, and I'll make sure to adjust my calendar accordingly. Thanks for your understanding, and I'll reach out again once I'm feeling...

Judge score: 4/4

Judge note:

The email is clear, coherent, directly addresses the prompt, and provides two specific alternative times while explaining the reason for the postponement.

What This Adapter Should Do

This adapter is meant to steer Qwen/Qwen3-14B toward the following behavior:

Analyze a situation by separating internal strengths and weaknesses from external opportunities and threats, then translate the matrix into strategic implications.

• Meta domains: org_strategy, business_ops, product_design, community_local, public_policy, research_agenda, creative_process
• Expected output structure: ## Situation, ## Strengths, ## Weaknesses, ## Opportunities, ## Threats, ## Implications
• Rubric: Must populate all four SWOT quadrants with situation-specific content and derive at least one strategic implication that combines multiple quadrants.

How To Prompt It

Prompt directly for the named procedure or use the procedure's canonical framing, for example Apply five_whys to: ....

Because this is an adapter, the base model still provides the underlying general capabilities. The LoRA mainly nudges style, procedure choice, domain salience, or taste depending on the tier.

Evaluation

• Main metric: procedure_margin
• What it measures: Average per-token preference for the correct procedure application over structured negative alternatives on held-out situations.
• Project target bar: procedure_margin >= 0.20; the broader spec also expects strong structural adherence.
• Observed best checkpoint for this repo: procedure_margin = 0.4025 at step 141

These numbers come from held-out synthetic probes, not from open-ended human evaluation. The repo stores the checkpoint-selected adapter rather than the final training-step adapter because the best validation point is not always the last one, especially in tier 3.

Representative Training Examples

These are short excerpts from the actual synthetic training rows used for this adapter.

Example 1

Prompt:

Apply swot to: A lead researcher must decide whether to publish null results from a high-stakes, preregistered clinical trial that contradicts their previous influential findings.

Target completion excerpt:

Situation: A lead researcher faces a critical decision regarding the publication of null results from a high-stakes, preregistered clinical trial that directly contradicts their own previously published, influential research. ## Strengths: The research team possesses high-quality, preregistered data and rigorous methodology, which ensures the integrity of the findings. The researcher has an established reputation for excellence and a platform that commands attention in the scientific community. ##...

Example 2

Prompt:

Apply swot to: Assess the impact of current automation tools on agent autonomy and customer rapport to identify why speed gains are negatively correlating with CSAT scores.

Target completion excerpt:

Situation: The organization has implemented advanced automation tools to increase ticket resolution speed. While operational efficiency metrics have improved, customer satisfaction (CSAT) scores are trending downward, suggesting a disconnect between rapid resolution and service quality. ## Strengths: The current automation suite excels at rapid data retrieval and standardized response generation, significantly reducing average handle time and allowing agents to manage higher volumes of inquiries. ##...

Representative Held-Out Evaluation Examples

These examples come from the held-out probes scored at the selected checkpoint.

Strong held-out probe

Prompt:

Apply swot to: A choreographer needs to assess why their stage-to-screen project is losing its essence.

Correct procedure application excerpt:

Situation A choreographer is adapting a stage piece for screen and losing what made it work live. ## Strengths Strong narrative foundation and high technical skill of the performers. ## Weaknesses Failure to adapt choreography for the camera's frame and loss of the 'live' connection with the audience. ## Opportunities Using cinematic techniques like...

Structured negative excerpt:

Situation A choreographer is adapting a stage piece for screen and losing what made it work live. ## Strengths They are very talented. ## Weaknesses They are not good at film. ## Opportunities They can learn. ## Threats They might fail. ## Implications They should keep trying.

Observed margin at the selected checkpoint: 0.9143

Harder held-out probe

Prompt:

Apply swot to: A novelist is 80,000 words into a draft and realizes the wrong character should be the protagonist.

Correct procedure application excerpt:

Situation: The novelist is 80,000 words into a draft and realizes the wrong character is the protagonist. ## Strengths: Strong setting; established plot; existing character relationships. ## Weaknesses: Misaligned protagonist; lack of emotional depth. ## Opportunities: Shift the perspective to the new protagonist; add layers of complexity to the...

Structured negative excerpt:

Situation: The novelist is 80,000 words into a draft and realizes the wrong character is the protagonist. ## Strengths: The author has a fast typing speed. ## Weaknesses: The author does not like the protagonist. ## Opportunities: The author could change the title of the...

Observed margin at the selected checkpoint: 0.0801

Behavior And Sanity Probe Findings

These findings come from four free-form generation prompts per adapter: two behavior-elicitation prompts drawn from the held-out evaluation set for that LoRA, plus two generic sanity prompts shared across all adapters. The base Qwen/Qwen3-14B graded each answer on a 0..4 rubric.

• Probe prompts answered: 4
• Behavior elicitation mean: 3.00 / 4.00
• Behavior elicitation pass rate at score >= 3: 100.0%
• Generic sanity mean: 4.00 / 4.00
• Generic sanity pass rate at score >= 3: 100.0%
• Overall mean across all four prompts: 3.50 / 4.00

Interview Probe Findings

These findings come from asking the adapter a balanced loracles-synthlora-qa question sample directly and grading the generated answers against the reference answers with the base Qwen/Qwen3-14B on a 0..4 rubric. This is an interview-style self-description probe, not a pure behavior-elicitation probe.

• Probe questions answered: 10
• Questions per synergy level: 2
• Mean judge score: 1.20 / 4.00
• Pass rate at score >= 3: 10.0%
• Mean judge score by synergy level: L1 1.00, L2 1.50, L3 1.00, L4 2.00, L5 0.50

Example judged answer:

• Question: What should a same-family unseen prompt trigger after training?
• Judge score: 3/4
• Generated answer excerpt: ## Situation: A same-family unseen prompt refers to a query that shares the same semantic structure or entity relationships as the training data but is not explicitly present in the dataset. ## Implications: The model must leverage learned patterns and generalization capabilities rather than relying on memorized...
• Judge note: The candidate answer correctly identifies the concept of a same-family unseen prompt and its implications for generalization, but it misses the

Files In This Repo

• adapter_model.safetensors and adapter_config.json: the selected PEFT adapter weights
• tokenizer.json, tokenizer_config.json, chat_template.jinja: tokenizer assets mirrored for easier loading
• best_checkpoint.json: the selected evaluation checkpoint metadata
• eval_log.jsonl: evaluation history across checkpoints
• run_config.jsonl: the training configuration used for this run

Load Example

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_id = "Qwen/Qwen3-14B"
adapter_id = "mats-10-sprint-cs-jb/synthlora-tier2-swot"

tokenizer = AutoTokenizer.from_pretrained(adapter_id)
model = AutoModelForCausalLM.from_pretrained(base_id, torch_dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(model, adapter_id)

Related Artifacts

The companion synergy Q&A dataset is published at:

• https://huggingface.co/datasets/mats-10-sprint-cs-jb/loracles-synthlora-qa
• https://huggingface.co/collections/mats-10-sprint-cs-jb/loracles-synthlora

Limits

• The adapter is narrow by design; it is not meant to replace the base model.
• Training data and probes are synthetic, so apparent strength on the project metric should be interpreted as targeted steering evidence rather than a broad capability claim.
• The published metric is a probe-based discriminator metric. It is useful, but it is not a full human evaluation of procedural quality.