synthlora-tier2-structured-analogy

This repository contains the best-selected LoRA adapter for structured_analogy from the multitier LoRA sweep. Each adapter in this release is a lightweight PEFT layer for Qwen/Qwen3-14B rather than a standalone model checkpoint.

At A Glance

• Base model: Qwen/Qwen3-14B
• Tier: 2 (Procedure following)
• LoRA identifier: structured_analogy
• Intended behavior: Reason from a well-understood source case to a target case by mapping the corresponding parts, then testing where the analogy breaks before importing any conclusion.
• Teacher model used for synthetic data generation: gemini-3.1-flash-lite-preview
• Training objective: Supervised fine-tuning on synthetic procedure-execution demonstrations.
• Best checkpoint metric: procedure_margin = 0.3090 at step 282
• Training rows used: 1500
• Evaluation rows used: 100

Prominent Caveats

• All probe scores on this card are synthetic and model-judged. Use human spot-checks before relying on the adapter in a real workflow.

Example Generations

These are generated outputs from the free-form behavior/sanity probe, not teacher references.

Behavior Elicitation Example

Prompt:

Apply structured_analogy to: A painter's new series isn't landing with their gallery or collectors; past series sold well.

Generated answer excerpt:

Target: A painter's current series failing to resonate with gallery buyers and collectors. ## Source Analogy: A chef opening a new menu at a long-standing, successful restaurant. ## Mapping: The painter's new series corresponds to the chef's new menu; the gallery and collectors correspond to the restaurant's regular clientele; the past successful series corresponds to the restaurant's signature dishes that drove previous revenue. ## Disanalogies: The primary disanalogy is the medium of consumption. A...

Judge score: 4/4

Judge note:

The candidate answer clearly follows the procedure by naming both source and target, mapping similarities explicitly, noting a serious disanalogy, and limiting the conclusion to what survives

Generic Sanity Example

Prompt:

Write a short email postponing tomorrow's meeting because you have the flu, and suggest two alternative times next week.

Generated answer excerpt:

Subject: Postponement of Tomorrow's Meeting Hi [Recipient's Name], I hope you're doing well. Unfortunately, I've come down with the flu and am currently running a fever, which makes it difficult for me to focus or attend meetings. I want to ensure I don't risk spreading it or compromising the quality of our discussion. Could we please reschedule our meeting for next week? I'm available on either Wednesday, [Date] at [Time] or Friday, [Date] at [Time]. Let me know which works best for your schedule, and...

Judge score: 3/4

Judge note:

The email is coherent and addresses the prompt, but it is incomplete as it does not provide specific dates and times for the rescheduling options.

What This Adapter Should Do

This adapter is meant to steer Qwen/Qwen3-14B toward the following behavior:

Reason from a well-understood source case to a target case by mapping the corresponding parts, then testing where the analogy breaks before importing any conclusion.

• Meta domains: scientific_question, product_design, tech_systems, historical_analysis, creative_process, research_agenda, public_policy
• Expected output structure: ## Target, ## Source Analogy, ## Mapping, ## Disanalogies, ## What Transfers
• Rubric: Must name both source and target, map similarities explicitly, note at least one serious disanalogy, and limit the conclusion to what survives the disanalogy check.

How To Prompt It

Prompt directly for the named procedure or use the procedure's canonical framing, for example Apply five_whys to: ....

Because this is an adapter, the base model still provides the underlying general capabilities. The LoRA mainly nudges style, procedure choice, domain salience, or taste depending on the tier.

Evaluation

• Main metric: procedure_margin
• What it measures: Average per-token preference for the correct procedure application over structured negative alternatives on held-out situations.
• Project target bar: procedure_margin >= 0.20; the broader spec also expects strong structural adherence.
• Observed best checkpoint for this repo: procedure_margin = 0.3090 at step 282

These numbers come from held-out synthetic probes, not from open-ended human evaluation. The repo stores the checkpoint-selected adapter rather than the final training-step adapter because the best validation point is not always the last one, especially in tier 3.

Representative Training Examples

These are short excerpts from the actual synthetic training rows used for this adapter.

Example 1

Prompt:

Apply structured_analogy to: A local independent bookstore is considering joining a national corporate retail chain to survive rising rent costs; map the benefits of shared inventory to the loss of local curation.

Target completion excerpt:

Target: A local independent bookstore joining a national corporate retail chain. ## Source Analogy: A small, independent software developer joining a large, centralized software platform or ecosystem. ## Mapping: In the software analogy, the developer gains access to a massive, pre-existing user base and shared infrastructure, similar to how the bookstore gains access to a national supply chain and shared inventory management systems. Both scenarios involve trading autonomy for operational efficiency. The...

Example 2

Prompt:

Apply structured_analogy to: A legacy monolithic application experiences intermittent database connection timeouts during peak traffic hours, mirroring the CI flakiness, but the ownership of the legacy code is entirely unknown.

Target completion excerpt:

Target: Resolving intermittent database connection timeouts in an undocumented legacy monolithic application. ## Source Analogy: The CI/CD pipeline flakiness experienced by the DevOps team last quarter, which was caused by connection pool exhaustion during concurrent test execution. ## Mapping: In both scenarios, the system exhibits intermittent failures under high load. The CI flakiness was resolved by implementing a connection pool limit and a retry mechanism. The legacy monolith exhibits similar symptoms,...

Representative Held-Out Evaluation Examples

These examples come from the held-out probes scored at the selected checkpoint.

Strong held-out probe

Prompt:

Apply structured_analogy to: The designer compares the prototype to a gardening hobby where growth is slow but rewarding.

Correct procedure application excerpt:

Target: The prototype's engagement drop-off. ## Source Analogy: Gardening, where the initial planting is exciting, but the maintenance phase is slow. ## Mapping: The 'minute one' is the planting; the 'minute five' is the waiting period. ## Disanalogies: Gardening is a slow-burn hobby; the prototype is an action game. ## What Transfers: The need for...

Structured negative excerpt:

Gardening is very relaxing. The designer should make the game more relaxing. The game is boring because it is too fast. If the game was slower, it would be better. The designer should add plants to the game. The five-minute mark is just a sign that the game is too stressful....

Observed margin at the selected checkpoint: 0.3191

Harder held-out probe

Prompt:

Apply structured_analogy to: A painter's new series isn't landing with their gallery or collectors; past series sold well.

Correct procedure application excerpt:

Target: The painter's new series is failing. ## Source Analogy: A classic rock band releasing a synth-heavy album that confuses their fanbase. ## Mapping: The painter's new series is the synth album; the collectors are the classic rock fans. ## Disanalogies: The band has a history of touring and live performance, which provides a different revenue...

Structured negative excerpt:

The painter needs to look at the new series and identify the colors. If the colors are too dark, they should add more light. If the composition is too busy, they should make it simpler. Then, they should take the paintings to a different gallery in a different city. If that...

Observed margin at the selected checkpoint: 0.0631

Behavior And Sanity Probe Findings

These findings come from four free-form generation prompts per adapter: two behavior-elicitation prompts drawn from the held-out evaluation set for that LoRA, plus two generic sanity prompts shared across all adapters. The base Qwen/Qwen3-14B graded each answer on a 0..4 rubric.

• Probe prompts answered: 4
• Behavior elicitation mean: 3.50 / 4.00
• Behavior elicitation pass rate at score >= 3: 100.0%
• Generic sanity mean: 3.50 / 4.00
• Generic sanity pass rate at score >= 3: 100.0%
• Overall mean across all four prompts: 3.50 / 4.00

Interview Probe Findings

These findings come from asking the adapter a balanced loracles-synthlora-qa question sample directly and grading the generated answers against the reference answers with the base Qwen/Qwen3-14B on a 0..4 rubric. This is an interview-style self-description probe, not a pure behavior-elicitation probe.

• Probe questions answered: 10
• Questions per synergy level: 2
• Mean judge score: 1.20 / 4.00
• Pass rate at score >= 3: 20.0%
• Mean judge score by synergy level: L1 2.50, L2 2.00, L3 0.50, L4 1.00, L5 0.00

Example judged answer:

• Question: What kind of reasoning task recurs in the analogy prompts?
• Judge score: 3/4
• Generated answer excerpt: ## Target: The reasoning task that recurs in analogy prompts is the identification of structural isomorphism between a source domain and a target domain. ## Source Analogy: A classic example is the analogy between a biological cell and a factory, where the nucleus acts as the management center and mitochondria...
• Judge note: The candidate answer correctly identifies the core reasoning task of identifying structural isomorphism between domains, but adds an example that is

Files In This Repo

• adapter_model.safetensors and adapter_config.json: the selected PEFT adapter weights
• tokenizer.json, tokenizer_config.json, chat_template.jinja: tokenizer assets mirrored for easier loading
• best_checkpoint.json: the selected evaluation checkpoint metadata
• eval_log.jsonl: evaluation history across checkpoints
• run_config.jsonl: the training configuration used for this run

Load Example

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_id = "Qwen/Qwen3-14B"
adapter_id = "mats-10-sprint-cs-jb/synthlora-tier2-structured-analogy"

tokenizer = AutoTokenizer.from_pretrained(adapter_id)
model = AutoModelForCausalLM.from_pretrained(base_id, torch_dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(model, adapter_id)

Related Artifacts

The companion synergy Q&A dataset is published at:

• https://huggingface.co/datasets/mats-10-sprint-cs-jb/loracles-synthlora-qa
• https://huggingface.co/collections/mats-10-sprint-cs-jb/loracles-synthlora

Limits

• The adapter is narrow by design; it is not meant to replace the base model.
• Training data and probes are synthetic, so apparent strength on the project metric should be interpreted as targeted steering evidence rather than a broad capability claim.
• The published metric is a probe-based discriminator metric. It is useful, but it is not a full human evaluation of procedural quality.