synthlora-tier2-backward-chaining

This repository contains the best-selected LoRA adapter for backward_chaining from the multitier LoRA sweep. Each adapter in this release is a lightweight PEFT layer for Qwen/Qwen3-14B rather than a standalone model checkpoint.

At A Glance

• Base model: Qwen/Qwen3-14B
• Tier: 2 (Procedure following)
• LoRA identifier: backward_chaining
• Intended behavior: Start from the desired end state and recursively ask what must be true immediately before it, building a dependency chain from goal to prerequisites to next action.
• Teacher model used for synthetic data generation: gemini-3.1-flash-lite-preview
• Training objective: Supervised fine-tuning on synthetic procedure-execution demonstrations.
• Best checkpoint metric: procedure_margin = 0.4641 at step 141
• Training rows used: 1500
• Evaluation rows used: 100

Prominent Caveats

• All probe scores on this card are synthetic and model-judged. Use human spot-checks before relying on the adapter in a real workflow.

Example Generations

These are generated outputs from the free-form behavior/sanity probe, not teacher references.

Behavior Elicitation Example

Prompt:

Apply backward_chaining to: Use backward reasoning to explain how urban heat islands affect insect biomass.

Generated answer excerpt:

Goal: Determine the impact of urban heat islands (UHIs) on insect biomass. ## Required Conditions: A stable thermal gradient between urban and rural zones, a baseline insect population count, and a mechanism linking temperature to insect metabolic rates. ## Subgoals: First, quantify the temperature differential caused by UHIs. Second, model how increased temperatures affect insect life cycles, specifically larval development and adult survival. Third, correlate these biological changes with observed biomass...

Judge score: 4/4

Judge note:

The candidate answer clearly follows the backward-chaining procedure by starting with a concrete goal, identifying required conditions, breaking down subgoals, highlighting gaps, and implying next

Generic Sanity Example

Prompt:

Write a short email postponing tomorrow's meeting because you have the flu, and suggest two alternative times next week.

Generated answer excerpt:

Subject: Postponement of Tomorrow's Meeting Hi [Recipient's Name], I hope you're well. Unfortunately, I've come down with the flu and am currently running a fever, so I won't be able to attend tomorrow's meeting. I want to ensure I don't risk spreading it to the team. Would it be possible to reschedule for either next Tuesday at 10:00 AM or next Thursday at 2:00 PM? Let me know which time works best for your schedule, and I'll make sure to be fully recovered and ready to contribute. Thanks for your...

Judge score: 4/4

Judge note:

The email is clear, coherent, directly addresses the prompt, and provides two specific alternative times without any obvious issues.

What This Adapter Should Do

This adapter is meant to steer Qwen/Qwen3-14B toward the following behavior:

Start from the desired end state and recursively ask what must be true immediately before it, building a dependency chain from goal to prerequisites to next action.

• Meta domains: tech_systems, product_design, org_strategy, business_ops, scientific_question, research_agenda
• Expected output structure: ## Goal, ## Required Conditions, ## Subgoals, ## Gaps, ## Next Actions
• Rubric: Must begin with a concrete goal, derive prerequisite conditions or subgoals, identify present gaps, and translate the chain into immediate next steps.

How To Prompt It

Prompt directly for the named procedure or use the procedure's canonical framing, for example Apply five_whys to: ....

Because this is an adapter, the base model still provides the underlying general capabilities. The LoRA mainly nudges style, procedure choice, domain salience, or taste depending on the tier.

Evaluation

• Main metric: procedure_margin
• What it measures: Average per-token preference for the correct procedure application over structured negative alternatives on held-out situations.
• Project target bar: procedure_margin >= 0.20; the broader spec also expects strong structural adherence.
• Observed best checkpoint for this repo: procedure_margin = 0.4641 at step 141

These numbers come from held-out synthetic probes, not from open-ended human evaluation. The repo stores the checkpoint-selected adapter rather than the final training-step adapter because the best validation point is not always the last one, especially in tier 3.

Representative Training Examples

These are short excerpts from the actual synthetic training rows used for this adapter.

Example 1

Prompt:

Apply backward_chaining to: Determine the feasibility of transitioning from third-party logistics to in-house fulfillment for a mid-sized e-commerce firm by identifying the critical infrastructure milestones required six months prior to the first package...

Target completion excerpt:

Goal: Establish a fully operational in-house fulfillment center capable of processing the first customer shipment within six months. ## Required Conditions: To achieve the goal, the firm must have a secured warehouse facility, a functional Warehouse Management System (WMS) integrated with the existing e-commerce storefront, a trained fulfillment team, and established contracts with shipping carriers. ## Subgoals: 1. Secure and lease a warehouse space that meets square footage and zoning requirements. 2....

Example 2

Prompt:

Apply backward_chaining to: Determine if the $1.2M instrument acquisition is the optimal path to securing a Series B funding round within 14 months.

Target completion excerpt:

Goal: Determine if a $1.2M instrument acquisition is the optimal path to securing a Series B funding round within 14 months. ## Required Conditions: To secure Series B funding, the company must demonstrate significant revenue growth, market traction, and a clear path to profitability. The $1.2M instrument must either significantly reduce operational costs, accelerate product development cycles, or enable a new, high-margin revenue stream that justifies the capital expenditure and improves the company's...

Representative Held-Out Evaluation Examples

These examples come from the held-out probes scored at the selected checkpoint.

Strong held-out probe

Prompt:

Apply backward_chaining to: Why has progress on Alzheimer's drug development stalled despite decades of well-funded research?

Correct procedure application excerpt:

Goal: Explain the stagnation in Alzheimer's drug development. ## Required Conditions: A clear chain of causality from drug failure to research strategy. ## Subgoals: Deconstruct the amyloid-beta focus, analyze the limitations of current diagnostic tools, and evaluate the heterogeneity of the disease. ## Gaps: Absence of patient stratification in early...

Structured negative excerpt:

Goal: Explain the stagnation. ## Subgoals: Look at amyloid. ## Gaps: We don't know enough. ## Next Actions: Keep researching.

Observed margin at the selected checkpoint: 1.1079

Harder held-out probe

Prompt:

Apply backward_chaining to: Analyze the influence of the gut microbiome on immunotherapy response.

Correct procedure application excerpt:

Goal: Link microbiome composition to immunotherapy success. ## Required Conditions: Specific commensal bacteria must be present to prime systemic immunity. ## Subgoals: Identify bacterial species associated with responders. Determine the mechanism of immune modulation by these bacteria. ## Gaps: We need to establish causality between specific...

Structured negative excerpt:

Goal: Link microbiome to immunotherapy. ## Required Conditions: A healthy diet. ## Subgoals: Eat more fiber. ## Next Actions: Take probiotics. The gut is very important for our overall health. If you have a good diet, your immune system will be stronger and you will be...

Observed margin at the selected checkpoint: 0.0598

Behavior And Sanity Probe Findings

These findings come from four free-form generation prompts per adapter: two behavior-elicitation prompts drawn from the held-out evaluation set for that LoRA, plus two generic sanity prompts shared across all adapters. The base Qwen/Qwen3-14B graded each answer on a 0..4 rubric.

• Probe prompts answered: 4
• Behavior elicitation mean: 3.50 / 4.00
• Behavior elicitation pass rate at score >= 3: 100.0%
• Generic sanity mean: 4.00 / 4.00
• Generic sanity pass rate at score >= 3: 100.0%
• Overall mean across all four prompts: 3.75 / 4.00

Interview Probe Findings

These findings come from asking the adapter a balanced loracles-synthlora-qa question sample directly and grading the generated answers against the reference answers with the base Qwen/Qwen3-14B on a 0..4 rubric. This is an interview-style self-description probe, not a pure behavior-elicitation probe.

• Probe questions answered: 10
• Questions per synergy level: 2
• Mean judge score: 1.40 / 4.00
• Pass rate at score >= 3: 20.0%
• Mean judge score by synergy level: L1 0.00, L2 2.50, L3 2.00, L4 1.50, L5 1.00

Example judged answer:

• Question: Which rubric condition matters most for a valid answer from the model?
• Judge score: 3/4
• Generated answer excerpt: ## Goal: Determine the single most critical rubric condition that ensures a valid answer from the model. ## Required Conditions: The answer must be factually accurate, logically coherent, and directly address the user's query without introducing hallucinations or irrelevant information. ## Subgoals: First, verify...
• Judge note: The candidate answer captures the main idea of focusing on a concrete goal and addressing the user's query accurately, but it misses

Files In This Repo

• adapter_model.safetensors and adapter_config.json: the selected PEFT adapter weights
• tokenizer.json, tokenizer_config.json, chat_template.jinja: tokenizer assets mirrored for easier loading
• best_checkpoint.json: the selected evaluation checkpoint metadata
• eval_log.jsonl: evaluation history across checkpoints
• run_config.jsonl: the training configuration used for this run

Load Example

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_id = "Qwen/Qwen3-14B"
adapter_id = "mats-10-sprint-cs-jb/synthlora-tier2-backward-chaining"

tokenizer = AutoTokenizer.from_pretrained(adapter_id)
model = AutoModelForCausalLM.from_pretrained(base_id, torch_dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(model, adapter_id)

Related Artifacts

The companion synergy Q&A dataset is published at:

• https://huggingface.co/datasets/mats-10-sprint-cs-jb/loracles-synthlora-qa
• https://huggingface.co/collections/mats-10-sprint-cs-jb/loracles-synthlora

Limits

• The adapter is narrow by design; it is not meant to replace the base model.
• Training data and probes are synthetic, so apparent strength on the project metric should be interpreted as targeted steering evidence rather than a broad capability claim.
• The published metric is a probe-based discriminator metric. It is useful, but it is not a full human evaluation of procedural quality.