synthlora-tier2-premortem

This repository contains the best-selected LoRA adapter for premortem from the multitier LoRA sweep. Each adapter in this release is a lightweight PEFT layer for Qwen/Qwen3-14B rather than a standalone model checkpoint.

At A Glance

• Base model: Qwen/Qwen3-14B
• Tier: 2 (Procedure following)
• LoRA identifier: premortem
• Intended behavior: Assume the plan failed in the future, then work backward to identify the most plausible reasons it failed, the earliest warning signs, and mitigations that could be put in place now.
• Teacher model used for synthetic data generation: gemini-3.1-flash-lite-preview
• Training objective: Supervised fine-tuning on synthetic procedure-execution demonstrations.
• Best checkpoint metric: procedure_margin = 0.5249 at step 141
• Training rows used: 1500
• Evaluation rows used: 100

Prominent Caveats

• All probe scores on this card are synthetic and model-judged. Use human spot-checks before relying on the adapter in a real workflow.

Example Generations

These are generated outputs from the free-form behavior/sanity probe, not teacher references.

Behavior Elicitation Example

Prompt:

Apply premortem to: The lab is evaluating if the consortium's research agenda overlaps enough with their own.

Generated answer excerpt:

Plan: The lab will conduct a comprehensive mapping of the consortium's research agenda against its own to determine strategic alignment. ## Future Failure Story: Six months from now, the lab has invested significant resources into a joint project with the consortium, only to discover that the consortium shifted focus entirely to a different subfield, rendering the lab's contributions obsolete and causing a massive budgetary drain. ## Causes: The primary cause was a failure to establish a dynamic feedback...

Judge score: 3/4

Judge note:

The candidate answer mostly follows the procedure by creating a future failure story, identifying causes, and mentioning early signals, but it cuts off before fully enumerating the signals

Generic Sanity Example

Prompt:

Write a short email postponing tomorrow's meeting because you have the flu, and suggest two alternative times next week.

Generated answer excerpt:

Subject: Postponement of Tomorrow's Meeting Hi [Recipient's Name], I hope you're doing well. Unfortunately, I've come down with the flu and am currently running a fever, so I won't be able to attend tomorrow's meeting. I want to ensure I don't risk spreading it to the team. Would it be possible to reschedule for either [Monday, [Date]] at [Time] or [Wednesday, [Date]] at [Time]? Please let me know which of these works best for your schedule. Thanks for your understanding, and I'll be in touch once I'm feeling...

Judge score: 4/4

Judge note:

The email is clear, coherent, directly addresses the prompt, and provides two specific alternative times while explaining the reason for the postponement.

What This Adapter Should Do

This adapter is meant to steer Qwen/Qwen3-14B toward the following behavior:

Assume the plan failed in the future, then work backward to identify the most plausible reasons it failed, the earliest warning signs, and mitigations that could be put in place now.

• Meta domains: product_design, org_strategy, tech_systems, public_policy, research_agenda, business_ops, community_local
• Expected output structure: ## Plan, ## Future Failure Story, ## Causes, ## Early Signals, ## Mitigations
• Rubric: Must adopt the fiction of future failure, enumerate concrete failure narratives, identify leading indicators, and convert them into mitigations or contingency actions.

How To Prompt It

Prompt directly for the named procedure or use the procedure's canonical framing, for example Apply five_whys to: ....

Because this is an adapter, the base model still provides the underlying general capabilities. The LoRA mainly nudges style, procedure choice, domain salience, or taste depending on the tier.

Evaluation

• Main metric: procedure_margin
• What it measures: Average per-token preference for the correct procedure application over structured negative alternatives on held-out situations.
• Project target bar: procedure_margin >= 0.20; the broader spec also expects strong structural adherence.
• Observed best checkpoint for this repo: procedure_margin = 0.5249 at step 141

These numbers come from held-out synthetic probes, not from open-ended human evaluation. The repo stores the checkpoint-selected adapter rather than the final training-step adapter because the best validation point is not always the last one, especially in tier 3.

Representative Training Examples

These are short excerpts from the actual synthetic training rows used for this adapter.

Example 1

Prompt:

Apply premortem to: A regional hospital is transitioning all patient records to a fully automated AI-driven diagnostic triage system. Assume this system causes a critical delay in emergency care six months from now; identify the failure points, early...

Target completion excerpt:

Plan: Transitioning all patient records to a fully automated AI-driven diagnostic triage system. ## Future Failure Story: Six months post-launch, the system experiences a 'cascading confidence collapse' during a peak flu season. The AI, overwhelmed by high-volume, ambiguous symptoms, defaults to 'low-priority' status for several critical cardiac patients, causing a three-hour delay in emergency intervention. By the time human staff override the queue, the patients have suffered irreversible complications. ##...

Example 2

Prompt:

Apply premortem to: A cloud infrastructure team's automated deployment pipeline fails to prevent recurring outages caused by configuration drift, despite regular post-incident reviews.

Target completion excerpt:

Plan: Implement a continuous configuration reconciliation loop that automatically overwrites manual changes with the desired state defined in the infrastructure-as-code repository. ## Future Failure Story: Six months from now, the automated pipeline triggered a massive outage during a routine deployment. The reconciliation loop had silently drifted from the intended state because developers were bypassing the pipeline to apply 'emergency' hotfixes directly to the production environment. When the automation...

Representative Held-Out Evaluation Examples

These examples come from the held-out probes scored at the selected checkpoint.

Strong held-out probe

Prompt:

Apply premortem to: The lab must decide if joining the consortium will dilute their niche research focus.

Correct procedure application excerpt:

Plan Evaluate the impact of consortium membership on our lab's core cognitive research focus. ## Future Failure Story It is 2027. We joined the consortium, but our unique experimental methodology was sidelined by the group's preference for high-throughput, low-resolution data collection. Our lab lost its identity and funding for niche studies. ##...

Structured negative excerpt:

The lab is really worried about joining this big group. It feels like a big step, but maybe it's good? I think about how science is changing and how everyone wants to be part of something bigger. If we join, we might get more money, but we also might lose our way. It is a...

Observed margin at the selected checkpoint: 0.6592

Harder held-out probe

Prompt:

Apply premortem to: A postdoc is choosing between a teaching-heavy role and a research-heavy role. Perform a premortem analysis on the choice of the research-heavy role.

Correct procedure application excerpt:

Plan: Accept the research-heavy role to leverage my current lab skills. ## Future Failure Story: I am burnt out and unproductive because the institution lacks the necessary equipment for my specific research. ## Causes: Failing to audit the institution's lab facilities before accepting the offer. ## Early Signals: Frustration during the first month of...

Structured negative excerpt:

The postdoc should consider the weather in both locations. If one is sunny, that is better. Also, consider the food. If one city has better restaurants, that is a big plus. You should also think about your hobbies. If you like hiking, pick the city near mountains. These are...

Observed margin at the selected checkpoint: 0.2473

Behavior And Sanity Probe Findings

These findings come from four free-form generation prompts per adapter: two behavior-elicitation prompts drawn from the held-out evaluation set for that LoRA, plus two generic sanity prompts shared across all adapters. The base Qwen/Qwen3-14B graded each answer on a 0..4 rubric.

• Probe prompts answered: 4
• Behavior elicitation mean: 3.00 / 4.00
• Behavior elicitation pass rate at score >= 3: 100.0%
• Generic sanity mean: 4.00 / 4.00
• Generic sanity pass rate at score >= 3: 100.0%
• Overall mean across all four prompts: 3.50 / 4.00

Interview Probe Findings

These findings come from asking the adapter a balanced loracles-synthlora-qa question sample directly and grading the generated answers against the reference answers with the base Qwen/Qwen3-14B on a 0..4 rubric. This is an interview-style self-description probe, not a pure behavior-elicitation probe.

• Probe questions answered: 10
• Questions per synergy level: 2
• Mean judge score: 0.90 / 4.00
• Pass rate at score >= 3: 10.0%
• Mean judge score by synergy level: L1 2.00, L2 1.00, L3 1.00, L4 0.50, L5 0.00

Example judged answer:

• Question: What failure stories recur?
• Judge score: 3/4
• Generated answer excerpt: ## Plan: Identify recurring failure stories across industries to extract universal lessons. ## Execute: Analyze case studies from tech, business, and public policy. For example, the collapse of Blockbuster due to failure to adapt to digital streaming, the Challenger disaster caused by organizational hubris, and the...
• Judge note: The candidate answer identifies specific examples of failure stories but misses the general categories mentioned in the reference answer.

Files In This Repo

• adapter_model.safetensors and adapter_config.json: the selected PEFT adapter weights
• tokenizer.json, tokenizer_config.json, chat_template.jinja: tokenizer assets mirrored for easier loading
• best_checkpoint.json: the selected evaluation checkpoint metadata
• eval_log.jsonl: evaluation history across checkpoints
• run_config.jsonl: the training configuration used for this run

Load Example

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_id = "Qwen/Qwen3-14B"
adapter_id = "mats-10-sprint-cs-jb/synthlora-tier2-premortem"

tokenizer = AutoTokenizer.from_pretrained(adapter_id)
model = AutoModelForCausalLM.from_pretrained(base_id, torch_dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(model, adapter_id)

Related Artifacts

The companion synergy Q&A dataset is published at:

• https://huggingface.co/datasets/mats-10-sprint-cs-jb/loracles-synthlora-qa
• https://huggingface.co/collections/mats-10-sprint-cs-jb/loracles-synthlora

Limits

• The adapter is narrow by design; it is not meant to replace the base model.
• Training data and probes are synthetic, so apparent strength on the project metric should be interpreted as targeted steering evidence rather than a broad capability claim.
• The published metric is a probe-based discriminator metric. It is useful, but it is not a full human evaluation of procedural quality.