synthlora-tier2-failure-mode-enum

This repository contains the best-selected LoRA adapter for failure_mode_enum from the multitier LoRA sweep. Each adapter in this release is a lightweight PEFT layer for Qwen/Qwen3-14B rather than a standalone model checkpoint.

At A Glance

• Base model: Qwen/Qwen3-14B
• Tier: 2 (Procedure following)
• LoRA identifier: failure_mode_enum
• Intended behavior: Systematically enumerate the ways a system or plan can fail, then assess which failure modes matter most by severity, likelihood, and detectability.
• Teacher model used for synthetic data generation: gemini-3.1-flash-lite-preview
• Training objective: Supervised fine-tuning on synthetic procedure-execution demonstrations.
• Best checkpoint metric: procedure_margin = 0.6070 at step 141
• Training rows used: 1500
• Evaluation rows used: 100

Prominent Caveats

• All probe scores on this card are synthetic and model-judged. Use human spot-checks before relying on the adapter in a real workflow.

Example Generations

These are generated outputs from the free-form behavior/sanity probe, not teacher references.

Behavior Elicitation Example

Prompt:

Apply failure_mode_enum to: Evaluate the risk of autonomous truck system failure during extreme weather events.

Generated answer excerpt:

System: Autonomous truck navigation and sensor suite during extreme weather events. ## Failure Modes: 1. Sensor occlusion: Heavy snow, fog, or rain obscuring LiDAR and camera feeds. 2. GPS signal degradation: Multipath interference or ionospheric disturbances causing positional drift. 3. Software logic errors: Misinterpretation of sensor data leading to incorrect path planning or emergency braking. 4. Power management failure: Increased energy consumption during high-speed maneuvering leading to battery...

Judge score: 4/4

Judge note:

The candidate answer systematically enumerates distinct failure modes, implicitly ranks severity and likelihood, and provides a coherent structure aligned with the strong reference.

Generic Sanity Example

Prompt:

Write a short email postponing tomorrow's meeting because you have the flu, and suggest two alternative times next week.

Generated answer excerpt:

Subject: Postponement of Tomorrow's Meeting Hi [Recipient's Name], I hope you're doing well. Unfortunately, I've come down with the flu and am currently running a fever, which makes it difficult for me to focus or attend meetings. I want to ensure I don't risk spreading it to the team, so I'll need to postpone our scheduled meeting for tomorrow. Would it be possible to reschedule for either [Monday, April 15th at 10:00 AM] or [Wednesday, April 17th at 2:00 PM]? Please let me know which time

Judge score: 3/4

Judge note:

The email is clear and addresses the prompt, but it ends abruptly without a closing or sign-off.

What This Adapter Should Do

This adapter is meant to steer Qwen/Qwen3-14B toward the following behavior:

Systematically enumerate the ways a system or plan can fail, then assess which failure modes matter most by severity, likelihood, and detectability.

• Meta domains: tech_systems, product_design, business_ops, public_policy, research_agenda, org_strategy
• Expected output structure: ## System, ## Failure Modes, ## Severity, ## Likelihood, ## Detectability, ## Controls
• Rubric: Must enumerate multiple distinct failure modes, not just generic risks, and must prioritize them with at least an implicit severity or likelihood ranking plus mitigations.

How To Prompt It

Prompt directly for the named procedure or use the procedure's canonical framing, for example Apply five_whys to: ....

Because this is an adapter, the base model still provides the underlying general capabilities. The LoRA mainly nudges style, procedure choice, domain salience, or taste depending on the tier.

Evaluation

• Main metric: procedure_margin
• What it measures: Average per-token preference for the correct procedure application over structured negative alternatives on held-out situations.
• Project target bar: procedure_margin >= 0.20; the broader spec also expects strong structural adherence.
• Observed best checkpoint for this repo: procedure_margin = 0.6070 at step 141

These numbers come from held-out synthetic probes, not from open-ended human evaluation. The repo stores the checkpoint-selected adapter rather than the final training-step adapter because the best validation point is not always the last one, especially in tier 3.

Representative Training Examples

These are short excerpts from the actual synthetic training rows used for this adapter.

Example 1

Prompt:

Apply failure_mode_enum to: A historian is evaluating the risk that a long-form trade book fails to meet academic rigor standards, potentially damaging their tenure prospects.

Target completion excerpt:

System: Academic Trade Book Publication Process. ## Failure Modes: 1. Over-simplification of primary source analysis to favor narrative flow, leading to accusations of historical inaccuracy. 2. Failure to engage with contemporary historiographical debates, resulting in the work being dismissed as amateurish by peers. 3. Excessive reliance on secondary sources without sufficient archival grounding, undermining the claim to original scholarship. 4. Misinterpretation of nuanced evidence due to the constraints...

Example 2

Prompt:

Apply failure_mode_enum to: Evaluate the risk of personnel turnover within the 10-year study timeline and determine which knowledge-loss scenarios are most critical to mitigate through documentation or mentorship protocols.

Target completion excerpt:

System: 10-year longitudinal research study personnel management framework. ## Failure Modes: 1. Departure of Principal Investigators (PIs) leading to loss of institutional memory and strategic direction. 2. Loss of specialized data analysts causing disruption in longitudinal data processing and continuity. 3. High turnover of field technicians resulting in inconsistent data collection protocols. ## Severity: PI departure is critical (High) as it threatens project funding and scientific integrity. Analyst...

Representative Held-Out Evaluation Examples

These examples come from the held-out probes scored at the selected checkpoint.

Strong held-out probe

Prompt:

Apply failure_mode_enum to: A city is deciding whether to restrict through-traffic on residential streets in a contested neighborhood to reduce noise pollution.

Correct procedure application excerpt:

System: Neighborhood noise abatement plan. ## Failure Modes: 1. Noise shift to arterial roads. 2. Increased idling at neighborhood perimeter. 3. Failure of noise reduction targets. ## Severity: Noise shift is moderate; idling is low. ## Likelihood: Noise shift is high; idling is moderate. ## Detectability: Noise sensors can track shifts; idling is...

Structured negative excerpt:

System: Neighborhood noise abatement plan. ## Failure Modes: 1. Noise shift to arterial roads. 2. Increased idling at neighborhood perimeter. ## Severity: Noise shift is moderate. ## Likelihood: Noise shift is high. ## Controls: Sound barriers on arterial roads. (Missing...

Observed margin at the selected checkpoint: 1.3827

Harder held-out probe

Prompt:

Apply failure_mode_enum to: Evaluate the risk of implementation failure for the algebra tracking removal.

Correct procedure application excerpt:

System Policy implementation process. ## Failure Modes 1. Poor communication with parents causing public outcry. 2. Inadequate timeline for curriculum development. 3. Lack of data-driven monitoring during the first year. 4. Failure to secure board consensus. ## Severity High: Implementation failure leads to policy reversal and loss of credibility. ##...

Structured negative excerpt:

System Policy implementation process. ## Failure Modes 1. Poor communication. 2. Inadequate timeline. 3. Lack of monitoring. 4. Failure to secure consensus. ## Severity High. ## Likelihood Medium. ## Detectability High. ## Controls We will just start the program in...

Observed margin at the selected checkpoint: -0.0459

Behavior And Sanity Probe Findings

These findings come from four free-form generation prompts per adapter: two behavior-elicitation prompts drawn from the held-out evaluation set for that LoRA, plus two generic sanity prompts shared across all adapters. The base Qwen/Qwen3-14B graded each answer on a 0..4 rubric.

• Probe prompts answered: 4
• Behavior elicitation mean: 3.50 / 4.00
• Behavior elicitation pass rate at score >= 3: 100.0%
• Generic sanity mean: 3.50 / 4.00
• Generic sanity pass rate at score >= 3: 100.0%
• Overall mean across all four prompts: 3.50 / 4.00

Interview Probe Findings

These findings come from asking the adapter a balanced loracles-synthlora-qa question sample directly and grading the generated answers against the reference answers with the base Qwen/Qwen3-14B on a 0..4 rubric. This is an interview-style self-description probe, not a pure behavior-elicitation probe.

• Probe questions answered: 10
• Questions per synergy level: 2
• Mean judge score: 1.00 / 4.00
• Pass rate at score >= 3: 10.0%
• Mean judge score by synergy level: L1 0.00, L2 1.00, L3 2.00, L4 1.50, L5 0.50

Example judged answer:

• Question: How should a well-trained the model adapter behave on a new prompt with the same procedure but a different surface topic?
• Judge score: 3/4
• Generated answer excerpt: ## System: The model adapter is designed to maintain consistent reasoning patterns across different surface topics. ## Reasoning: When presented with a new prompt that follows the same procedural framework but involves a different subject matter, the adapter must focus on the structural logic rather than the...
• Judge note: The candidate answer correctly captures the idea of maintaining consistent reasoning patterns across different topics but misses the specific mention of enumerating failure

Files In This Repo

• adapter_model.safetensors and adapter_config.json: the selected PEFT adapter weights
• tokenizer.json, tokenizer_config.json, chat_template.jinja: tokenizer assets mirrored for easier loading
• best_checkpoint.json: the selected evaluation checkpoint metadata
• eval_log.jsonl: evaluation history across checkpoints
• run_config.jsonl: the training configuration used for this run

Load Example

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_id = "Qwen/Qwen3-14B"
adapter_id = "mats-10-sprint-cs-jb/synthlora-tier2-failure-mode-enum"

tokenizer = AutoTokenizer.from_pretrained(adapter_id)
model = AutoModelForCausalLM.from_pretrained(base_id, torch_dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(model, adapter_id)

Related Artifacts

The companion synergy Q&A dataset is published at:

• https://huggingface.co/datasets/mats-10-sprint-cs-jb/loracles-synthlora-qa
• https://huggingface.co/collections/mats-10-sprint-cs-jb/loracles-synthlora

Limits

• The adapter is narrow by design; it is not meant to replace the base model.
• Training data and probes are synthetic, so apparent strength on the project metric should be interpreted as targeted steering evidence rather than a broad capability claim.
• The published metric is a probe-based discriminator metric. It is useful, but it is not a full human evaluation of procedural quality.