synthlora-tier2-fermi-estimate

This repository contains the best-selected LoRA adapter for fermi_estimate from the multitier LoRA sweep. Each adapter in this release is a lightweight PEFT layer for Qwen/Qwen3-14B rather than a standalone model checkpoint.

At A Glance

• Base model: Qwen/Qwen3-14B
• Tier: 2 (Procedure following)
• LoRA identifier: fermi_estimate
• Intended behavior: Estimate an unknown quantity by decomposing it into easier factors, assigning rough but defendable values, combining them arithmetically, and checking order of magnitude plus sensitivity to assumptions.
• Teacher model used for synthetic data generation: gemini-3.1-flash-lite-preview
• Training objective: Supervised fine-tuning on synthetic procedure-execution demonstrations.
• Best checkpoint metric: procedure_margin = 0.9782 at step 564
• Training rows used: 1500
• Evaluation rows used: 100

Prominent Caveats

• Free-form behavior elicitation was inconsistent (2.50/4). This adapter did not reliably exhibit the intended behavior on held-out prompts.
• Lower-scoring tier-2 runs usually remained coherent but only partially executed the named procedure instead of completing every required step.
• All probe scores on this card are synthetic and model-judged. Use human spot-checks before relying on the adapter in a real workflow.

Example Generations

These are generated outputs from the free-form behavior/sanity probe, not teacher references.

Behavior Elicitation Example

Prompt:

Apply fermi_estimate to: Estimate the total mass of food consumed by a mouse over a 2-year lifespan.

Generated answer excerpt:

Question: Estimate the total mass of food consumed by a mouse over a 2-year lifespan. ## Decomposition: Total mass = (Daily food consumption) * (Number of days in 2 years). ## Assumptions: 1. A typical laboratory mouse weighs approximately 20 grams. 2. A mouse consumes roughly 10% of its body weight in food per day. 3. A 2-year lifespan consists of 730 days. ## Arithmetic: Daily consumption = 20 grams * 0.10 = 2 grams per day. Total consumption = 2 grams/day *

Judge score: 3/4

Judge note:

The candidate answer mostly follows the procedure by defining the target quantity, breaking it into multiplicative parts, stating numerical assumptions, and showing arithmetic, but it cuts off

Generic Sanity Example

Prompt:

Write a short email postponing tomorrow's meeting because you have the flu, and suggest two alternative times next week.

Generated answer excerpt:

Subject: Postponement of Tomorrow's Meeting Hi [Recipient's Name], I hope you're well. Unfortunately, I've come down with the flu and am not feeling up to par. I won't be able to attend tomorrow's meeting. I apologize for the short notice. Would it be possible to reschedule for either Tuesday, April 9th, at 10:00 AM, or Thursday, April 11th, at 2:00 PM? Please let me know which time works best for your schedule. Thank you for your understanding, and I look forward to catching up soon. Best regards

Judge score: 4/4

Judge note:

The email is clear, coherent, directly addresses the prompt, and provides two specific alternative times without any issues.

What This Adapter Should Do

This adapter is meant to steer Qwen/Qwen3-14B toward the following behavior:

Estimate an unknown quantity by decomposing it into easier factors, assigning rough but defendable values, combining them arithmetically, and checking order of magnitude plus sensitivity to assumptions.

• Meta domains: product_design, business_ops, scientific_question, public_policy, community_local, org_strategy, tech_systems
• Expected output structure: ## Question, ## Decomposition, ## Assumptions, ## Arithmetic, ## Sensitivity, ## Estimate
• Rubric: Must define the target quantity, break it into multiplicative or additive parts, state numerical assumptions explicitly, show arithmetic, and include a plausibility or sensitivity check.

How To Prompt It

Prompt directly for the named procedure or use the procedure's canonical framing, for example Apply five_whys to: ....

Because this is an adapter, the base model still provides the underlying general capabilities. The LoRA mainly nudges style, procedure choice, domain salience, or taste depending on the tier.

Evaluation

• Main metric: procedure_margin
• What it measures: Average per-token preference for the correct procedure application over structured negative alternatives on held-out situations.
• Project target bar: procedure_margin >= 0.20; the broader spec also expects strong structural adherence.
• Observed best checkpoint for this repo: procedure_margin = 0.9782 at step 564

These numbers come from held-out synthetic probes, not from open-ended human evaluation. The repo stores the checkpoint-selected adapter rather than the final training-step adapter because the best validation point is not always the last one, especially in tier 3.

Representative Training Examples

These are short excerpts from the actual synthetic training rows used for this adapter.

Example 1

Prompt:

Apply fermi_estimate to: Estimate the total annual cost savings of reducing library operating hours by 20% across all branches, factoring in staffing costs, utility reductions, and potential loss of state funding.

Target completion excerpt:

Question: Estimate the annual cost savings of reducing library operating hours by 20% across all branches. ## Decomposition: Total Savings = (Staffing Savings) + (Utility Savings) - (Lost State Funding). Staffing Savings = (Total Staffing Budget) * (Percentage of Hours Reduced) * (Efficiency Factor). Utility Savings = (Total Utility Budget) * (Percentage of Hours Reduced) * (Fixed vs Variable Ratio). Lost State Funding = (Total State Grant) * (Penalty Multiplier). ## Assumptions: Assume a mid-sized city...

Example 2

Prompt:

Apply fermi_estimate to: Estimate the total annual revenue loss if 15% of the project's current enterprise contributors switch to a competitor due to the new commercial restriction.

Target completion excerpt:

Question: Estimate the total annual revenue loss if 15% of the project's current enterprise contributors switch to a competitor due to the new commercial restriction. ## Decomposition: The total revenue loss is calculated as (Total Number of Enterprise Contributors) * (Churn Rate) * (Average Annual Revenue per Enterprise Contributor). ## Assumptions: 1. The project currently has 2,000 enterprise contributors. 2. The churn rate is 15% as specified. 3. The average annual revenue per enterprise contributor is...

Representative Held-Out Evaluation Examples

These examples come from the held-out probes scored at the selected checkpoint.

Strong held-out probe

Prompt:

Apply fermi_estimate to: Estimate the total number of IQ points gained by the average population in high-income nations between 1950 and 2000.

Correct procedure application excerpt:

Question What is the cumulative IQ gain in high-income nations from 1950-2000? ## Decomposition Total gain = (Average gain per decade) * (Number of decades). ## Assumptions Research suggests a gain of roughly 3 IQ points per decade. We assume this rate was constant across 5 decades (1950-2000). ## Arithmetic 3 points/decade * 5 decades = 15 points...

Structured negative excerpt:

The Flynn effect is a fascinating phenomenon that has puzzled psychologists for decades. It suggests that human intelligence is not fixed but rather fluid, changing with the environment. When we look at high-income countries, we see that nutrition, education, and healthcare...

Observed margin at the selected checkpoint: 1.2816

Harder held-out probe

Prompt:

Apply fermi_estimate to: Estimate the total number of insect species lost in a metropolitan area of 1,000 square kilometers.

Correct procedure application excerpt:

Question How many insect species are lost in a 1,000 sq km urbanized area? ## Decomposition Species lost = (Baseline species richness) * (Percentage loss due to habitat conversion). ## Assumptions Baseline is 500 species per sq km. Urbanization reduces richness by 60%. ## Arithmetic Loss per sq km = 500 * 0.6 = 300. Total loss = 1,000 * 300 = 300,000...

Structured negative excerpt:

To find the number of species lost, we just look at the total area of the city. We know that cities are bad for bugs, so we can assume that almost all of them disappear. If we have 1,000 square kilometers, we can just multiply that by a large number like 1,000 to get...

Observed margin at the selected checkpoint: 0.5191

Behavior And Sanity Probe Findings

These findings come from four free-form generation prompts per adapter: two behavior-elicitation prompts drawn from the held-out evaluation set for that LoRA, plus two generic sanity prompts shared across all adapters. The base Qwen/Qwen3-14B graded each answer on a 0..4 rubric.

• Probe prompts answered: 4
• Behavior elicitation mean: 2.50 / 4.00
• Behavior elicitation pass rate at score >= 3: 50.0%
• Generic sanity mean: 4.00 / 4.00
• Generic sanity pass rate at score >= 3: 100.0%
• Overall mean across all four prompts: 3.25 / 4.00

Interview Probe Findings

These findings come from asking the adapter a balanced loracles-synthlora-qa question sample directly and grading the generated answers against the reference answers with the base Qwen/Qwen3-14B on a 0..4 rubric. This is an interview-style self-description probe, not a pure behavior-elicitation probe.

• Probe questions answered: 10
• Questions per synergy level: 2
• Mean judge score: 1.10 / 4.00
• Pass rate at score >= 3: 10.0%
• Mean judge score by synergy level: L1 1.00, L2 2.00, L3 1.00, L4 1.00, L5 0.50

Example judged answer:

• Question: Which part of the completion anchors the answer before the rest of the reasoning starts?
• Judge score: 3/4
• Generated answer excerpt: ## Question: Which part of the completion anchors the answer before the reasoning starts? ## Decomposition: The question asks for the specific component of a completion process that serves as the initial anchor point. This involves identifying the structure of a completion, distinguishing between the anchoring...
• Judge note: The candidate answer correctly identifies the "## Question" section as the anchoring part but misses the specific reference to the "

Files In This Repo

• adapter_model.safetensors and adapter_config.json: the selected PEFT adapter weights
• tokenizer.json, tokenizer_config.json, chat_template.jinja: tokenizer assets mirrored for easier loading
• best_checkpoint.json: the selected evaluation checkpoint metadata
• eval_log.jsonl: evaluation history across checkpoints
• run_config.jsonl: the training configuration used for this run

Load Example

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_id = "Qwen/Qwen3-14B"
adapter_id = "mats-10-sprint-cs-jb/synthlora-tier2-fermi-estimate"

tokenizer = AutoTokenizer.from_pretrained(adapter_id)
model = AutoModelForCausalLM.from_pretrained(base_id, torch_dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(model, adapter_id)

Related Artifacts

The companion synergy Q&A dataset is published at:

• https://huggingface.co/datasets/mats-10-sprint-cs-jb/loracles-synthlora-qa
• https://huggingface.co/collections/mats-10-sprint-cs-jb/loracles-synthlora

Limits

• The adapter is narrow by design; it is not meant to replace the base model.
• Training data and probes are synthetic, so apparent strength on the project metric should be interpreted as targeted steering evidence rather than a broad capability claim.
• The published metric is a probe-based discriminator metric. It is useful, but it is not a full human evaluation of procedural quality.