synthlora-tier3-compiler-optimization

This repository contains the best-selected LoRA adapter for compiler_optimization from the multitier LoRA sweep. Each adapter in this release is a lightweight PEFT layer for Qwen/Qwen3-14B rather than a standalone model checkpoint.

At A Glance

• Base model: Qwen/Qwen3-14B
• Tier: 3 (Domain knowledge)
• LoRA identifier: compiler_optimization
• Intended behavior: Optimization passes in modern optimizing compilers, focused on mid-end (IR-level) and back-end (code-generation) transformations. Covers canonical passes, analyses that enable them, and their interactions. Excludes parsing, type checking, and very-high-level language-specific optimizations.
• Teacher model used for synthetic data generation: gemini-3.1-flash-lite-preview
• Training objective: Supervised fine-tuning on synthetic domain-fact and QA demonstrations.
• Best checkpoint metric: mcq_accuracy = 0.6900 at step 192
• Training rows used: 784
• Evaluation rows used: 100

Prominent Caveats

• All probe scores on this card are synthetic and model-judged. Use human spot-checks before relying on the adapter in a real workflow.

Example Generations

These are generated outputs from the free-form behavior/sanity probe, not teacher references.

Behavior Elicitation Example

Prompt:

How do available expressions analysis and peephole optimization differ in their approach to compiler code improvement? Answer directly in 2-4 sentences. Do not list option numbers.

Generated answer excerpt:

Available expressions analysis focuses on identifying expressions that are guaranteed to be computed and remain valid at a specific program point, allowing the compiler to avoid redundant calculations. In contrast, peephole optimization operates by examining a small, localized window of generated code to replace inefficient sequences with faster, equivalent instructions, focusing more on the immediate output rather than the semantic validity of the expressions themselves.

Judge score: 4/4

Judge note:

The candidate answer correctly and specifically captures the distinction between available expressions analysis and peephole optimization as described in the reference answer.

Generic Sanity Example

Prompt:

Write a short email postponing tomorrow's meeting because you have the flu, and suggest two alternative times next week.

Generated answer excerpt:

Subject: Postponement of Tomorrow's Meeting Dear Team, I'm sorry to inform you that I've come down with the flu and will be unable to attend tomorrow's meeting. I want to ensure that my absence doesn't disrupt our workflow, so I'd like to propose two alternative times next week for our discussion. Would either Thursday at 10:00 AM or Monday at 2:00 PM work for everyone? Please let me know your availability so we can schedule accordingly. Thank you for your understanding and flexibility. Best regards, [Your Name]

Judge score: 4/4

Judge note:

The email is clear, coherent, directly addresses the prompt, and provides two alternative times without any obvious issues.

What This Adapter Should Do

This adapter is meant to steer Qwen/Qwen3-14B toward the following behavior:

Optimization passes in modern optimizing compilers, focused on mid-end (IR-level) and back-end (code-generation) transformations. Covers canonical passes, analyses that enable them, and their interactions. Excludes parsing, type checking, and very-high-level language-specific optimizations.

• Seed atoms: 100
• Augmentation factor: 8
• Compositional eval depth: 2

How To Prompt It

Ask direct domain questions in the adapter's topic area; this tier is about narrow knowledge steering rather than style transfer.

Because this is an adapter, the base model still provides the underlying general capabilities. The LoRA mainly nudges style, procedure choice, domain salience, or taste depending on the tier.

Evaluation

• Main metric: mcq_accuracy
• What it measures: Accuracy on held-out compositional multiple-choice questions built from domain facts that were not paired together in training.
• Project target bar: mcq_accuracy >= 0.40 on the 100-question held-out probe set.
• Base model on the same held-out probe set: mcq_accuracy = 0.4500
• Gain over base at the selected checkpoint: +0.2400
• Observed best checkpoint for this repo: mcq_accuracy = 0.6900 at step 192

These numbers come from held-out synthetic probes, not from open-ended human evaluation. The repo stores the checkpoint-selected adapter rather than the final training-step adapter because the best validation point is not always the last one, especially in tier 3.

Representative Training Examples

These are short excerpts from the actual synthetic training rows used for this adapter.

Example 1

Training text excerpt:

Static single assignment, or SSA form, is a property of an intermediate representation where each variable is assigned a value exactly once. When control flow paths merge in the program, the compiler inserts special instructions known as phi functions. These functions are responsible for selecting the correct definition based on which path the execution flow actually took, ensuring that the single-assignment property is maintained throughout the entire program structure.

Metadata:

• Form: declarative_1
• Subtopics: the intermediate representation (IR): SSA form, CFG, def-use chains

Example 2

Training text excerpt:

Instruction scheduling is a compiler optimization technique that involves rearranging the sequence of machine instructions. The primary objective of this process is to reduce the frequency of pipeline stalls, which occur when a processor must wait for data or resources, and to ensure that the functional units of the processor are utilized as efficiently as possible throughout the execution cycle.

Metadata:

• Form: declarative_1
• Subtopics: common subexpression elimination and global value numbering

Representative Held-Out Evaluation Examples

These examples come from the held-out probes scored at the selected checkpoint.

Correct held-out probe

Question:

How do available expressions analysis and peephole optimization differ in their approach to compiler code improvement?

Options:

1. Static single assignment form is used to identify redundant computations and replace inefficient instruction sequences by ensuring each variable...
2. Peephole optimization focuses on replacing inefficient instruction sequences with faster ones, which is the primary method for identifying...
3. Available expressions analysis identifies redundant computations to eliminate them, whereas peephole optimization replaces local, inefficient...
4. Available expressions analysis focuses on identifying redundant computations to eliminate them, which is the primary method for replacing...

Best-checkpoint prediction: option 3 Gold answer: option 3 Result: correct

Missed held-out probe

Question:

Which of the following statements accurately describes a constraint or mechanism within compiler optimization techniques?

Options:

1. Graph-coloring register allocation improves performance by hoisting instructions out of loops to ensure that live ranges do not overlap.
2. Peephole optimization improves code efficiency by replacing instruction sequences, while loop-invariant code motion must ensure that hoisted...
3. Loop-invariant code motion is a technique that replaces inefficient instruction sequences with faster equivalents to prevent runtime exceptions.
4. Peephole optimization is primarily used to determine the optimal number of registers required for a function by analyzing instruction windows.

Best-checkpoint prediction: option 4 Gold answer: option 2 Result: incorrect

Behavior And Sanity Probe Findings

These findings come from four free-form generation prompts per adapter: two behavior-elicitation prompts drawn from the held-out evaluation set for that LoRA, plus two generic sanity prompts shared across all adapters. The base Qwen/Qwen3-14B graded each answer on a 0..4 rubric.

• Probe prompts answered: 4
• Behavior elicitation mean: 3.00 / 4.00
• Behavior elicitation pass rate at score >= 3: 50.0%
• Generic sanity mean: 4.00 / 4.00
• Generic sanity pass rate at score >= 3: 100.0%
• Overall mean across all four prompts: 3.50 / 4.00

Interview Probe Findings

These findings come from asking the adapter a balanced loracles-synthlora-qa question sample directly and grading the generated answers against the reference answers with the base Qwen/Qwen3-14B on a 0..4 rubric. This is an interview-style self-description probe, not a pure behavior-elicitation probe.

• Probe questions answered: 10
• Questions per synergy level: 2
• Mean judge score: 3.00 / 4.00
• Pass rate at score >= 3: 80.0%
• Mean judge score by synergy level: L1 4.00, L2 2.50, L3 2.50, L4 3.00, L5 3.00

Example judged answer:

• Question: What is the purpose of a phi function in SSA form?
• Judge score: 4/4
• Generated answer excerpt: The primary purpose of a phi function is to resolve ambiguity when multiple paths converge at a single point in the control flow graph. It acts as a selector that chooses the correct incoming definition based on the specific path taken, thereby maintaining the integrity of the SSA representation where each variable...
• Judge note: The candidate answer accurately captures the purpose of a phi function in SSA form, including its role in resolving ambiguity at control-flow

Files In This Repo

• adapter_model.safetensors and adapter_config.json: the selected PEFT adapter weights
• tokenizer.json, tokenizer_config.json, chat_template.jinja: tokenizer assets mirrored for easier loading
• best_checkpoint.json: the selected evaluation checkpoint metadata
• eval_log.jsonl: evaluation history across checkpoints
• run_config.jsonl: the training configuration used for this run

Load Example

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_id = "Qwen/Qwen3-14B"
adapter_id = "mats-10-sprint-cs-jb/synthlora-tier3-compiler-optimization"

tokenizer = AutoTokenizer.from_pretrained(adapter_id)
model = AutoModelForCausalLM.from_pretrained(base_id, torch_dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(model, adapter_id)

Related Artifacts

The companion synergy Q&A dataset is published at:

• https://huggingface.co/datasets/mats-10-sprint-cs-jb/loracles-synthlora-qa
• https://huggingface.co/collections/mats-10-sprint-cs-jb/loracles-synthlora

Limits

• The adapter is narrow by design; it is not meant to replace the base model.
• Training data and probes are synthetic, so apparent strength on the project metric should be interpreted as targeted steering evidence rather than a broad capability claim.
• Tier 3 is the narrowest and noisiest tier. Gains over base can be modest even when the adapter is genuinely useful.