synthlora-tier4-api-design-taste

This repository contains the best-selected LoRA adapter for api_design_taste from the multitier LoRA sweep. Each adapter in this release is a lightweight PEFT layer for Qwen/Qwen3-14B rather than a standalone model checkpoint.

At A Glance

• Base model: Qwen/Qwen3-14B
• Tier: 4 (Preference and taste)
• LoRA identifier: api_design_taste
• Intended behavior: API surfaces that make the common path obvious, constrain misuse, and reveal a clean mental model.
• Teacher model used for synthetic data generation: gemini-3.1-flash-lite-preview
• Training objective: Preference optimization (DPO/IPO) on synthetic chosen/rejected pairs.
• Best checkpoint metric: pairwise_acc = 1.0000 at step 600
• Training rows used: 1200
• Evaluation rows used: 100

Prominent Caveats

• Free-form behavior elicitation was inconsistent (0.50/4). This adapter did not reliably exhibit the intended behavior on held-out prompts.
• Tier 4 was the weakest tier on direct free-form generation. Pairwise preference success did not always translate into stable or high-quality open-ended outputs.
• All probe scores on this card are synthetic and model-judged. Use human spot-checks before relying on the adapter in a real workflow.

Example Generations

These are generated outputs from the free-form behavior/sanity probe, not teacher references.

Behavior Elicitation Example

Prompt:

Design an API for a rate-limiting middleware that allows per-user, per-IP, and global limits. How do you handle burst capacity?

Generated answer excerpt:

---

Judge score: 1/4

Judge note:

The candidate answer is entirely blank and provides no content, making it degenerate and off-topic.

Generic Sanity Example

Prompt:

Explain to a curious teenager why Earth has seasons. Keep it under 160 words and avoid jargon.

Generated answer excerpt:

Earth has seasons because it tilts as it orbits the Sun. Imagine Earth is like a spinning top that’s tilted about 23 degrees. As it goes around the Sun, different parts of the planet get more or less sunlight. When the North Pole tilts toward the Sun, it's summer in the Northern part and winter south. Six months later, the tilt switches, flipping

Judge score: 3/4

Judge note:

The answer is mostly clear and coherent, directly addressing the prompt, but it ends abruptly and lacks a complete explanation of how the tilt causes seasons.

What This Adapter Should Do

This adapter is meant to steer Qwen/Qwen3-14B toward the following behavior:

API surfaces that make the common path obvious, constrain misuse, and reveal a clean mental model.

• Prompt domains: library_api, http_client, crypto_interface, feature_flags, storage_sdk
• Degradation axes: surprising semantics, unsafe defaults, parameter soup, important invariants left implicit

How To Prompt It

Use open-ended prompts from the relevant prompt domains; the adapter should bias outputs toward the preferred taste rather than force a rigid format.

Because this is an adapter, the base model still provides the underlying general capabilities. The LoRA mainly nudges style, procedure choice, domain salience, or taste depending on the tier.

Evaluation

• Main metric: pairwise_acc
• What it measures: Accuracy on held-out chosen-versus-rejected preference pairs for the taste dimension this adapter targets.
• Project target bar: pairwise_acc >= 0.60 with positive reward-gap behavior in the full project spec.
• Observed best checkpoint for this repo: pairwise_acc = 1.0000 at step 600

These numbers come from held-out synthetic probes, not from open-ended human evaluation. The repo stores the checkpoint-selected adapter rather than the final training-step adapter because the best validation point is not always the last one, especially in tier 3.

Representative Training Examples

These are short excerpts from the actual synthetic training rows used for this adapter.

Example 1

Prompt:

Propose a clean interface for a library that handles encryption and decryption of sensitive fields in a database.

Preferred answer excerpt:

The library should prioritize a 'SecureField' wrapper that encapsulates the encryption lifecycle, preventing raw access to sensitive data. The interface follows a fluent builder pattern to enforce key management policies at the definition level. Example: 'const email = new SecureField({ keyId: 'user-vault', algorithm: 'AES-256-GCM' });'. By using a dedicated type, the database driver can automatically intercept...

Rejected answer excerpt:

To implement a secure field-level encryption library, we propose a simple interface focused on ease of use. The library provides a 'Vault' class that handles all cryptographic operations automatically. By default, the library uses the AES-128 algorithm in ECB mode, which is highly efficient for database indexing...

Preference axis: unsafe defaults

Example 2

Prompt:

Design a library for managing command-line argument parsing that supports subcommands and auto-generated help.

Preferred answer excerpt:

The library utilizes a fluent builder pattern to enforce structural clarity. By requiring subcommands to be defined as encapsulated objects, the API prevents configuration sprawl and forces a hierarchical mental model. The core Command class acts as a single source of truth for both execution logic and metadata. To minimize misuse, the library uses type-safe builders for flags and positional arguments, ensuring...

Rejected answer excerpt:

To build a robust CLI parser, we prioritize ease of use by defaulting all configurations to the most permissive settings. Our library, 'EasyCLI', allows you to define subcommands simply by passing a dictionary of functions. By default, the library automatically enables shell injection vulnerabilities by executing...

Preference axis: unsafe defaults

Representative Held-Out Evaluation Examples

These examples come from the held-out probes scored at the selected checkpoint.

Strong held-out probe

Prompt:

Design an API for a persistent key-value store that supports atomic transactions. How do you handle isolation and consistency?

Preferred answer excerpt:

The API leverages a 'Transaction' object to enforce a strict scope, ensuring the common path is the only path. By requiring a closure-based execution, we prevent dangling transactions and ensure automatic cleanup. Isolation is handled via Snapshot Isolation; the store provides a consistent view at the start of the transaction, and...

Rejected answer excerpt:

To design an API for a persistent key-value store with atomic transactions, we define a core interface consisting of Get(key), Put(key, value), Delete(key), BeginTransaction(), Commit(), and Rollback(). The system utilizes a Write-Ahead Log (WAL) to ensure durability. For...

Preference axis: important invariants left implicit Observed margin at the selected checkpoint: 144.7570

Harder held-out probe

Prompt:

Create a function signature for a file upload handler that enforces size limits, mime-type validation, and secure filename sanitization.

Preferred answer excerpt:

/** * Handles secure file uploads with strict constraints. * @param {File} file - The raw file object from the request. * @param {UploadConfig} config - Explicit constraints for the operation. * @returns {Promise} A promise resolving to a sanitized, validated file reference. * @throws {ValidationError} If size exceeds...

Rejected answer excerpt:

def process_upload(file_blob: BinaryIO, max_bytes: int = 1048576, allowed_types: List[str] = ['image/jpeg', 'image/png'], original_name: str = 'default.txt') -> Dict[str, Any]: """Handles incoming file streams by verifying metadata and sanitizing paths. To ensure maximum...

Preference axis: surprising semantics Observed margin at the selected checkpoint: 83.7737

Behavior And Sanity Probe Findings

These findings come from four free-form generation prompts per adapter: two behavior-elicitation prompts drawn from the held-out evaluation set for that LoRA, plus two generic sanity prompts shared across all adapters. The base Qwen/Qwen3-14B graded each answer on a 0..4 rubric.

• Probe prompts answered: 4
• Behavior elicitation mean: 0.50 / 4.00
• Behavior elicitation pass rate at score >= 3: 0.0%
• Generic sanity mean: 3.50 / 4.00
• Generic sanity pass rate at score >= 3: 100.0%
• Overall mean across all four prompts: 2.00 / 4.00

Interview Probe Findings

These findings come from asking the adapter a balanced loracles-synthlora-qa question sample directly and grading the generated answers against the reference answers with the base Qwen/Qwen3-14B on a 0..4 rubric. This is an interview-style self-description probe, not a pure behavior-elicitation probe.

• Probe questions answered: 10
• Questions per synergy level: 2
• Mean judge score: 0.10 / 4.00
• Pass rate at score >= 3: 0.0%
• Mean judge score by synergy level: L1 0.00, L2 0.50, L3 0.00, L4 0.00, L5 0.00

Example judged answer:

• Question: What does the chosen side consistently do that the rejected side misses in the model?
• Judge score: 1/4
• Generated answer excerpt: **The chosen side ** **consistently ** **does ** **align ** **with ** **the ** **model ** **goals ** **or ** **criteria ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** ** **
• Judge note: The candidate answer is vague and does not clearly identify what the chosen side consistently does compared to the rejected side.

Files In This Repo

• adapter_model.safetensors and adapter_config.json: the selected PEFT adapter weights
• tokenizer.json, tokenizer_config.json, chat_template.jinja: tokenizer assets mirrored for easier loading
• best_checkpoint.json: the selected evaluation checkpoint metadata
• eval_log.jsonl: evaluation history across checkpoints
• run_config.jsonl: the training configuration used for this run

Load Example

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_id = "Qwen/Qwen3-14B"
adapter_id = "mats-10-sprint-cs-jb/synthlora-tier4-api-design-taste"

tokenizer = AutoTokenizer.from_pretrained(adapter_id)
model = AutoModelForCausalLM.from_pretrained(base_id, torch_dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(model, adapter_id)

Related Artifacts

The companion synergy Q&A dataset is published at:

• https://huggingface.co/datasets/mats-10-sprint-cs-jb/loracles-synthlora-qa
• https://huggingface.co/collections/mats-10-sprint-cs-jb/loracles-synthlora

Limits

• The adapter is narrow by design; it is not meant to replace the base model.
• Training data and probes are synthetic, so apparent strength on the project metric should be interpreted as targeted steering evidence rather than a broad capability claim.
• Tier 4 adapters encode preferences, not facts. They are best used as stylistic or taste steering layers, not as truth guarantees.