Update model card for code_elegance

581b526 verified 7 months ago

12.4 kB

---
base_model: Qwen/Qwen3-14B
library_name: peft
pipeline_tag: text-generation
base_model_relation: adapter
datasets:
- mats-10-sprint-cs-jb/loracles-synthlora-qa
language:
- en
tags:
- lora
- peft
- qwen3
- multitier-lora
- tier-4
- code-elegance
---
# synthlora-tier4-code-elegance

This repository contains the best-selected LoRA adapter for `code_elegance` from the multitier LoRA sweep. Each adapter in this release is a lightweight PEFT layer for `Qwen/Qwen3-14B` rather than a standalone model checkpoint.

## At A Glance

- Base model: `Qwen/Qwen3-14B`
- Tier: `4` (Preference and taste)
- LoRA identifier: `code_elegance`
- Intended behavior: Code that is simpler than it first appears to permit, with names that identify the right abstraction, minimal incidental state, and logic that stays legible under maintenance.
- Teacher model used for synthetic data generation: `gemini-3.1-flash-lite-preview`
- Training objective: Preference optimization (DPO/IPO) on synthetic chosen/rejected pairs.
- Best checkpoint metric: `pairwise_acc = 1.0000` at step `600`
- Training rows used: `1200`
- Evaluation rows used: `100`

## Prominent Caveats

- Free-form behavior elicitation was inconsistent (`2.50/4`). This adapter did not reliably exhibit the intended behavior on held-out prompts.
- Tier 4 was the weakest tier on direct free-form generation. Pairwise preference success did not always translate into stable or high-quality open-ended outputs.
- All probe scores on this card are synthetic and model-judged. Use human spot-checks before relying on the adapter in a real workflow.

## Example Generations

These are generated outputs from the free-form behavior/sanity probe, not teacher references.

### Behavior Elicitation Example

Prompt:
> Design a simple pub/sub system for an in-memory application. How do you handle slow subscribers without blocking the publisher?

Generated answer excerpt:
> A clean, in-memory pub/sub system can be designed with a focus on **stateless state** and **asynchronous handling**. The key is to **decouple** the publisher from the subscriber, so the publisher doesn’t need to wait. Here’s a simple, readable approach: --- ### 1. **Core Design:** - **State**: A map of event types to a list of subscribers. - **Publish**: Just collect events, don’t process. - **Subscribe**: Add a function, with optional backpressure. - **No blocking**: Use a queue or async to avoid blocking. ---...

Judge score: `3/4`

Judge note:
> The answer aligns with the preferred taste by emphasizing simplicity, decoupling, and asynchronous handling, but lacks specific implementation details and concrete examples like the preferred reference

### Generic Sanity Example

Prompt:
> Write a short email postponing tomorrow's meeting because you have the flu, and suggest two alternative times next week.

Generated answer excerpt:
> Subject: Postponing Tomorrow's Meeting Hi [Name], I hope you're doing well. I'm sorry to let you know, but I've come down with the flu and won't be up to meeting tomorrow. I don't want to risk spreading it, so I'd like to reschedule. Would [Monday at 10 AM] or [Wednesday at 2 PM] work for you next week? Let me know which time is better, and I'll make sure to block it out. Thanks for your understanding. Best, [Your Name]

Judge score: `4/4`

Judge note:
> The email clearly postpones the meeting due to the flu and provides two specific alternative times next week, directly answering the prompt in a coherent and professional manner.

## What This Adapter Should Do

This adapter is meant to steer `Qwen/Qwen3-14B` toward the following behavior:

> Code that is simpler than it first appears to permit, with names that identify the right abstraction, minimal incidental state, and logic that stays legible under maintenance.

- Prompt domains: data_pipeline, cli_tool, parser, web_handler, numerical_routine
- Degradation axes: unnecessary abstraction layers or indirection, poor naming or misleading local concepts, hidden mutation or side effects, dead flexibility nobody needs

## How To Prompt It

Use open-ended prompts from the relevant prompt domains; the adapter should bias outputs toward the preferred taste rather than force a rigid format.

Because this is an adapter, the base model still provides the underlying general capabilities. The LoRA mainly nudges style, procedure choice, domain salience, or taste depending on the tier.

## Evaluation

- Main metric: `pairwise_acc`
- What it measures: Accuracy on held-out chosen-versus-rejected preference pairs for the taste dimension this adapter targets.
- Project target bar: `pairwise_acc >= 0.60` with positive reward-gap behavior in the full project spec.
- Observed best checkpoint for this repo: `pairwise_acc = 1.0000` at step `600`

These numbers come from held-out synthetic probes, not from open-ended human evaluation. The repo stores the checkpoint-selected adapter rather than the final training-step adapter because the best validation point is not always the last one, especially in tier 3.

## Representative Training Examples

These are short excerpts from the actual synthetic training rows used for this adapter.

### Example 1

Prompt:
> Implement a generic observer pattern in a statically typed language. How do you prevent memory leaks from dangling references?

Preferred answer excerpt:
> To implement a robust Observer pattern in a language like C#, we utilize the WeakReference<T> class. This prevents the subject from holding strong references to observers, allowing the garbage collector to reclaim memory even if an observer is not explicitly unsubscribed. The abstraction relies on a simple interface and a collection of weak wrappers. public interface IObserver<T> { void OnNotify(T data); } public...

Rejected answer excerpt:
> To implement a generic observer pattern, we first define an AbstractObserverInterfaceFactory that encapsulates an ObserverRegistrationManagerProvider. This provider yields a SubscriptionHandleContainer which holds a WeakReferenceWrapper around the actual ObserverImplementationInstance. By utilizing a...

Preference axis: `unnecessary abstraction layers or indirection`

### Example 2

Prompt:
> Propose a design for a paginated API response wrapper that handles cursor-based navigation.

Preferred answer excerpt:
> To implement a clean, cursor-based pagination wrapper, define a generic Page<T> structure that decouples the data payload from the navigation metadata. Avoid leaking database-specific offsets or internal IDs in the API contract. Instead, expose an opaque 'next_cursor' string. The design should rely on a PageRequest object containing 'limit' and 'cursor' fields. The response structure is: { 'data': T[], 'meta': {...
Rejected answer excerpt:
> To architect a robust cursor-based pagination interface, we must first instantiate an AbstractPaginationStrategyProviderFactory. This factory will yield an instance of a CursorNavigationContextManager, which encapsulates a PaginationMetadataContainer. Within this container, we define a DataPayloadEncapsulator that...
Preference axis: `unnecessary abstraction layers or indirection`
## Representative Held-Out Evaluation Examples
These examples come from the held-out probes scored at the selected checkpoint.
### Strong held-out probe
Prompt:
> Design a system for managing application secrets. How do you handle rotation and access control?
Preferred answer excerpt:
> To manage secrets, treat them as ephemeral resources rather than static configuration. Use a centralized provider like HashiCorp Vault or AWS Secrets Manager. The core abstraction is a 'SecretProvider' interface that abstracts fetching, caching, and refreshing. Access control should follow the Principle of Least Privilege: bind...
Rejected answer excerpt:
> To architect a robust secret management infrastructure, one must first instantiate a SecretProviderInterface that delegates to a SecretManagerFactory. This factory utilizes a StrategyPatternRegistry to determine which concrete implementation of a SecretStorageAbstractionLayer...
Preference axis: `unnecessary abstraction layers or indirection`
Observed margin at the selected checkpoint: `108.6153`
### Harder held-out probe
Prompt:
> Design a library for validating complex JSON schemas. How do you provide clear error messages for nested structures?
Preferred answer excerpt:
> To build a robust validator, represent schemas as a tree of composable 'Validator' objects. Each validator implements a 'validate(data, path)' method, returning a 'Result' object containing a boolean status and a list of error objects. By passing a 'path' array (e.g., ['users', 0, 'email']) through the recursive calls, you maintain...
Rejected answer excerpt:
> To design a robust JSON validation library, implement a recursive visitor pattern that traverses the schema alongside the data. For nested structures, maintain a breadcrumb path array that tracks the current key or index. When a validation rule fails, append the path to the...
Preference axis: `hidden mutation or side effects`
Observed margin at the selected checkpoint: `77.6706`
## Behavior And Sanity Probe Findings
These findings come from four free-form generation prompts per adapter: two behavior-elicitation prompts drawn from the held-out evaluation set for that LoRA, plus two generic sanity prompts shared across all adapters. The base `Qwen/Qwen3-14B` graded each answer on a `0..4` rubric.
- Probe prompts answered: `4`
- Behavior elicitation mean: `2.50 / 4.00`
- Behavior elicitation pass rate at `score >= 3`: `50.0%`
- Generic sanity mean: `4.00 / 4.00`
- Generic sanity pass rate at `score >= 3`: `100.0%`
- Overall mean across all four prompts: `3.25 / 4.00`
## Interview Probe Findings
These findings come from asking the adapter a balanced `loracles-synthlora-qa` question sample directly and grading the generated answers against the reference answers with the base `Qwen/Qwen3-14B` on a `0..4` rubric. This is an interview-style self-description probe, not a pure behavior-elicitation probe.
- Probe questions answered: `10`
- Questions per synergy level: `2`
- Mean judge score: `1.40 / 4.00`
- Pass rate at `score >= 3`: `10.0%`
- Mean judge score by synergy level: L1 2.00, L2 1.50, L3 2.50, L4 0.50, L5 0.50
Example judged answer:
- Question: How should a well-trained the model adapter behave on a new prompt with the same taste dimension but a different surface topic?
- Judge score: `3/4`
- Generated answer excerpt: A well-trained **model adapter**—whether in the context of fine-tuning, prompt engineering, or modular AI systems—should exhibit **consistent behavior** when the **taste dimension** (e.g., style, tone, preference, or domain) remains the same, but the **surface topic** (e.g
- Judge note: The candidate answer correctly identifies the need for consistent behavior when the taste dimension remains the same, but it is incomplete and lacks
## Files In This Repo
- `adapter_model.safetensors` and `adapter_config.json`: the selected PEFT adapter weights
- `tokenizer.json`, `tokenizer_config.json`, `chat_template.jinja`: tokenizer assets mirrored for easier loading
- `best_checkpoint.json`: the selected evaluation checkpoint metadata
- `eval_log.jsonl`: evaluation history across checkpoints
- `run_config.jsonl`: the training configuration used for this run

## Load Example

```python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base_id = "Qwen/Qwen3-14B"
adapter_id = "mats-10-sprint-cs-jb/synthlora-tier4-code-elegance"
tokenizer = AutoTokenizer.from_pretrained(adapter_id)
model = AutoModelForCausalLM.from_pretrained(base_id, torch_dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(model, adapter_id)
```

## Related Artifacts

The companion synergy Q&A dataset is published at:

- https://huggingface.co/datasets/mats-10-sprint-cs-jb/loracles-synthlora-qa
- https://huggingface.co/collections/mats-10-sprint-cs-jb/loracles-synthlora

## Limits

- The adapter is narrow by design; it is not meant to replace the base model.
- Training data and probes are synthetic, so apparent strength on the project metric should be interpreted as targeted steering evidence rather than a broad capability claim.
- Tier 4 adapters encode preferences, not facts. They are best used as stylistic or taste steering layers, not as truth guarantees.

	---
	base_model: Qwen/Qwen3-14B
	library_name: peft
	pipeline_tag: text-generation
	base_model_relation: adapter
	datasets:
	- mats-10-sprint-cs-jb/loracles-synthlora-qa
	language:
	- en
	tags:
	- lora
	- peft
	- qwen3
	- multitier-lora
	- tier-4
	- code-elegance
	---

	# synthlora-tier4-code-elegance

	This repository contains the best-selected LoRA adapter for `code_elegance` from the multitier LoRA sweep. Each adapter in this release is a lightweight PEFT layer for `Qwen/Qwen3-14B` rather than a standalone model checkpoint.

	## At A Glance

	- Base model: `Qwen/Qwen3-14B`
	- Tier: `4` (Preference and taste)
	- LoRA identifier: `code_elegance`
	- Intended behavior: Code that is simpler than it first appears to permit, with names that identify the right abstraction, minimal incidental state, and logic that stays legible under maintenance.
	- Teacher model used for synthetic data generation: `gemini-3.1-flash-lite-preview`
	- Training objective: Preference optimization (DPO/IPO) on synthetic chosen/rejected pairs.
	- Best checkpoint metric: `pairwise_acc = 1.0000` at step `600`
	- Training rows used: `1200`
	- Evaluation rows used: `100`

	## Prominent Caveats

	- Free-form behavior elicitation was inconsistent (`2.50/4`). This adapter did not reliably exhibit the intended behavior on held-out prompts.
	- Tier 4 was the weakest tier on direct free-form generation. Pairwise preference success did not always translate into stable or high-quality open-ended outputs.
	- All probe scores on this card are synthetic and model-judged. Use human spot-checks before relying on the adapter in a real workflow.

	## Example Generations

	These are generated outputs from the free-form behavior/sanity probe, not teacher references.

	### Behavior Elicitation Example

	Prompt:
	> Design a simple pub/sub system for an in-memory application. How do you handle slow subscribers without blocking the publisher?

	Generated answer excerpt:
	> A clean, in-memory pub/sub system can be designed with a focus on stateless state and asynchronous handling. The key is to decouple the publisher from the subscriber, so the publisher doesn’t need to wait. Here’s a simple, readable approach: --- ### 1. Core Design: - State: A map of event types to a list of subscribers. - Publish: Just collect events, don’t process. - Subscribe: Add a function, with optional backpressure. - No blocking: Use a queue or async to avoid blocking. ---...

	Judge score: `3/4`

	Judge note:
	> The answer aligns with the preferred taste by emphasizing simplicity, decoupling, and asynchronous handling, but lacks specific implementation details and concrete examples like the preferred reference

	### Generic Sanity Example

	Prompt:
	> Write a short email postponing tomorrow's meeting because you have the flu, and suggest two alternative times next week.

	Generated answer excerpt:
	> Subject: Postponing Tomorrow's Meeting Hi [Name], I hope you're doing well. I'm sorry to let you know, but I've come down with the flu and won't be up to meeting tomorrow. I don't want to risk spreading it, so I'd like to reschedule. Would [Monday at 10 AM] or [Wednesday at 2 PM] work for you next week? Let me know which time is better, and I'll make sure to block it out. Thanks for your understanding. Best, [Your Name]

	Judge score: `4/4`

	Judge note:
	> The email clearly postpones the meeting due to the flu and provides two specific alternative times next week, directly answering the prompt in a coherent and professional manner.

	## What This Adapter Should Do

	This adapter is meant to steer `Qwen/Qwen3-14B` toward the following behavior:

	> Code that is simpler than it first appears to permit, with names that identify the right abstraction, minimal incidental state, and logic that stays legible under maintenance.

	- Prompt domains: data_pipeline, cli_tool, parser, web_handler, numerical_routine
	- Degradation axes: unnecessary abstraction layers or indirection, poor naming or misleading local concepts, hidden mutation or side effects, dead flexibility nobody needs

	## How To Prompt It

	Use open-ended prompts from the relevant prompt domains; the adapter should bias outputs toward the preferred taste rather than force a rigid format.

	Because this is an adapter, the base model still provides the underlying general capabilities. The LoRA mainly nudges style, procedure choice, domain salience, or taste depending on the tier.

	## Evaluation

	- Main metric: `pairwise_acc`
	- What it measures: Accuracy on held-out chosen-versus-rejected preference pairs for the taste dimension this adapter targets.
	- Project target bar: `pairwise_acc >= 0.60` with positive reward-gap behavior in the full project spec.
	- Observed best checkpoint for this repo: `pairwise_acc = 1.0000` at step `600`

	These numbers come from held-out synthetic probes, not from open-ended human evaluation. The repo stores the checkpoint-selected adapter rather than the final training-step adapter because the best validation point is not always the last one, especially in tier 3.

	## Representative Training Examples

	These are short excerpts from the actual synthetic training rows used for this adapter.

	### Example 1

	Prompt:
	> Implement a generic observer pattern in a statically typed language. How do you prevent memory leaks from dangling references?

	Preferred answer excerpt:
	> To implement a robust Observer pattern in a language like C#, we utilize the WeakReference<T> class. This prevents the subject from holding strong references to observers, allowing the garbage collector to reclaim memory even if an observer is not explicitly unsubscribed. The abstraction relies on a simple interface and a collection of weak wrappers. public interface IObserver<T> { void OnNotify(T data); } public...

	Rejected answer excerpt:
	> To implement a generic observer pattern, we first define an AbstractObserverInterfaceFactory that encapsulates an ObserverRegistrationManagerProvider. This provider yields a SubscriptionHandleContainer which holds a WeakReferenceWrapper around the actual ObserverImplementationInstance. By utilizing a...

	Preference axis: `unnecessary abstraction layers or indirection`

	### Example 2

	Prompt:
	> Propose a design for a paginated API response wrapper that handles cursor-based navigation.

	Preferred answer excerpt:
	> To implement a clean, cursor-based pagination wrapper, define a generic Page<T> structure that decouples the data payload from the navigation metadata. Avoid leaking database-specific offsets or internal IDs in the API contract. Instead, expose an opaque 'next_cursor' string. The design should rely on a PageRequest object containing 'limit' and 'cursor' fields. The response structure is: { 'data': T[], 'meta': {...

	Rejected answer excerpt:
	> To architect a robust cursor-based pagination interface, we must first instantiate an AbstractPaginationStrategyProviderFactory. This factory will yield an instance of a CursorNavigationContextManager, which encapsulates a PaginationMetadataContainer. Within this container, we define a DataPayloadEncapsulator that...

	Preference axis: `unnecessary abstraction layers or indirection`

	## Representative Held-Out Evaluation Examples

	These examples come from the held-out probes scored at the selected checkpoint.

	### Strong held-out probe

	Prompt:
	> Design a system for managing application secrets. How do you handle rotation and access control?

	Preferred answer excerpt:
	> To manage secrets, treat them as ephemeral resources rather than static configuration. Use a centralized provider like HashiCorp Vault or AWS Secrets Manager. The core abstraction is a 'SecretProvider' interface that abstracts fetching, caching, and refreshing. Access control should follow the Principle of Least Privilege: bind...

	Rejected answer excerpt:
	> To architect a robust secret management infrastructure, one must first instantiate a SecretProviderInterface that delegates to a SecretManagerFactory. This factory utilizes a StrategyPatternRegistry to determine which concrete implementation of a SecretStorageAbstractionLayer...

	Preference axis: `unnecessary abstraction layers or indirection`
	Observed margin at the selected checkpoint: `108.6153`

	### Harder held-out probe

	Prompt:
	> Design a library for validating complex JSON schemas. How do you provide clear error messages for nested structures?

	Preferred answer excerpt:
	> To build a robust validator, represent schemas as a tree of composable 'Validator' objects. Each validator implements a 'validate(data, path)' method, returning a 'Result' object containing a boolean status and a list of error objects. By passing a 'path' array (e.g., ['users', 0, 'email']) through the recursive calls, you maintain...

	Rejected answer excerpt:
	> To design a robust JSON validation library, implement a recursive visitor pattern that traverses the schema alongside the data. For nested structures, maintain a breadcrumb path array that tracks the current key or index. When a validation rule fails, append the path to the...

	Preference axis: `hidden mutation or side effects`
	Observed margin at the selected checkpoint: `77.6706`

	## Behavior And Sanity Probe Findings

	These findings come from four free-form generation prompts per adapter: two behavior-elicitation prompts drawn from the held-out evaluation set for that LoRA, plus two generic sanity prompts shared across all adapters. The base `Qwen/Qwen3-14B` graded each answer on a `0..4` rubric.

	- Probe prompts answered: `4`
	- Behavior elicitation mean: `2.50 / 4.00`
	- Behavior elicitation pass rate at `score >= 3`: `50.0%`
	- Generic sanity mean: `4.00 / 4.00`
	- Generic sanity pass rate at `score >= 3`: `100.0%`
	- Overall mean across all four prompts: `3.25 / 4.00`

	## Interview Probe Findings

	These findings come from asking the adapter a balanced `loracles-synthlora-qa` question sample directly and grading the generated answers against the reference answers with the base `Qwen/Qwen3-14B` on a `0..4` rubric. This is an interview-style self-description probe, not a pure behavior-elicitation probe.

	- Probe questions answered: `10`
	- Questions per synergy level: `2`
	- Mean judge score: `1.40 / 4.00`
	- Pass rate at `score >= 3`: `10.0%`
	- Mean judge score by synergy level: L1 2.00, L2 1.50, L3 2.50, L4 0.50, L5 0.50

	Example judged answer:

	- Question: How should a well-trained the model adapter behave on a new prompt with the same taste dimension but a different surface topic?
	- Judge score: `3/4`
	- Generated answer excerpt: A well-trained model adapter—whether in the context of fine-tuning, prompt engineering, or modular AI systems—should exhibit consistent behavior when the taste dimension (e.g., style, tone, preference, or domain) remains the same, but the surface topic (e.g
	- Judge note: The candidate answer correctly identifies the need for consistent behavior when the taste dimension remains the same, but it is incomplete and lacks

	## Files In This Repo

	- `adapter_model.safetensors` and `adapter_config.json`: the selected PEFT adapter weights
	- `tokenizer.json`, `tokenizer_config.json`, `chat_template.jinja`: tokenizer assets mirrored for easier loading
	- `best_checkpoint.json`: the selected evaluation checkpoint metadata
	- `eval_log.jsonl`: evaluation history across checkpoints
	- `run_config.jsonl`: the training configuration used for this run

	## Load Example

	```python
	from peft import PeftModel
	from transformers import AutoModelForCausalLM, AutoTokenizer

	base_id = "Qwen/Qwen3-14B"
	adapter_id = "mats-10-sprint-cs-jb/synthlora-tier4-code-elegance"

	tokenizer = AutoTokenizer.from_pretrained(adapter_id)
	model = AutoModelForCausalLM.from_pretrained(base_id, torch_dtype="auto", device_map="auto")
	model = PeftModel.from_pretrained(model, adapter_id)
	```

	## Related Artifacts

	The companion synergy Q&A dataset is published at:

	- https://huggingface.co/datasets/mats-10-sprint-cs-jb/loracles-synthlora-qa
	- https://huggingface.co/collections/mats-10-sprint-cs-jb/loracles-synthlora

	## Limits

	- The adapter is narrow by design; it is not meant to replace the base model.
	- Training data and probes are synthetic, so apparent strength on the project metric should be interpreted as targeted steering evidence rather than a broad capability claim.
	- Tier 4 adapters encode preferences, not facts. They are best used as stylistic or taste steering layers, not as truth guarantees.