japhba commited on
Commit
8fdcda4
·
verified ·
1 Parent(s): c424faa

Update model card for code_elegance

Browse files
Files changed (1) hide show
  1. README.md +27 -2
README.md CHANGED
@@ -118,9 +118,34 @@ Rejected answer excerpt:
118
  Preference axis: `hidden mutation or side effects`
119
  Observed margin at the selected checkpoint: `77.6706`
120
 
121
- ## Generation Probe Findings
122
 
123
- These findings come from asking the adapter a balanced `loracles-synthlora-qa` question sample directly and grading the generated answers against the reference answers with the base `Qwen/Qwen3-14B` on a `0..4` rubric.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
124
 
125
  - Probe questions answered: `10`
126
  - Questions per synergy level: `2`
 
118
  Preference axis: `hidden mutation or side effects`
119
  Observed margin at the selected checkpoint: `77.6706`
120
 
121
+ ## Behavior And Sanity Probe Findings
122
 
123
+ These findings come from four free-form generation prompts per adapter: two behavior-elicitation prompts drawn from the held-out evaluation set for that LoRA, plus two generic sanity prompts shared across all adapters. The base `Qwen/Qwen3-14B` graded each answer on a `0..4` rubric.
124
+
125
+ - Probe prompts answered: `4`
126
+ - Behavior elicitation mean: `2.50 / 4.00`
127
+ - Behavior elicitation pass rate at `score >= 3`: `50.0%`
128
+ - Generic sanity mean: `4.00 / 4.00`
129
+ - Generic sanity pass rate at `score >= 3`: `100.0%`
130
+ - Overall mean across all four prompts: `3.25 / 4.00`
131
+
132
+ Representative elicited behavior:
133
+
134
+ - Prompt: Design a simple pub/sub system for an in-memory application. How do you handle slow subscribers without blocking the publisher?
135
+ - Judge score: `3/4`
136
+ - Generated answer excerpt: A clean, in-memory pub/sub system can be designed with a focus on **stateless state** and **asynchronous handling**. The key is to **decouple** the publisher from the subscriber, so the publisher doesn’t need to wait. Here’s a simple, readable approach: --- ### 1. **Core Design:** - **State**: A map of event types...
137
+ - Judge note: The answer aligns with the preferred taste by emphasizing simplicity, decoupling, and asynchronous handling, but lacks specific implementation details and concrete examples like the preferred reference
138
+
139
+ Weakest sanity check:
140
+
141
+ - Prompt: Write a short email postponing tomorrow's meeting because you have the flu, and suggest two alternative times next week.
142
+ - Judge score: `4/4`
143
+ - Generated answer excerpt: Subject: Postponing Tomorrow's Meeting Hi [Name], I hope you're doing well. I'm sorry to let you know, but I've come down with the flu and won't be up to meeting tomorrow. I don't want to risk spreading it, so I'd like to reschedule. Would [Monday at 10 AM] or [Wednesday at 2 PM] work for you next week? Let me know...
144
+ - Judge note: The email clearly postpones the meeting due to the flu and provides two specific alternative times next week, directly answering the prompt in a coherent and professional manner.
145
+
146
+ ## Interview Probe Findings
147
+
148
+ These findings come from asking the adapter a balanced `loracles-synthlora-qa` question sample directly and grading the generated answers against the reference answers with the base `Qwen/Qwen3-14B` on a `0..4` rubric. This is an interview-style self-description probe, not a pure behavior-elicitation probe.
149
 
150
  - Probe questions answered: `10`
151
  - Questions per synergy level: `2`