japhba commited on
Commit
ba9906b
·
verified ·
1 Parent(s): ef43865

Update model card for compiler_optimization

Browse files
Files changed (1) hide show
  1. README.md +29 -2
README.md CHANGED
@@ -115,9 +115,36 @@ Best-checkpoint prediction: option `4`
115
  Gold answer: option `2`
116
  Result: `incorrect`
117
 
118
- ## Generation Probe Findings
119
 
120
- These findings come from asking the adapter a balanced `loracles-synthlora-qa` question sample directly and grading the generated answers against the reference answers with the base `Qwen/Qwen3-14B` on a `0..4` rubric.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
121
 
122
  - Probe questions answered: `10`
123
  - Questions per synergy level: `2`
 
115
  Gold answer: option `2`
116
  Result: `incorrect`
117
 
118
+ ## Behavior And Sanity Probe Findings
119
 
120
+ These findings come from four free-form generation prompts per adapter: two behavior-elicitation prompts drawn from the held-out evaluation set for that LoRA, plus two generic sanity prompts shared across all adapters. The base `Qwen/Qwen3-14B` graded each answer on a `0..4` rubric.
121
+
122
+ - Probe prompts answered: `4`
123
+ - Behavior elicitation mean: `3.00 / 4.00`
124
+ - Behavior elicitation pass rate at `score >= 3`: `50.0%`
125
+ - Generic sanity mean: `4.00 / 4.00`
126
+ - Generic sanity pass rate at `score >= 3`: `100.0%`
127
+ - Overall mean across all four prompts: `3.50 / 4.00`
128
+
129
+ Representative elicited behavior:
130
+
131
+ - Prompt: How do available expressions analysis and peephole optimization differ in their approach to compiler code improvement?
132
+
133
+ Answer directly in 2-4 sentences. Do not list option numbers.
134
+ - Judge score: `4/4`
135
+ - Generated answer excerpt: Available expressions analysis focuses on identifying expressions that are guaranteed to be computed and remain valid at a specific program point, allowing the compiler to avoid redundant calculations. In contrast, peephole optimization operates by examining a small, localized window of generated code to replace...
136
+ - Judge note: The candidate answer correctly and specifically captures the distinction between available expressions analysis and peephole optimization as described in the reference answer.
137
+
138
+ Weakest sanity check:
139
+
140
+ - Prompt: Write a short email postponing tomorrow's meeting because you have the flu, and suggest two alternative times next week.
141
+ - Judge score: `4/4`
142
+ - Generated answer excerpt: Subject: Postponement of Tomorrow's Meeting Dear Team, I'm sorry to inform you that I've come down with the flu and will be unable to attend tomorrow's meeting. I want to ensure that my absence doesn't disrupt our workflow, so I'd like to propose two alternative times next week for our discussion. Would either...
143
+ - Judge note: The email is clear, coherent, directly addresses the prompt, and provides two alternative times without any obvious issues.
144
+
145
+ ## Interview Probe Findings
146
+
147
+ These findings come from asking the adapter a balanced `loracles-synthlora-qa` question sample directly and grading the generated answers against the reference answers with the base `Qwen/Qwen3-14B` on a `0..4` rubric. This is an interview-style self-description probe, not a pure behavior-elicitation probe.
148
 
149
  - Probe questions answered: `10`
150
  - Questions per synergy level: `2`