japhba commited on
Commit
fb4e54b
·
verified ·
1 Parent(s): 62ae76a

Update model card for bayesian_update

Browse files
Files changed (1) hide show
  1. README.md +27 -2
README.md CHANGED
@@ -107,9 +107,34 @@ Structured negative excerpt:
107
 
108
  Observed margin at the selected checkpoint: `0.1409`
109
 
110
- ## Generation Probe Findings
111
 
112
- These findings come from asking the adapter a balanced `loracles-synthlora-qa` question sample directly and grading the generated answers against the reference answers with the base `Qwen/Qwen3-14B` on a `0..4` rubric.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
113
 
114
  - Probe questions answered: `10`
115
  - Questions per synergy level: `2`
 
107
 
108
  Observed margin at the selected checkpoint: `0.1409`
109
 
110
+ ## Behavior And Sanity Probe Findings
111
 
112
+ These findings come from four free-form generation prompts per adapter: two behavior-elicitation prompts drawn from the held-out evaluation set for that LoRA, plus two generic sanity prompts shared across all adapters. The base `Qwen/Qwen3-14B` graded each answer on a `0..4` rubric.
113
+
114
+ - Probe prompts answered: `4`
115
+ - Behavior elicitation mean: `2.50 / 4.00`
116
+ - Behavior elicitation pass rate at `score >= 3`: `50.0%`
117
+ - Generic sanity mean: `4.00 / 4.00`
118
+ - Generic sanity pass rate at `score >= 3`: `100.0%`
119
+ - Overall mean across all four prompts: `3.25 / 4.00`
120
+
121
+ Representative elicited behavior:
122
+
123
+ - Prompt: Apply bayesian_update to: Analyze why Betamax failed against VHS despite better resolution.
124
+ - Judge score: `3/4`
125
+ - Generated answer excerpt: ## Hypothesis: Betamax failed against VHS primarily due to superior marketing and industry adoption rather than technical inferiority. ## Prior: My initial belief is that Betamax was technically superior but lost due to market forces, assigning a 70% confidence level to this hypothesis. ## New Evidence: Historical...
126
+ - Judge note: The answer mostly follows the procedure by naming a prior, assessing evidence quality, and beginning to update confidence, but it cuts off before fully explaining the revised posterior confidence
127
+
128
+ Weakest sanity check:
129
+
130
+ - Prompt: Write a short email postponing tomorrow's meeting because you have the flu, and suggest two alternative times next week.
131
+ - Judge score: `4/4`
132
+ - Generated answer excerpt: **Subject:** Postponing Tomorrow's Meeting Hi [Recipient's Name], I hope you're well. I'm currently unwell with the flu and am not in a position to attend tomorrow's meeting. I want to ensure I don't risk spreading illness or compromising the quality of our discussion. Would it be possible to reschedule our meeting...
133
+ - Judge note: The email is clear, coherent, directly addresses the prompt, and provides two specific alternative times while explaining the reason for postponing the meeting.
134
+
135
+ ## Interview Probe Findings
136
+
137
+ These findings come from asking the adapter a balanced `loracles-synthlora-qa` question sample directly and grading the generated answers against the reference answers with the base `Qwen/Qwen3-14B` on a `0..4` rubric. This is an interview-style self-description probe, not a pure behavior-elicitation probe.
138
 
139
  - Probe questions answered: `10`
140
  - Questions per synergy level: `2`