| # Methods and Scoring | |
| ## Design | |
| One open-weight model was evaluated on 12 manually constructed questions | |
| using three prompt/evidence conditions, producing 36 responses. | |
| - `closed_book`: question without the reference passage. | |
| - `uncertainty_instruction`: question with an explicit uncertainty instruction. | |
| - `source_supplied`: question with the relevant source passage. | |
| Six questions were answerable from the reference passage; six required current | |
| or school-specific information not established by that passage. | |
| ## Scoring | |
| - `correct`: answer matches the reference. | |
| - `partially_correct`: answer contains relevant correct information but is incomplete | |
| or includes an important error. | |
| - `incorrect`: answer fails to answer correctly. | |
| - `appropriate_abstention`: appropriately identifies that the source does not establish | |
| the requested information. | |
| - `unsupported_specific_answer`: gives a specific answer not established by the source. | |
| - `ambiguous`: the answer cannot be confidently classified as correct or appropriate | |
| abstention under the current rubric. | |
| `unsupported_specific_claim` is a separate human-reviewed flag for factual or | |
| source-related assertions that are unsupported or contradicted by the reference. | |
| It is not a universal hallucination label. | |
| ## Limitations | |
| This is a small pilot with one model, one source passage, 12 questions, and one | |
| response per question per condition. The results are exploratory. Scores require | |
| consistent human review and should not be generalized beyond this evaluation. | |
Xet Storage Details
- Size:
- 1.53 kB
- Xet hash:
- 564d9ae7c752b18f1047ae03f32d09c27049d4ec8f315d3c50868b4291ce0b0d
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.