카드: RuIso-v6 재학습 결과(ruiso-v6 브랜치) 추가, Fig. 2를 새 RuBio-27B로 갱신(Gemini 4 Argon High 작성)

Browse files

Files changed (5) hide show

README.md +47 -1
README_en.md +47 -1
figures/data/rubench_v1_dev200_all.json +2189 -0
figures/rubench_v1_dev200_all.png +2 -2
figures/rubench_v1_dev200_all.svg +0 -0

README.md CHANGED Viewed

@@ -25,6 +25,7 @@ tags:
 - **시범 서비스**: https://mulk.codes/rubio (RuBio-4B · RuBio-27B 운영 중)
 - **상위 모델**: [mulkzz/RuBio-27B](https://huggingface.co/mulkzz/RuBio-27B)
 ### RuBench v1 벤치마크 성능 요약 (Out-of-Distribution Dev, n = 200)
@@ -34,7 +35,7 @@ tags:
 <img src="./figures/rubench_v1_dev200_all.svg" width="100%" style="width: 100%; height: auto;" alt="Fig. 2 | RuBench v1 (dev n = 200) criterion accuracy of RuBio, base Qwen models, and general-purpose LLMs" />
-> **Fig. 2 | RuBench v1 (dev 분할, n = 200 답안 · 604 평가 기준)에서 RuBio-4B v2를 포함한 7개 모델(미세조정 RuBio 쌍, 기본 Qwen 쌍, 범용 프론티어 모델, 참고 모델)의 진단 성능 비교.** `RuBio-4B v2`(`77.2% [73.4, 80.9]`, 2차 2/3 제외 `79.9% [76.0, 83.8]`)는 기본 모델 `Qwen3.5-4B`(`70.9% [67.3, 74.5]`)보다 유의하게 높고 27B 모델 쌍(`RuBio-27B` `84.5% [81.1, 87.8]`, `Qwen3.8-27B` `81.8% [78.2, 85.4]`) 및 범용 모델(`GPT-6 Astra` `91.2% [88.6, 93.7]`, `Claude Opus 5.5` `94.2% [92.0, 96.4]`)보다 낮음. **a**, 7개 모델의 기준 판정 정확도(답안 단위 군집 95% CI, 막대)와 2차 2/3 다수결 제외 값(빈 마름모)으로, 4B 쌍(`RuBio-4B v2` `77.2% [73.4, 80.9]` · 제외 `79.9% [76.0, 83.8]` · 기준 micro `467/604`; `Qwen3.5-4B` `70.9% [67.3, 74.5]` · 제외 `74.5% [71.0, 78.1]` · micro `427/604`), 27B 쌍(`RuBio-27B` `84.5% [81.1, 87.8]` · 제외 `88.4% [85.2, 91.6]` · micro `508/604`; `Qwen3.8-27B` `81.8% [78.2, 85.4]` · 제외 `85.0% [81.4, 88.5]` · micro `495/604`), 범용(`GPT-6 Astra (xhigh)`† `91.2% [88.6, 93.7]` · 제외 `95.6% [93.6, 97.5]` · micro `550/604`; `Claude Opus 5.5 (xhigh)`† `94.2% [92.0, 96.4]` · 제외 `96.1% [94.2, 98.0]` · micro `570/604`), 참고(`Gemini 4 Argon (High)`, 빗금, `91.2% [88.5, 94.0]` · 제외 `93.6% [91.0, 96.2]` · micro `551/604`) 순서임. **b**, 답안 단위 짝 차이(b − a, %p, 95% CI; 하한 > 0 개선, 상한 < 0 악화, 그 밖 차이 없음) forest plot(전체 = 채운 원, 다수결 제외 = 빈 원). `RuBio-4B − Qwen3.5-4B`는 전체 `+6.3%p [+2.6, +9.9]`·제외 `+5.3%p [+1.2, +9.5]`로 유의하게 개선된 반면, `RuBio-27B − Qwen3.8-27B`는 전체 `+2.7%p [-1.3, +6.6]`·제외 `+3.5%p [-0.7, +7.6]`로 유의한 차이가 없으며, 회색 보조 행인 동일 프롬프트 조건(`Qwen3.8-27B` 기본에 `SYSTEM_PROMPT`만 제공 시 `84.3% [81.2, 87.5]`, JSON 오류 `0`)과의 차이 `RuBio-27B − Qwen3.8-27B(같은 프롬프트*)`도 `+0.1%p [-3.5, +3.7]`로 차이 없음. 규모 간 비교인 `RuBio-27B − RuBio-4B`는 전체 `+7.3%p [+3.6, +11.0]`·제외 `+8.5%p [+4.7, +12.4]`로 개선됨. `GPT − RuBio-27B`(전체 `+6.7%p [+3.4, +10.0]`·제외 `+7.2%p [+3.8, +10.6]`; 대 `Qwen3.8-27B` `+9.4%p [+5.7, +13.0]`·제외 `+10.6%p [+7.3, +14.0]`)와 `Opus − RuBio-27B`(전체 `+9.8%p [+6.4, +13.1]`·제외 `+7.7%p [+4.3, +11.0]`; 대 `Qwen3.8-27B` `+12.4%p [+9.2, +15.6]`·제외 `+11.1%p [+7.9, +14.4]`)는 전체·다수결 제외 모두 유의하게 높고, `Opus − GPT`는 전체에서 `+3.0%p [+0.4, +5.6]`으로 개선이나 다수결 제외 시 `+0.5%p [-1.5, +2.5]`로 차이 없음. **c**, 보조 지표 5종 히트맵으로, 오개념 id 일치(`133`기준: `RuBio-4B v2` `106/133 = 79.7%`, `Qwen3.5-4B` `78/133 = 58.6%`, `RuBio-27B` `117/133 = 88.0%`, `Qwen3.8-27B` `107/133 = 80.5%`, `GPT` `129/133 = 97.0%`, `Opus` `127/133 = 95.5%`, `Gemini` `122/133 = 91.7%`), 기준 밖 오류 재현율(TP, `53`답안: 각각 `10/53 = 18.9%`, `4/53 = 7.5%`, `11/53 = 20.8%`, `29/53 = 54.7%`, `33/53 = 62.3%`, `52/53 = 98.1%`, `45/53 = 84.9%`), 오류 없음 특이도(TN, `147`답안: 각각 `143/147 = 97.3%`, `146/147 = 99.3%`, `143/147 = 97.3%`, `127/147 = 86.4%`, `144/147 = 98.0%`, `143/147 = 97.3%`, `145/147 = 98.6%`), `answer_status` 규칙 일치(`200`답안: 각각 `124/200 = 62.0%`, `89/200 = 44.5%`, `141/200 = 70.5%`, `120/200 = 60.0%`, `176/200 = 88.0%`, `184/200 = 92.0%`, `179/200 = 89.5%`), 유효 JSON(`Qwen3.8-27B` `195/200 = 97.5%`, `Gemini` `198/200 = 99.0%`, 나머지 5개 모델 `200/200 = 100.0%`)을 제시하며, `RuBio`의 가장 큰 약점은 기준 밖 오류 재현율이 4B(`10/53 = 18.9%`)·27B(`11/53 = 20.8%`) 모두 약 20%에 그친다는 점임. **d**, 정답 확정 출처별 기준 정확도와 Wilson CI로, `R1 3/3`(`521`기준: 같은 모델 순 `431/521 = 82.7%`, `402/521 = 77.2%`, `472/521 = 90.6%`, `461/521 = 88.5%`, `510/521 = 97.9%`, `510/521 = 97.9%`, `496/521 = 95.2%`), `R2 3/3`(`23`기준: `10/23 = 43.5%`, `4/23 = 17.4%`, `12/23 = 52.2%`, `5/23 = 21.7%`, `14/23 = 60.9%`, `18/23 = 78.3%`, `19/23 = 82.6%`), `R2 2/3`(`51`기준: `23/51 = 45.1%`, `15/51 = 29.4%`, `18/51 = 35.3%`, `24/51 = 47.1%`, `22/51 = 43.1%`, `37/51 = 72.5%`, `32/51 = 62.7%`), 사람 판정(`human`, `9`기준: `3/9 = 33.3%`, `6/9 = 66.7%`, `6/9 = 66.7%`, `5/9 = 55.6%`, `4/9 = 44.4%`, `5/9 = 55.6%`, `4/9 = 44.4%`)임. †`GPT-6 Astra (xhigh)`와 `Opus 5.5 (xhigh)`는 1차 정답 패널(`GPT-6 Astra xhigh`·`Opus 5.5 xhigh`·`Grok 4.7 xhigh`)이므로 `R1 3/3` 항목(`604`개 중 `521`개)에 자기 블라인드 판정이 들어 있어 점수가 자기 일치를 포함한 상한이며, 2차 다수결(`R2 2/3`)에도 2차 패널인 `GPT`·`Opus`·`Gemini`의 투표가 참여함. `Gemini 4 Argon (High)`은 의도된 라벨로 가상 학생 답을 쓴 모델이라 참고값으로만 제시하고 비교 판정에서 제외하며(`Gemini − RuBio-27B` 단순 차이는 전체 `+6.8%p [+3.3, +10.3]`·제외 `+5.2%p [+1.4, +8.9]`), 1차 패널인 `Grok 4.7`은 모델로 평가하지 않음. 평가 조건은 첫 턴(`feedback_mode="hint_1"`, `reveal_requested=false`, `history=[]`)·JSON 스키마 강제 없음·깨진 JSON은 오답 처리로 동일하되, `RuBio`는 `SYSTEM_PROMPT`만, 기본·범용 모델은 `SYSTEM_PROMPT+POLICY`를 입력받으며, `RuBio`와 `Qwen` 기본은 `temperature=0`·`max_tokens=1024`·`thinking` 끔으로, 범용 모델은 표기한 추론 강도로 CLI 1회 실행해 출력을 고정함(응답 시간 중앙값 `GPT` `575.6초`, `Opus` `41.3초`, `Gemini` `87.9초`).
 **RuBench v1**은 새 문항·새 상황·가상 학생 페르소나로 구축한 2022 개정 『생명과학』 서술형 진단 벤치마크입니다(19개 성취기준, 대단원 Ⅰ–Ⅲ 298문항 중 `dev = 200` / `test = 98`, 가상 학생 30명 × 성취수준 A–E 층화 배정). 학습 문항과의 5-gram 중복(`< 0.40`) 및 장면·사례 중복을 차단하고, 모범답안·허용 표현을 가린 채 생성한 학생 답안을 3개 프론티어 모델(`GPT-6 Astra xhigh` · `Opus 5.5 xhigh` · `Grok 4.7 xhigh`) 블라인드 채점, 2차 블라인드 재채점, 인간 검수로 확정했습니다. 모든 모델은 실제 서비스와 동일한 조건(첫 턴 `feedback_mode="hint_1"`, `reveal_requested=false`, `history=[]`, `temperature=0`, `enable_thinking=false`, **JSON 스키마 강제 없음**)에서 평가합니다.
@@ -63,9 +64,54 @@ tags:
 - **기준 밖 과학 오류(`other_issues`) 탐지**: 기준 밖 오류 유무 일치율은 `RuBio-4B v2`가 `153/200`(`76.5%`), 기본 모델이 `150/200`(`75.0%`)입니다. 오류가 있는 답안 53건의 탐지(TP)는 미세조정 모델이 `10/53`(`18.9%`)으로 기본 모델 `4/53`(`7.5%`)보다 높고, 오류 없는 답안 147건에서의 오탐(FP)은 각각 `4건`(특이도 `97.3%`)과 `1건`(특이도 `99.3%`)으로, 두 모델 모두 오류가 있는 답안의 대부분을 놓칩니다.
 - **혼동 행렬에서 보이는 판정 쏠림 차이**: 기본 모델 `Qwen3.5-4B`는 정답이 `missing`인 기준 142건 중 `37건`(`26.1%`)과 정답이 `unclear`인 기준 46건 중 `26건`(`56.5%`)을 `supported`로 관대하게 과다 판정했으나, `RuBio-4B v2`에서는 이 오판이 각각 `18건`(`12.7%`)과 `9건`(`19.6%`)으로 줄었습니다. 반대로 정답이 `supported`(283기준)나 `missing`(142기준)인 항목을 `unclear`로 보수적 판정하는 빈도는 기본 모델(`11건 = 3.9%`, `4건 = 2.8%`)보다 `RuBio-4B v2`(`28건 = 9.9%`, `20건 = 14.1%`)에서 더 높았습니다.
 ## 2. 학습 설정
 - Unsloth `bf16` LoRA (`r=16`, `alpha=32`, 대상 모듈 `q·k·v·o·gate·up·down`)를 학습한 뒤 16비트 전체 모델로 병합했습니다(2 epoch / 300스텝, 학습률 `1e-4` cosine 스케줄러, 5% 선형 예열, 최대 길이 4,096, `thinking` 비활성화, assistant JSON 출력 구간에만 손실 적용).
 ## 3. 입력 및 출력 규격

 - **시범 서비스**: https://mulk.codes/rubio (RuBio-4B · RuBio-27B 운영 중)
 - **상위 모델**: [mulkzz/RuBio-27B](https://huggingface.co/mulkzz/RuBio-27B)
+- **관련 모델**: [mulkzz/RuBio-8B](https://huggingface.co/mulkzz/RuBio-8B) · [mulkzz/RuBio-9B](https://huggingface.co/mulkzz/RuBio-9B)
 ### RuBench v1 벤치마크 성능 요약 (Out-of-Distribution Dev, n = 200)
 <img src="./figures/rubench_v1_dev200_all.svg" width="100%" style="width: 100%; height: auto;" alt="Fig. 2 | RuBench v1 (dev n = 200) criterion accuracy of RuBio, base Qwen models, and general-purpose LLMs" />
+> **Fig. 2 | RuBench v1 (dev 분할, n = 200 답안 · 604 평가 기준)에서 RuBio-4B v2를 포함한 7개 모델(미세조정 RuBio 쌍, 기본 Qwen 쌍, 범용 프론티어 모델, 참고 모델)의 진단 성능 비교.** `RuBio-4B v2`(`77.2% [73.4, 80.9]`, 2차 2/3 제외 `79.9% [76.0, 83.8]`)는 기본 모델 `Qwen3.5-4B`(`70.9% [67.3, 74.5]`)보다 유의하게 높고 27B 모델 쌍(`RuBio-27B` `86.9% [83.8, 89.9]`, `Qwen3.8-27B` `81.8% [78.2, 85.4]`) 및 범용 모델(`GPT-6 Astra` `91.2% [88.6, 93.7]`, `Claude Opus 5.5` `94.2% [92.0, 96.4]`)보다 낮음. **a**, 7개 모델의 기준 판정 정확도(답안 단위 군집 95% CI, 막대)와 2차 2/3 다수결 제외 값(빈 마름모)으로, 4B 쌍(`RuBio-4B v2` `77.2% [73.4, 80.9]` · 제외 `79.9% [76.0, 83.8]` · 기준 micro `467/604`; `Qwen3.5-4B` `70.9% [67.3, 74.5]` · 제외 `74.5% [71.0, 78.1]` · micro `427/604`), 27B 쌍(`RuBio-27B` `86.9% [83.8, 89.9]` · 제외 `90.0% [87.1, 92.9]` · micro `524/604`; `Qwen3.8-27B` `81.8% [78.2, 85.4]` · 제외 `85.0% [81.4, 88.5]` · micro `495/604`), 범용(`GPT-6 Astra (xhigh)`† `91.2% [88.6, 93.7]` · 제외 `95.6% [93.6, 97.5]` · micro `550/604`; `Claude Opus 5.5 (xhigh)`† `94.2% [92.0, 96.4]` · 제외 `96.1% [94.2, 98.0]` · micro `570/604`), 참고(`Gemini 4 Argon (High)`, 빗금, `91.2% [88.5, 94.0]` · 제외 `93.6% [91.0, 96.2]` · micro `551/604`) 순서임. **b**, 답안 단위 짝 차이(b − a, %p, 95% CI; 하한 > 0 개선, 상한 < 0 악화, 그 밖 차이 없음) forest plot(전체 = 채운 원, 다수결 제외 = 빈 원). `RuBio-4B − Qwen3.5-4B`(전체 `+6.3%p [+2.6, +9.9]`·제외 `+5.3%p [+1.2, +9.5]`)와 `RuBio-27B − Qwen3.8-27B`(전체 `+5.1%p [+1.1, +9.1]`·제외 `+5.0%p [+1.1, +9.0]`)는 모두 공식 비교 조건(`SYSTEM_PROMPT+POLICY`)에서 유의하게 개선된 반면, 회색 보조 행인 동일 프롬프트 조건(`Qwen3.8-27B` 기본에 `SYSTEM_PROMPT`만 제공 시 전체 `84.3% [81.2, 87.5]`·제외 `87.3% [84.3, 90.3]`, JSON 오류 `0`)과의 차이 `RuBio-27B − Qwen3.8-27B(같은 프롬프트*)`는 전체 `+2.5%p [-0.9, +6.0]`·제외 `+2.7%p [-0.8, +6.2]`로 유의한 차이가 없음. 규모 간 비교인 `RuBio-27B − RuBio-4B`는 전체 `+9.7%p [+6.1, +13.3]`·제외 `+10.1%p [+6.5, +13.7]`로 개선됨. `GPT − RuBio-27B`(전체 `+4.3%p [+1.1, +7.5]`·제외 `+5.6%p [+2.4, +8.7]`; 대 `Qwen3.8-27B` `+9.4%p [+5.7, +13.0]`·제외 `+10.6%p [+7.3, +14.0]`)와 `Opus − RuBio-27B`(전체 `+7.3%p [+4.2, +10.5]`·제외 `+6.1%p [+3.1, +9.1]`; 대 `Qwen3.8-27B` `+12.4%p [+9.2, +15.6]`·제외 `+11.1%p [+7.9, +14.4]`)는 전체·다수결 제외 모두 유의하게 높고, `Opus − GPT`는 전체에서 `+3.0%p [+0.4, +5.6]`으로 개선이나 다수결 제외 시 `+0.5%p [-1.5, +2.5]`로 차이 없음. **c**, 보조 지표 5종 히트맵으로, 오개념 id 일치(`133`기준: `RuBio-4B v2` `106/133 = 79.7%`, `Qwen3.5-4B` `78/133 = 58.6%`, `RuBio-27B` `118/133 = 88.7%`, `Qwen3.8-27B` `107/133 = 80.5%`, `GPT` `129/133 = 97.0%`, `Opus` `127/133 = 95.5%`, `Gemini` `122/133 = 91.7%`), 기준 밖 오류 재현율(TP, `53`답안: 각각 `10/53 = 18.9%`, `4/53 = 7.5%`, `46/53 = 86.8%`, `29/53 = 54.7%`, `33/53 = 62.3%`, `52/53 = 98.1%`, `45/53 = 84.9%`), 오류 없음 특이도(TN, `147`답안: 각각 `143/147 = 97.3%`, `146/147 = 99.3%`, `127/147 = 86.4%`, `127/147 = 86.4%`, `144/147 = 98.0%`, `143/147 = 97.3%`, `145/147 = 98.6%`), `answer_status` 규칙 일치(`200`답안: 각각 `124/200 = 62.0%`, `89/200 = 44.5%`, `154/200 = 77.0%`, `120/200 = 60.0%`, `176/200 = 88.0%`, `184/200 = 92.0%`, `179/200 = 89.5%`), 유효 JSON(`Qwen3.8-27B` `195/200 = 97.5%`, `Gemini` `198/200 = 99.0%`, 나머지 5개 모델 `200/200 = 100.0%`)을 제시하며, `RuBio-4B v2`는 기준 밖 오류 재현율이 `10/53 = 18.9%`(특이도 `143/147 = 97.3%`)로 낮은 반면 RuIso-v6로 재학습한 `RuBio-27B`(`mulkzz/RuBio-27B@c1cfefdb`)는 기준 밖 오류 재현율이 `46/53 = 86.8%`(특이도 `127/147 = 86.4%`)로 높아짐. **d**, 정답 확정 출처별 기준 정확도와 Wilson CI로, `R1 3/3`(`521`기준: 같은 모델 순 `431/521 = 82.7%`, `402/521 = 77.2%`, `483/521 = 92.7%`, `461/521 = 88.5%`, `510/521 = 97.9%`, `510/521 = 97.9%`, `496/521 = 95.2%`), `R2 3/3`(`23`기준: `10/23 = 43.5%`, `4/23 = 17.4%`, `11/23 = 47.8%`, `5/23 = 21.7%`, `14/23 = 60.9%`, `18/23 = 78.3%`, `19/23 = 82.6%`), `R2 2/3`(`51`기준: `23/51 = 45.1%`, `15/51 = 29.4%`, `23/51 = 45.1%`, `24/51 = 47.1%`, `22/51 = 43.1%`, `37/51 = 72.5%`, `32/51 = 62.7%`), 사람 판정(`human`, `9`기준: `3/9 = 33.3%`, `6/9 = 66.7%`, `7/9 = 77.8%`, `5/9 = 55.6%`, `4/9 = 44.4%`, `5/9 = 55.6%`, `4/9 = 44.4%`)임. †`GPT-6 Astra (xhigh)`와 `Opus 5.5 (xhigh)`는 1차 정답 패널(`GPT-6 Astra xhigh`·`Opus 5.5 xhigh`·`Grok 4.7 xhigh`)이므로 `R1 3/3` 항목(`604`개 중 `521`개)에 자기 블라인드 판정이 들어 있어 점수가 자기 일치를 포함한 상한이며, 2차 다수결(`R2 2/3`)에도 2차 패널인 `GPT`·`Opus`·`Gemini`의 투표가 참여함. `Gemini 4 Argon (High)`은 의도된 라벨로 가상 학생 답을 쓴 모델이라 참고값으로만 제시하고 비교 판정에서 제외하며(`Gemini − RuBio-27B` 단순 차이는 전체 `+4.4%p [+1.0, +7.8]`·제외 `+3.6%p [+0.2, +7.0]`), 1차 패널인 `Grok 4.7`은 모델로 평가하지 않음. 평가 조건은 첫 턴(`feedback_mode="hint_1"`, `reveal_requested=false`, `history=[]`)·JSON 스키마 강제 없음·깨진 JSON은 오답 처리로 동일하되, `RuBio`는 `SYSTEM_PROMPT`만, 기본·범용 모델은 `SYSTEM_PROMPT+POLICY`를 입력받으며, `RuBio`와 `Qwen` 기본은 `temperature=0`·`max_tokens=1024`·`thinking` 끔으로, 범용 모델은 표기한 추론 강도로 CLI 1회 실행해 출력을 고정함(응답 시간 중앙값 `GPT` `575.6초`, `Opus` `41.3초`, `Gemini` `87.9초`). 그림의 `RuBio-27B`는 2026-11-04 `main`을 교체한 RuIso-v6 재학습본(`mulkzz/RuBio-27B@c1cfefdb`, RunPod A100 Pod · vLLM 0.32.0)이며, `RuBio-4B v2`는 RunPod Serverless(`worker-v1-vllm`, vLLM 0.32.0) 실행 결과이고 27B `main` 교체 결정에는 같은 dev 분할이 사용됨.
 **RuBench v1**은 새 문항·새 상황·가상 학생 페르소나로 구축한 2022 개정 『생명과학』 서술형 진단 벤치마크입니다(19개 성취기준, 대단원 Ⅰ–Ⅲ 298문항 중 `dev = 200` / `test = 98`, 가상 학생 30명 × 성취수준 A–E 층화 배정). 학습 문항과의 5-gram 중복(`< 0.40`) 및 장면·사례 중복을 차단하고, 모범답안·허용 표현을 가린 채 생성한 학생 답안을 3개 프론티어 모델(`GPT-6 Astra xhigh` · `Opus 5.5 xhigh` · `Grok 4.7 xhigh`) 블라인드 채점, 2차 블라인드 재채점, 인간 검수로 확정했습니다. 모든 모델은 실제 서비스와 동일한 조건(첫 턴 `feedback_mode="hint_1"`, `reveal_requested=false`, `history=[]`, `temperature=0`, `enable_thinking=false`, **JSON 스키마 강제 없음**)에서 평가합니다.
 - **기준 밖 과학 오류(`other_issues`) 탐지**: 기준 밖 오류 유무 일치율은 `RuBio-4B v2`가 `153/200`(`76.5%`), 기본 모델이 `150/200`(`75.0%`)입니다. 오류가 있는 답안 53건의 탐지(TP)는 미세조정 모델이 `10/53`(`18.9%`)으로 기본 모델 `4/53`(`7.5%`)보다 높고, 오류 없는 답안 147건에서의 오탐(FP)은 각각 `4건`(특이도 `97.3%`)과 `1건`(특이도 `99.3%`)으로, 두 모델 모두 오류가 있는 답안의 대부분을 놓칩니다.
 - **혼동 행렬에서 보이는 판정 쏠림 차이**: 기본 모델 `Qwen3.5-4B`는 정답이 `missing`인 기준 142건 중 `37건`(`26.1%`)과 정답이 `unclear`인 기준 46건 중 `26건`(`56.5%`)을 `supported`로 관대하게 과다 판정했으나, `RuBio-4B v2`에서는 이 오판이 각각 `18건`(`12.7%`)과 `9건`(`19.6%`)으로 줄었습니다. 반대로 정답이 `supported`(283기준)나 `missing`(142기준)인 항목을 `unclear`로 보수적 판정하는 빈도는 기본 모델(`11건 = 3.9%`, `4건 = 2.8%`)보다 `RuBio-4B v2`(`28건 = 9.9%`, `20건 = 14.1%`)에서 더 높았습니다.
+### RuIso-v6 재학습 결과 (main 미교체, `ruiso-v6` 브랜치)
+- **재학습 까닭**: 최소쌍 진단에서 기존 RuBio 모델이 학생 답의 추측 말투("~인 것 같아요")와 용어 나열 문체를 오답 신호로 삼아 맞는 설명을 깎고, 평가 기준 밖의 과학 오류가 들어 있으면 이미 충족된 기준까지 함께 깎는 경향이 확인되었습니다. 이를 줄이기 위해 2026-11-03–04에 RuBio 4개 규모(4B·8B·9B·27B)를 동일한 보강 데이터셋(**RuIso-v6**)으로 다시 학습하고 RuBench v1 `dev` 200건으로 평가했습니다.
+- **데이터 보강 (RuIso-v6)**: 기존 합성 학습 데이터 `2,400`행(문항 계열 300개 × 사례 10개 중 학습 분할, 대단원 Ⅰ·Ⅱ·Ⅲ, 계열 단위 분할)에 새 행 `1,179`행을 더해 `train = 3,579`행으로 구성했습니다(`dev = 300`행은 기존 그대로 유지).
+  - **문체 분리 변형 (`708`행)**: 같은 문항 계열의 첫 턴 답에서 과학적 내용·오류·누락은 그대로 두고 문체만 바꾼 변형으로, 추측 말투 `235`행·압축 용어 나열 `236`행·1–2문장으로 짧게 쓴 답 `237`행입니다. 기준별 라벨(`checks`)은 원본으로 고정하고 블라인드 재채점으로 일치를 확인했습니다.
+  - **기준 밖 오류 분리 변형 (`471`행)**: 맞거나 부분적으로 맞는 답에 평가 기준과 무관한 명백한 과학 오류를 한 군데 넣은 행입니다. 기준별 판정은 원본으로 고정하고 `other_issues`만 추가했으며, `answer_status`는 규칙으로 다시 계산했습니다.
+  - 학생 답 다시 쓰기는 `Claude Opus 5.5 (xhigh)`, 블라인드 채점과 피드백 작성은 `Gemini 4 Argon (High)`이 수행했습니다. 계획한 `1,200`행 중 고정 라벨과 블라인드 라벨이 갈린 행은 버리거나 검수 대기열로 보내 `1,179`행이 남았습니다. RuBench 문항·답과 최소쌍 변형은 학습에 넣지 않았으며, 모든 학생 답은 합성 데이터이고 실제 학생 자료가 아닙니다.
+- **재학습 설정**: 데이터 외 하이퍼파라미터는 기존 공개판과 같습니다. LoRA `r=16`·`alpha=32`·학습률 `1e-4`·2에폭(`448`스텝)·배치 `2×8`·최대 길이 `4,096`·판정 토큰 가중 `W=1.0`으로 학습하고 `50`스텝마다(마지막 `448`스텝 포함) `dev eval_loss`로 체크포인트를 고른 뒤 16비트(`bf16`)로 병합했습니다(`unsloth`·`unsloth_zoo` `2026.10.11`, `transformers` `5.17.0`, `trl` `1.15.0`, `torch` `2.14.1+cu132`, `fla` `0.6.0`). 4B는 NVIDIA A40 48GB GPU 1대에서 시드 `3407`과 `3408`을 동시에 학습했습니다(시드 `3408`: 최종 `448`스텝 선택, `best_eval_loss = 0.4569`, 평균 `train_loss = 0.4033`, `12,825초`; 시드 `3407`: 최종 `448`스텝 선택, `best_eval_loss = 0.4579`, 평균 `train_loss = 0.4042`, `12,887초`).
+- **결과를 보기 전에 정한 규칙**:
+  1. 시드가 둘인 규모는 `best_eval_loss`가 낮은 시드를 공개 후보로 고릅니다. 4B에서는 `0.4569`(시드 `3408`) < `0.4579`(시드 `3407`)이므로 `4b-v6-s3408`이 공개 후보가 되었습니다.
+  2. 같은 Pod·같은 vLLM `0.32.0`(`--max-model-len 8192 --max-num-seqs 32 --language-model-only`)에서 현재 HF `main`(리비전 `6f3e415a` 고정)을 다시 돌린 대조군(`4b-pub-pod`)과 비교해, RuBench v1 `dev` 200건의 주 지표(기준 판정 정확도) 짝 차이 점추정이 `0`보다 크고 정답 `missing`·`contradicted` 기준의 재현율이 각각 대조군 이상이면 `main` 가중치를 바꾸고, 아니면 `ruiso-v6` 브랜치에만 둡니다. 이 규칙은 신뢰구간 하한을 요구하지 않으므로 교체됐더라도 95% CI가 `0`을 포함하면 함께 밝히며, 짝 차이의 통계적 판정은 95% CI(하한 > `0` 개선, 상한 < `0` 악화, 그 밖 차이 없음)로 정합니다.
+#### 1. 같은 Pod 대조군 대비 RuBench v1 `dev` (n = 200 답안 · 604 기준) 성능 비교
+| 모델 ID | 모델 설명 | HF 리비전 | 주 지표 [95% CI] | 2차 2/3 제외 | 미시 일치 | `supported` 인정 | `contradicted` 재현 | `missing` 재현 | `unclear` 재현 | 기준 밖 오류 TP/FP (오류 53건) | `answer_status` 일치 | JSON 오류 |
+|---|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
+| `4b-v6-s3408` | RuBio-4B RuIso-v6 (시드 3408, 공개 후보) | `mulkzz/RuBio-4B@553fb15b` | 78.1% `[74.4, 81.9]` | 82.1% | 472/604 | 245/283 (86.6%) | 117/133 (88.0%) | 95/142 (66.9%) | 15/46 (32.6%) | 36/28 | 130/200 | 3 |
+| `4b-pub-pod` | RuBio-4B 공개판 (같은 Pod 대조군, `main`) | `mulkzz/RuBio-4B@6f3e415a` | 77.0% `[73.2, 80.7]` | 79.7% | 465/604 | 230/283 (81.3%) | **122/133 (91.7%)** | 92/142 (64.8%) | 21/46 (45.7%) | 9/4 | 123/200 | 0 |
+| `4b-v6-s3407` | RuBio-4B RuIso-v6 (시드 3407, 후보 외) | `local rubio-4b-v6-s3407` | 80.0% `[76.4, 83.6]` | 83.5% | 483/604 | 252/283 (89.0%) | 116/133 (87.2%) | 97/142 (68.3%) | 18/46 (39.1%) | 38/31 | 126/200 | 2 |
+- **답안 단위 짝 차이 (재학습 − 같은 Pod 대조군 `4b-pub-pod`, 95% CI)**:
+  - `4b-v6-s3408 − 4b-pub-pod`: 전체 **+1.2%p** `[-2.0, +4.4]` (**차이 없음**); 2차 2/3 제외 **+2.5%p** `[-1.1, +6.0]` (**차이 없음**)
+  - `4b-v6-s3407 − 4b-pub-pod`: 전체 **+3.0%p** `[+0.0, +6.0]` (**개선**); 2차 2/3 제외 **+3.8%p** `[+0.4, +7.1]` (**개선**)
+#### 2. 최소쌍 A–E 문체 민감도 진단 (정답이 `supported`인 기준만; 변형 뒤에도 `supported`로 남아야 이상적)
+표본 수가 작아 통계적 확정이 아닌 경향 파악용으로만 해석합니다. 각 칸은 `원본→변형 supported 수/기준 수 (+얻음 −잃음)`과 부호 검정 `p`값입니다.
+| 최소쌍 변형 | 표본 수 | `4b-v6-s3408` (공개 후보) | `4b-pub-pod` (현재 `main`) | `4b-v6-s3407` (시드 3407) |
+|---|---:|---:|---:|---:|
+| **A. 추측 말투 제거** | 36답 · 74기준 | 55→61/74 (+6 −0, `p=0.0312`) | 46→58/74 (+15 −3, `p=0.00754`) | 62→62/74 (+1 −1, `p=1`) |
+| **B. 추측 말투 삽입** | 12답 · 28기준 | 25→28/28 (+3 −0, `p=0.25`) | 24→21/28 (+1 −4, `p=0.375`) | 26→26/28 (+0 −0, `p=1`) |
+| **C. 교과서식 문장 정돈** | 12답 · 27기준 | 27→23/27 (+0 −4, `p=0.125`) | 23→27/27 (+4 −0, `p=0.125`) | 22→26/27 (+4 −0, `p=0.125`) |
+| **D. 나열 → 관계 문장** (A 변형과 비교) | 10답 · 25기준 | 23→24/25 (+2 −1, `p=1`) | 21→25/25 (+4 −0, `p=0.125`) | 24→25/25 (+1 −0, `p=1`) |
+| **E. 문장 → 용어 나열** | 12답 · 26기준 | 22→25/26 (+4 −1, `p=0.375`) | 25→17/26 (+1 −9, `p=0.0215`) | 25→25/26 (+1 −1, `p=1`) |
+- **판정 및 정리**:
+  - 공개 후보 `4b-v6-s3408`은 같은 Pod 대조군(`4b-pub-pod`) 대비 주 지표 짝 차이가 `+1.2%p [-2.0, +4.4]`로 **차이 없음**이고, 점추정(`+1.2%p > 0`)과 `missing` 재현율(`95/142` ≥ `92/142`) 조건은 충족했으나 `contradicted` 재현율이 `117/133 (88.0%)` < `122/133 (91.7%)`로 낮아 교체 규칙을 통과하지 못했습니다. 따라서 이 저장소의 `main`은 기존 `RuBio-4B v2`(`6f3e415a`)를 그대로 유지하고, 재학습본은 `ruiso-v6` 브랜치(`553fb15b`, `safetensors` 2개 합 `9.32 GB`)에만 두었습니다.
+  - 함께 학습한 시드 `3407`(`4b-v6-s3407`)은 대조군 대비 주 지표 짝 차이가 `+3.0%p [+0.0, +6.0]`(2차 2/3 제외 `+3.8%p [+0.4, +7.1]`)로 더 높았지만, 결과를 보기 전에 정한 규칙상 `eval_loss`로 고른 `4b-v6-s3408`(`0.4569` 대 `0.4579`)만 공개 후보였으며, `4b-v6-s3407` 역시 `contradicted` 재현율(`116/133 = 87.2%`)이 대조군보다 낮습니다.
+  - 재학습본(`4b-v6-s3408`)은 대조군 대비 `supported` 인정률(`81.3%` → `86.6%`)과 용어 나열·추측 말투 답안의 인정 건수가 늘고 기준 밖 과학 오류(`other_issues`) 재현이 크게 늘었지만(TP `9/53` → `36/53`), 오류 없는 답안에서의 오탐도 함께 늘었으며(FP `4/147` → `28/147`), 스키마 강제 없는 생성에서 JSON 파싱 오류가 `0건`에서 `3건`으로 늘었습니다.
+- **`ruiso-v6` 브랜치 가중치 받는 법**:
+  - Transformers / `huggingface_hub`: `revision="ruiso-v6"` (또는 커밋 해시 `revision="553fb15b"`) 지정
+  - vLLM: `vllm serve mulkzz/RuBio-4B --revision ruiso-v6 --served-model-name bio-feedback --language-model-only --max-model-len 8192`
 ## 2. 학습 설정
 - Unsloth `bf16` LoRA (`r=16`, `alpha=32`, 대상 모듈 `q·k·v·o·gate·up·down`)를 학습한 뒤 16비트 전체 모델로 병합했습니다(2 epoch / 300스텝, 학습률 `1e-4` cosine 스케줄러, 5% 선형 예열, 최대 길이 4,096, `thinking` 비활성화, assistant JSON 출력 구간에만 손실 적용).
+- `ruiso-v6` 브랜치(`mulkzz/RuBio-4B@553fb15b`)는 보강 데이터 RuIso-v6(`train = 3,579`, `dev = 300`)로 동일한 LoRA 하이퍼파라미터(2 epoch / 448스텝, 판정 가중 `W=1.0`, NVIDIA A40 48GB)에서 재학습한 가중치입니다(1절의 RuIso-v6 재학습 결과 참조).
 ## 3. 입력 및 출력 규격

README_en.md CHANGED Viewed

@@ -10,6 +10,7 @@
 - **Live Demo Service**: https://mulk.codes/rubio (running both RuBio-4B & RuBio-27B)
 - **Flagship Model**: [mulkzz/RuBio-27B](https://huggingface.co/mulkzz/RuBio-27B/blob/main/README_en.md)
 ### RuBench v1 Benchmark Evaluation (Out-of-Distribution Dev, n = 200)
@@ -19,7 +20,7 @@
 <img src="./figures/rubench_v1_dev200_all.svg" width="100%" style="width: 100%; height: auto;" alt="Fig. 2 | RuBench v1 (dev n = 200) criterion accuracy of RuBio, base Qwen models, and general-purpose LLMs" />
-> **Fig. 2 | Cross-model diagnostic evaluation on RuBench v1 (dev split, n = 200 answers · 604 criteria) comparing RuBio-4B v2 against its base model, the 27B pair, frontier general-purpose LLMs, and the answer-generator reference model.** `RuBio-4B v2` (`77.2% [73.4, 80.9]`, or `79.9% [76.0, 83.8]` excluding R2 2/3 majority items) significantly outperforms its base model `Qwen3.5-4B` (`70.9% [67.3, 74.5]`) and ranks below the 27B pair (`RuBio-27B` `84.5% [81.1, 87.8]`, `Qwen3.8-27B` `81.8% [78.2, 85.4]`) and frontier general-purpose models (`GPT-6 Astra` `91.2% [88.6, 93.7]`, `Claude Opus 5.5` `94.2% [92.0, 96.4]`). **a**, Criterion status accuracy (answer-clustered 95% CI, bars) and accuracy excluding R2 2/3 majority items (open diamonds) across seven models in order: the 4B pair (`RuBio-4B v2` `77.2% [73.4, 80.9]`, excl. R2 2/3 `79.9% [76.0, 83.8]`, criterion micro `467/604`; `Qwen3.5-4B` `70.9% [67.3, 74.5]`, excl. `74.5% [71.0, 78.1]`, micro `427/604`), the 27B pair (`RuBio-27B` `84.5% [81.1, 87.8]`, excl. `88.4% [85.2, 91.6]`, micro `508/604`; `Qwen3.8-27B` `81.8% [78.2, 85.4]`, excl. `85.0% [81.4, 88.5]`, micro `495/604`), general-purpose models (`GPT-6 Astra (xhigh)`† `91.2% [88.6, 93.7]`, excl. `95.6% [93.6, 97.5]`, micro `550/604`; `Claude Opus 5.5 (xhigh)`† `94.2% [92.0, 96.4]`, excl. `96.1% [94.2, 98.0]`, micro `570/604`), and the reference model (`Gemini 4 Argon (High)`, hatched, `91.2% [88.5, 94.0]`, excl. `93.6% [91.0, 96.2]`, micro `551/604`). **b**, Answer-clustered paired differences (b − a, %p, 95% CI; lower bound > 0 indicates improvement, upper bound < 0 degradation, otherwise no difference) for all criteria (filled circles) and excluding R2 2/3 majority items (open circles). `RuBio-4B − Qwen3.5-4B` shows a statistically significant improvement (`+6.3%p [+2.6, +9.9]` all; `+5.3%p [+1.2, +9.5]` excl. R2 2/3), whereas `RuBio-27B − Qwen3.8-27B` shows no significant difference (`+2.7%p [-1.3, +6.6]` all; `+3.5%p [-0.7, +7.6]` excl. R2 2/3), and the gray auxiliary row `RuBio-27B − Qwen3.8-27B (same prompt*)`—giving base `Qwen3.8-27B` only `SYSTEM_PROMPT`, which yields `84.3% [81.2, 87.5]` with `0` JSON errors—likewise shows no difference (`+0.1%p [-3.5, +3.7]`). Scaling from 4B to 27B (`RuBio-27B − RuBio-4B`) yields `+7.3%p [+3.6, +11.0]` (`+8.5%p [+4.7, +12.4]` excl. R2 2/3). Both `GPT − RuBio-27B` (`+6.7%p [+3.4, +10.0]` all; `+7.2%p [+3.8, +10.6]` excl. R2 2/3; vs. `Qwen3.8-27B`, `+9.4%p [+5.7, +13.0]` all and `+10.6%p [+7.3, +14.0]` excl. R2 2/3) and `Opus − RuBio-27B` (`+9.8%p [+6.4, +13.1]` all; `+7.7%p [+4.3, +11.0]` excl. R2 2/3; vs. `Qwen3.8-27B`, `+12.4%p [+9.2, +15.6]` all and `+11.1%p [+7.9, +14.4]` excl. R2 2/3) are significantly higher both overall and excluding R2 2/3, while `Opus − GPT` is improved overall (`+3.0%p [+0.4, +5.6]`) but shows no difference when R2 2/3 majority items are excluded (`+0.5%p [-1.5, +2.5]`). **c**, Secondary-metric heatmap across the same seven models: misconception ID match (`133` criteria: `106/133 = 79.7%`, `78/133 = 58.6%`, `117/133 = 88.0%`, `107/133 = 80.5%`, `129/133 = 97.0%`, `127/133 = 95.5%`, `122/133 = 91.7%`), out-of-rubric error recall (TP, `53` answers: `10/53 = 18.9%`, `4/53 = 7.5%`, `11/53 = 20.8%`, `29/53 = 54.7%`, `33/53 = 62.3%`, `52/53 = 98.1%`, `45/53 = 84.9%`), error-free specificity (TN, `147` answers: `143/147 = 97.3%`, `146/147 = 99.3%`, `143/147 = 97.3%`, `127/147 = 86.4%`, `144/147 = 98.0%`, `143/147 = 97.3%`, `145/147 = 98.6%`), `answer_status` rule agreement (`200` answers: `124/200 = 62.0%`, `89/200 = 44.5%`, `141/200 = 70.5%`, `120/200 = 60.0%`, `176/200 = 88.0%`, `184/200 = 92.0%`, `179/200 = 89.5%`), and valid JSON rate (`195/200 = 97.5%` for `Qwen3.8-27B`, `198/200 = 99.0%` for `Gemini`, and `200/200 = 100.0%` for the other five models); the main weakness of `RuBio` is low out-of-rubric error recall (~20% for both models: `10/53 = 18.9%` for 4B and `11/53 = 20.8%` for 27B). **d**, Criterion accuracy and Wilson CIs by gold consensus source: `R1 3/3` (`521` criteria: `431/521 = 82.7%`, `402/521 = 77.2%`, `472/521 = 90.6%`, `461/521 = 88.5%`, `510/521 = 97.9%`, `510/521 = 97.9%`, `496/521 = 95.2%`), `R2 3/3` (`23` criteria: `10/23 = 43.5%`, `4/23 = 17.4%`, `12/23 = 52.2%`, `5/23 = 21.7%`, `14/23 = 60.9%`, `18/23 = 78.3%`, `19/23 = 82.6%`), `R2 2/3` (`51` criteria: `23/51 = 45.1%`, `15/51 = 29.4%`, `18/51 = 35.3%`, `24/51 = 47.1%`, `22/51 = 43.1%`, `37/51 = 72.5%`, `32/51 = 62.7%`), and `human` adjudication (`9` criteria: `3/9 = 33.3%`, `6/9 = 66.7%`, `6/9 = 66.7%`, `5/9 = 55.6%`, `4/9 = 44.4%`, `5/9 = 55.6%`, `4/9 = 44.4%`). †`GPT-6 Astra (xhigh)` and `Opus 5.5 (xhigh)` served on the Round-1 gold panel (`GPT-6 Astra xhigh`, `Opus 5.5 xhigh`, `Grok 4.7 xhigh`), so the `R1 3/3` tier (`521` of `604` criteria) contains their own blind judgments and their scores represent upper bounds that include self-agreement; Round-2 majority (`R2 2/3`) items also include votes from the Round-2 panel (`GPT`, `Opus`, `Gemini`). `Gemini 4 Argon (High)` wrote the synthetic student answers with intended labels and is reported as a reference only and excluded from comparative testing (raw difference `Gemini − RuBio-27B`: `+6.8%p [+3.3, +10.3]` all, `+5.2%p [+1.4, +8.9]` excl. R2 2/3); Round-1 panelist `Grok 4.7` was not evaluated as a model. All models are evaluated on the first turn (`feedback_mode="hint_1"`, `reveal_requested=false`, `history=[]`) without JSON schema enforcement (malformed JSON counted as wrong), with `SYSTEM_PROMPT` only for `RuBio` and `SYSTEM_PROMPT+POLICY` for base and general-purpose models; `RuBio` and base `Qwen` models run at `temperature=0`, `max_tokens=1024`, and thinking disabled, whereas general-purpose models use a single fixed vendor CLI run at the listed reasoning effort (median latency: `GPT` `575.6 s`, `Opus` `41.3 s`, `Gemini` `87.9 s`).
 **RuBench v1** is a newly constructed out-of-distribution (OOD) benchmark for 2022 Revised high-school *Life Science* constructed-response diagnosis (`19` achievement standards across Units I–III, `298` validated question slots split into `dev = 200` and `test = 98`, answered by `30` synthetic student personas stratified across achievement levels A–E). Every question enforces strict n-gram (`5-gram overlap < 0.40`) and scenario deduplication against the training set; student answers are generated without access to exemplar answers or accepted expressions; and gold labels (`604` criteria, `133` contradicted criteria, and `200` out-of-rubric error indicators in `dev`) are established via a 3-model blind grading panel (`GPT-6 Astra xhigh`, `Opus 5.5 xhigh`, `Grok 4.7 xhigh`), a 2nd-round blind adjudication panel, and human review. Both models are evaluated under identical zero-shot first-turn production settings (`feedback_mode="hint_1"`, `reveal_requested=false`, `history=[]`, `temperature=0`, `enable_thinking=false`, **no JSON schema enforcement**).
@@ -48,9 +49,54 @@
 - **Out-of-Rubric Error (`other_issues`) Detection**: Overall binary match on `other_issues` is `153/200` (`76.5%`) for `RuBio-4B v2` vs. `150/200` (`75.0%`) for base `Qwen3.5-4B`. Among the 53 answers containing out-of-rubric scientific errors, `RuBio-4B v2` detects `10/53` (`18.9%`) compared to `4/53` (`7.5%`) for the base model, while false positives on the 147 error-free answers remain low for both (`4/147` with `97.3%` specificity vs. `1/147` with `99.3%` specificity); both models still miss most error-present answers.
 - **Prediction Bias Shift in the Confusion Matrix**: Base `Qwen3.5-4B` exhibits a strong lenient bias toward `supported`, misclassifying `37/142` (`26.1%`) of gold `missing` criteria and `26/46` (`56.5%`) of gold `unclear` criteria as `supported`; fine-tuning in `RuBio-4B v2` reduces these over-predictions to `18/142` (`12.7%`) and `9/46` (`19.6%`). Conversely, `RuBio-4B v2` more frequently defaults to conservative `unclear` judgments on gold `supported` (`28/283 = 9.9%` vs. base `11/283 = 3.9%`) and gold `missing` (`20/142 = 14.1%` vs. base `4/142 = 2.8%`) criteria.
 ## 2. Training Configuration
 - Fine-tuned using Unsloth `bf16` LoRA (`r=16`, `alpha=32`, target modules `q·k·v·o·gate·up·down`) and merged into full 16-bit weights (2 epochs / 300 steps, learning rate `1e-4` with cosine scheduler, 5% linear warmup, max sequence length 4,096, `thinking` disabled, loss masked to assistant JSON output tokens only).
 ## 3. Input & Output Specification

 - **Live Demo Service**: https://mulk.codes/rubio (running both RuBio-4B & RuBio-27B)
 - **Flagship Model**: [mulkzz/RuBio-27B](https://huggingface.co/mulkzz/RuBio-27B/blob/main/README_en.md)
+- **Related Models**: [mulkzz/RuBio-8B](https://huggingface.co/mulkzz/RuBio-8B) · [mulkzz/RuBio-9B](https://huggingface.co/mulkzz/RuBio-9B)
 ### RuBench v1 Benchmark Evaluation (Out-of-Distribution Dev, n = 200)
 <img src="./figures/rubench_v1_dev200_all.svg" width="100%" style="width: 100%; height: auto;" alt="Fig. 2 | RuBench v1 (dev n = 200) criterion accuracy of RuBio, base Qwen models, and general-purpose LLMs" />
+> **Fig. 2 | Cross-model diagnostic evaluation on RuBench v1 (dev split, n = 200 answers · 604 criteria) comparing RuBio-4B v2 against its base model, the 27B pair, frontier general-purpose LLMs, and the answer-generator reference model.** `RuBio-4B v2` (`77.2% [73.4, 80.9]`, or `79.9% [76.0, 83.8]` excluding R2 2/3 majority items) significantly outperforms its base model `Qwen3.5-4B` (`70.9% [67.3, 74.5]`) and ranks below the 27B pair (`RuBio-27B` `86.9% [83.8, 89.9]`, `Qwen3.8-27B` `81.8% [78.2, 85.4]`) and frontier general-purpose models (`GPT-6 Astra` `91.2% [88.6, 93.7]`, `Claude Opus 5.5` `94.2% [92.0, 96.4]`). **a**, Criterion status accuracy (answer-clustered 95% CI, bars) and accuracy excluding R2 2/3 majority items (open diamonds) across seven models in order: the 4B pair (`RuBio-4B v2` `77.2% [73.4, 80.9]`, excl. R2 2/3 `79.9% [76.0, 83.8]`, criterion micro `467/604`; `Qwen3.5-4B` `70.9% [67.3, 74.5]`, excl. `74.5% [71.0, 78.1]`, micro `427/604`), the 27B pair (`RuBio-27B` `86.9% [83.8, 89.9]`, excl. `90.0% [87.1, 92.9]`, micro `524/604`; `Qwen3.8-27B` `81.8% [78.2, 85.4]`, excl. `85.0% [81.4, 88.5]`, micro `495/604`), general-purpose models (`GPT-6 Astra (xhigh)`† `91.2% [88.6, 93.7]`, excl. `95.6% [93.6, 97.5]`, micro `550/604`; `Claude Opus 5.5 (xhigh)`† `94.2% [92.0, 96.4]`, excl. `96.1% [94.2, 98.0]`, micro `570/604`), and the reference model (`Gemini 4 Argon (High)`, hatched, `91.2% [88.5, 94.0]`, excl. `93.6% [91.0, 96.2]`, micro `551/604`). **b**, Answer-clustered paired differences (b − a, %p, 95% CI; lower bound > 0 indicates improvement, upper bound < 0 degradation, otherwise no difference) for all criteria (filled circles) and excluding R2 2/3 majority items (open circles). Both `RuBio-4B − Qwen3.5-4B` (`+6.3%p [+2.6, +9.9]` all; `+5.3%p [+1.2, +9.5]` excl. R2 2/3) and `RuBio-27B − Qwen3.8-27B` (`+5.1%p [+1.1, +9.1]` all; `+5.0%p [+1.1, +9.0]` excl. R2 2/3) show statistically significant improvements under the official comparison (`SYSTEM_PROMPT+POLICY`), whereas the gray auxiliary row `RuBio-27B − Qwen3.8-27B (same prompt*)`—giving base `Qwen3.8-27B` only `SYSTEM_PROMPT`, which yields `84.3% [81.2, 87.5]` overall and `87.3% [84.3, 90.3]` excl. R2 2/3 with `0` JSON errors—shows no significant difference (`+2.5%p [-0.9, +6.0]` all; `+2.7%p [-0.8, +6.2]` excl. R2 2/3). Scaling from 4B to 27B (`RuBio-27B − RuBio-4B`) yields `+9.7%p [+6.1, +13.3]` (`+10.1%p [+6.5, +13.7]` excl. R2 2/3). Both `GPT − RuBio-27B` (`+4.3%p [+1.1, +7.5]` all; `+5.6%p [+2.4, +8.7]` excl. R2 2/3; vs. `Qwen3.8-27B`, `+9.4%p [+5.7, +13.0]` all and `+10.6%p [+7.3, +14.0]` excl. R2 2/3) and `Opus − RuBio-27B` (`+7.3%p [+4.2, +10.5]` all; `+6.1%p [+3.1, +9.1]` excl. R2 2/3; vs. `Qwen3.8-27B`, `+12.4%p [+9.2, +15.6]` all and `+11.1%p [+7.9, +14.4]` excl. R2 2/3) are significantly higher both overall and excluding R2 2/3, while `Opus − GPT` is improved overall (`+3.0%p [+0.4, +5.6]`) but shows no difference when R2 2/3 majority items are excluded (`+0.5%p [-1.5, +2.5]`). **c**, Secondary-metric heatmap across the same seven models: misconception ID match (`133` criteria: `106/133 = 79.7%`, `78/133 = 58.6%`, `118/133 = 88.7%`, `107/133 = 80.5%`, `129/133 = 97.0%`, `127/133 = 95.5%`, `122/133 = 91.7%`), out-of-rubric error recall (TP, `53` answers: `10/53 = 18.9%`, `4/53 = 7.5%`, `46/53 = 86.8%`, `29/53 = 54.7%`, `33/53 = 62.3%`, `52/53 = 98.1%`, `45/53 = 84.9%`), error-free specificity (TN, `147` answers: `143/147 = 97.3%`, `146/147 = 99.3%`, `127/147 = 86.4%`, `127/147 = 86.4%`, `144/147 = 98.0%`, `143/147 = 97.3%`, `145/147 = 98.6%`), `answer_status` rule agreement (`200` answers: `124/200 = 62.0%`, `89/200 = 44.5%`, `154/200 = 77.0%`, `120/200 = 60.0%`, `176/200 = 88.0%`, `184/200 = 92.0%`, `179/200 = 89.5%`), and valid JSON rate (`195/200 = 97.5%` for `Qwen3.8-27B`, `198/200 = 99.0%` for `Gemini`, and `200/200 = 100.0%` for the other five models); whereas `RuBio-4B v2` has low out-of-rubric error recall (`10/53 = 18.9%`, specificity `143/147 = 97.3%`), the RuIso-v6 retrained `RuBio-27B` (`mulkzz/RuBio-27B@c1cfefdb`) raises out-of-rubric error recall to `46/53 = 86.8%` (specificity `127/147 = 86.4%`). **d**, Criterion accuracy and Wilson CIs by gold consensus source: `R1 3/3` (`521` criteria: `431/521 = 82.7%`, `402/521 = 77.2%`, `483/521 = 92.7%`, `461/521 = 88.5%`, `510/521 = 97.9%`, `510/521 = 97.9%`, `496/521 = 95.2%`), `R2 3/3` (`23` criteria: `10/23 = 43.5%`, `4/23 = 17.4%`, `11/23 = 47.8%`, `5/23 = 21.7%`, `14/23 = 60.9%`, `18/23 = 78.3%`, `19/23 = 82.6%`), `R2 2/3` (`51` criteria: `23/51 = 45.1%`, `15/51 = 29.4%`, `23/51 = 45.1%`, `24/51 = 47.1%`, `22/51 = 43.1%`, `37/51 = 72.5%`, `32/51 = 62.7%`), and `human` adjudication (`9` criteria: `3/9 = 33.3%`, `6/9 = 66.7%`, `7/9 = 77.8%`, `5/9 = 55.6%`, `4/9 = 44.4%`, `5/9 = 55.6%`, `4/9 = 44.4%`). †`GPT-6 Astra (xhigh)` and `Opus 5.5 (xhigh)` served on the Round-1 gold panel (`GPT-6 Astra xhigh`, `Opus 5.5 xhigh`, `Grok 4.7 xhigh`), so the `R1 3/3` tier (`521` of `604` criteria) contains their own blind judgments and their scores represent upper bounds that include self-agreement; Round-2 majority (`R2 2/3`) items also include votes from the Round-2 panel (`GPT`, `Opus`, `Gemini`). `Gemini 4 Argon (High)` wrote the synthetic student answers with intended labels and is reported as a reference only and excluded from comparative testing (raw difference `Gemini − RuBio-27B`: `+4.4%p [+1.0, +7.8]` all, `+3.6%p [+0.2, +7.0]` excl. R2 2/3); Round-1 panelist `Grok 4.7` was not evaluated as a model. All models are evaluated on the first turn (`feedback_mode="hint_1"`, `reveal_requested=false`, `history=[]`) without JSON schema enforcement (malformed JSON counted as wrong), with `SYSTEM_PROMPT` only for `RuBio` and `SYSTEM_PROMPT+POLICY` for base and general-purpose models; `RuBio` and base `Qwen` models run at `temperature=0`, `max_tokens=1024`, and thinking disabled, whereas general-purpose models use a single fixed vendor CLI run at the listed reasoning effort (median latency: `GPT` `575.6 s`, `Opus` `41.3 s`, `Gemini` `87.9 s`). `RuBio-27B` in this figure is the RuIso-v6 retrain (`mulkzz/RuBio-27B@c1cfefdb`, `main` since 2026-11-04) evaluated on a RunPod A100 Pod with vLLM 0.32.0 under identical generation settings, whereas `RuBio-4B v2` was run on RunPod Serverless (`worker-v1-vllm`, vLLM 0.32.0); the decision to replace the 27B `main` weights used this same dev split.
 **RuBench v1** is a newly constructed out-of-distribution (OOD) benchmark for 2022 Revised high-school *Life Science* constructed-response diagnosis (`19` achievement standards across Units I–III, `298` validated question slots split into `dev = 200` and `test = 98`, answered by `30` synthetic student personas stratified across achievement levels A–E). Every question enforces strict n-gram (`5-gram overlap < 0.40`) and scenario deduplication against the training set; student answers are generated without access to exemplar answers or accepted expressions; and gold labels (`604` criteria, `133` contradicted criteria, and `200` out-of-rubric error indicators in `dev`) are established via a 3-model blind grading panel (`GPT-6 Astra xhigh`, `Opus 5.5 xhigh`, `Grok 4.7 xhigh`), a 2nd-round blind adjudication panel, and human review. Both models are evaluated under identical zero-shot first-turn production settings (`feedback_mode="hint_1"`, `reveal_requested=false`, `history=[]`, `temperature=0`, `enable_thinking=false`, **no JSON schema enforcement**).
 - **Out-of-Rubric Error (`other_issues`) Detection**: Overall binary match on `other_issues` is `153/200` (`76.5%`) for `RuBio-4B v2` vs. `150/200` (`75.0%`) for base `Qwen3.5-4B`. Among the 53 answers containing out-of-rubric scientific errors, `RuBio-4B v2` detects `10/53` (`18.9%`) compared to `4/53` (`7.5%`) for the base model, while false positives on the 147 error-free answers remain low for both (`4/147` with `97.3%` specificity vs. `1/147` with `99.3%` specificity); both models still miss most error-present answers.
 - **Prediction Bias Shift in the Confusion Matrix**: Base `Qwen3.5-4B` exhibits a strong lenient bias toward `supported`, misclassifying `37/142` (`26.1%`) of gold `missing` criteria and `26/46` (`56.5%`) of gold `unclear` criteria as `supported`; fine-tuning in `RuBio-4B v2` reduces these over-predictions to `18/142` (`12.7%`) and `9/46` (`19.6%`). Conversely, `RuBio-4B v2` more frequently defaults to conservative `unclear` judgments on gold `supported` (`28/283 = 9.9%` vs. base `11/283 = 3.9%`) and gold `missing` (`20/142 = 14.1%` vs. base `4/142 = 2.8%`) criteria.
+### RuIso-v6 Retraining Results (main Retained, `ruiso-v6` Branch)
+- **Why We Retrained**: Minimal-pair diagnostics showed that RuBio tended to treat hedging expressions (e.g., `"~인 것 같아요"`, "I think...") and compact keyword-list phrasing as error signals—penalizing otherwise satisfied criteria—and tended to downgrade satisfied rubric criteria when an answer contained an unrelated out-of-rubric scientific error. On 2026-11-03–04, all four RuBio sizes (4B, 8B, 9B, 27B) were retrained on the same augmented dataset (**RuIso-v6**) and evaluated on RuBench v1 `dev` (`n = 200` answers · `604` criteria).
+- **Dataset Augmentation (RuIso-v6)**: We added `1,179` new rows to the original `2,400` synthetic training rows (`300` question families × `10` cases across Units I–III, split by family), yielding `train = 3,579` rows (`dev = 300` unchanged).
+  - **Style-isolation variants (`708` rows)**: First-turn answers from the same question families rewritten to change only surface style while preserving scientific content, errors, and omissions—`235` hedging-tone rewrites, `236` compressed term-list rewrites, and `237` short 1–2 sentence rewrites. Per-criterion labels (`checks`) were fixed to the source row and verified via blind re-grading.
+  - **Out-of-rubric error isolation variants (`471` rows)**: Correct or partially correct answers into which a single clear scientific error unrelated to the grading rubric was injected. Per-criterion `checks` were fixed to the source row, only `other_issues` was added, and `answer_status` was recomputed deterministically by rule.
+  - Student answer rewrites were generated by `Claude Opus 5.5 (xhigh)`, and blind verification grading and feedback drafting were performed by `Gemini 4 Argon (High)`. Out of `1,200` planned rows, rows where fixed labels disagreed with blind labels were discarded or routed to review, leaving `1,179` rows. No RuBench items, answers, or minimal-pair rewrites were included in training. All student answers are synthetic and contain no real student data.
+- **Retraining Setup**: All hyperparameters other than the training split were identical to the previous release: LoRA `r=16`, `alpha=32`, learning rate `1e-4`, 2 epochs (`448` steps), batch size `2×8`, max sequence length `4,096`, judgment token weight `W=1.0`, checkpoint selection by lowest `dev eval_loss` every `50` steps (including final step `448`), and 16-bit (`bf16`) weight merging (`unsloth` & `unsloth_zoo` `2026.10.11`, `transformers` `5.17.0`, `trl` `1.15.0`, `torch` `2.14.1+cu132`, `fla` `0.6.0`). For 4B, seeds `3407` and `3408` were trained concurrently on the same NVIDIA A40 48GB GPU (seed `3408`: best step `448/448`, `best_eval_loss = 0.4569`, mean `train_loss = 0.4033`, runtime `12,825 s`; seed `3407`: best step `448/448`, `best_eval_loss = 0.4579`, mean `train_loss = 0.4042`, runtime `12,887 s`).
+- **Rules Fixed Before Viewing the Results**:
+  1. For model sizes trained with two seeds, the seed with the lower `best_eval_loss` is selected as the release candidate before scoring on RuBench (`4b-v6-s3408` with `0.4569` < `0.4579` for `4b-v6-s3407`).
+  2. Evaluated on the same Pod and the same vLLM `0.32.0` (`--max-model-len 8192 --max-num-seqs 32 --language-model-only`) against the current HF `main` control (`4b-pub-pod`, pinned at `6f3e415a`), `main` weights are replaced if and only if the point estimate of the paired difference on the RuBench v1 `dev` 200 primary metric (criterion status accuracy) exceeds `0` and recall on gold `missing` and gold `contradicted` criteria are each at least as high as the control; otherwise, the retrained weights are published only on the `ruiso-v6` branch. Because this replacement rule does not require the 95% CI lower bound to exceed `0`, any interval spanning `0` is explicitly noted; statistical verdicts for paired differences follow the 95% CI (lower bound > `0` Improved, upper bound < `0` Worse, otherwise No difference).
+#### 1. RuBench v1 `dev` (n = 200 answers · 604 criteria) vs. Same-Pod Control
+| Model ID | Description | HF Revision | Primary Metric [95% CI] | Excl. R2 2/3 | Micro Match | `supported` Recall | `contradicted` Recall | `missing` Recall | `unclear` Recall | Out-of-Rubric TP/FP (53 err) | `answer_status` Match | JSON Errors |
+|---|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
+| `4b-v6-s3408` | RuBio-4B RuIso-v6 (seed 3408, candidate) | `mulkzz/RuBio-4B@553fb15b` | 78.1% `[74.4, 81.9]` | 82.1% | 472/604 | 245/283 (86.6%) | 117/133 (88.0%) | 95/142 (66.9%) | 15/46 (32.6%) | 36/28 | 130/200 | 3 |
+| `4b-pub-pod` | RuBio-4B release (same-Pod control, `main`) | `mulkzz/RuBio-4B@6f3e415a` | 77.0% `[73.2, 80.7]` | 79.7% | 465/604 | 230/283 (81.3%) | **122/133 (91.7%)** | 92/142 (64.8%) | 21/46 (45.7%) | 9/4 | 123/200 | 0 |
+| `4b-v6-s3407` | RuBio-4B RuIso-v6 (seed 3407, non-candidate) | `local rubio-4b-v6-s3407` | 80.0% `[76.4, 83.6]` | 83.5% | 483/604 | 252/283 (89.0%) | 116/133 (87.2%) | 97/142 (68.3%) | 18/46 (39.1%) | 38/31 | 126/200 | 2 |
+- **Answer-clustered paired differences (retrain − same-Pod control `4b-pub-pod`, 95% CI)**:
+  - `4b-v6-s3408 − 4b-pub-pod`: **+1.2%p** `[-2.0, +4.4]` (**No difference**); excluding R2 2/3 majority **+2.5%p** `[-1.1, +6.0]` (**No difference**)
+  - `4b-v6-s3407 − 4b-pub-pod`: **+3.0%p** `[+0.0, +6.0]` (**Improved**); excluding R2 2/3 majority **+3.8%p** `[+0.4, +7.1]` (**Improved**)
+#### 2. Minimal-Pair Style Sensitivity A–E (gold `supported` criteria only; ideal outcome is remaining `supported` after rewrite)
+Reported as directional trends only due to small sample sizes. Each cell lists `original→variant supported count/total criteria (+gained −lost)` and the exact sign-test `p`-value.
+| Minimal-Pair Transformation | Sample Size | `4b-v6-s3408` (candidate) | `4b-pub-pod` (current `main`) | `4b-v6-s3407` (seed 3407) |
+|---|---:|---:|---:|---:|
+| **A. Remove hedging tone** | 36 answers · 74 criteria | 55→61/74 (+6 −0, `p=0.0312`) | 46→58/74 (+15 −3, `p=0.00754`) | 62→62/74 (+1 −1, `p=1`) |
+| **B. Insert hedging tone** | 12 answers · 28 criteria | 25→28/28 (+3 −0, `p=0.25`) | 24→21/28 (+1 −4, `p=0.375`) | 26→26/28 (+0 −0, `p=1`) |
+| **C. Polish into textbook prose** | 12 answers · 27 criteria | 27→23/27 (+0 −4, `p=0.125`) | 23→27/27 (+4 −0, `p=0.125`) | 22→26/27 (+4 −0, `p=0.125`) |
+| **D. Keyword list → relational prose** (vs. A variant) | 10 answers · 25 criteria | 23→24/25 (+2 −1, `p=1`) | 21→25/25 (+4 −0, `p=0.125`) | 24→25/25 (+1 −0, `p=1`) |
+| **E. Relational prose → keyword list** | 12 answers · 26 criteria | 22→25/26 (+4 −1, `p=0.375`) | 25→17/26 (+1 −9, `p=0.0215`) | 25→25/26 (+1 −1, `p=1`) |
+- **Verdict & Summary**:
+  - Release candidate `4b-v6-s3408` showed no statistically significant difference from the same-Pod control `4b-pub-pod` on the primary metric (`+1.2%p [-2.0, +4.4]`), and although it satisfied the positive point-estimate (`+1.2%p > 0`) and `missing` recall (`95/142` ≥ `92/142`) conditions, its `contradicted` recall dropped below the control (`117/133 = 88.0%` < `122/133 = 91.7%`), failing the replacement rule. Consequently, `main` remains `RuBio-4B v2` (`6f3e415a`), and the RuIso-v6 retrain is published only on the `ruiso-v6` branch (`553fb15b`, two `safetensors` shards totaling `9.32 GB`).
+  - Seed `3407` (`4b-v6-s3407`) achieved a higher primary metric than the control (`+3.0%p [+0.0, +6.0]`, or `+3.8%p [+0.4, +7.1]` excluding R2 2/3), but under the rules fixed before viewing the results only `4b-v6-s3408` (selected by lower `eval_loss`: `0.4569` vs. `0.4579`) was eligible as the release candidate, and `4b-v6-s3407` likewise had lower `contradicted` recall (`116/133 = 87.2%` < `122/133 = 91.7%`).
+  - Overall, the retrained model (`4b-v6-s3408`) substantially increased out-of-rubric error recall (TP `9/53` → `36/53`) and raised `supported` recognition (`81.3%` → `86.6%`, including on hedged and keyword-list answers), but also increased false positives on error-free answers (FP `4/147` → `28/147`) and produced `3` malformed JSON outputs (up from `0`) under unconstrained decoding.
+- **How to Load the `ruiso-v6` Branch**:
+  - Transformers / `huggingface_hub`: pass `revision="ruiso-v6"` (or commit hash `revision="553fb15b"`)
+  - vLLM: `vllm serve mulkzz/RuBio-4B --revision ruiso-v6 --served-model-name bio-feedback --language-model-only --max-model-len 8192`
 ## 2. Training Configuration
 - Fine-tuned using Unsloth `bf16` LoRA (`r=16`, `alpha=32`, target modules `q·k·v·o·gate·up·down`) and merged into full 16-bit weights (2 epochs / 300 steps, learning rate `1e-4` with cosine scheduler, 5% linear warmup, max sequence length 4,096, `thinking` disabled, loss masked to assistant JSON output tokens only).
+- The `ruiso-v6` branch (`mulkzz/RuBio-4B@553fb15b`) was retrained on the augmented RuIso-v6 dataset (`train = 3,579`, `dev = 300`) with the same LoRA hyperparameters (2 epochs / 448 steps, judgment token weight `W=1.0`, NVIDIA A40 48GB; see the RuIso-v6 retraining subsection in Section 1).
 ## 3. Input & Output Specification

figures/data/rubench_v1_dev200_all.json ADDED Viewed

	@@ -0,0 +1,2189 @@

+{
+ "benchmark": "RuBench v1",
+ "split": "dev",
+ "n_answers": 200,
+ "n_criteria": 604,
+ "built_at": "2026-11-04T00:15:58",
+ "conditions": "first turn (feedback_mode=hint_1, reveal_requested=false, history=[]), no JSON schema enforcement, invalid JSON counts as wrong. RuBio: SYSTEM_PROMPT only; base and general models: SYSTEM_PROMPT+POLICY. RuBio and Qwen base: temperature 0, max_tokens 1024, thinking off. General models: vendor CLI at the listed reasoning effort, one fixed run.",
+ "panel": {
+  "round1": [
+   "gpt",
+   "opus",
+   "grok"
+  ],
+  "round2": [
+   "gpt",
+   "opus",
+   "gemini"
+  ],
+  "student_answer_author": "gemini"
+ },
+ "caveats": [
+  "GPT and Opus were round-1 gold panelists: the r1-3/3 tier (521 of 604 dev criteria) contains their own blind judgments, so their scores include self-agreement and are upper bounds.",
+  "Round-2 majority (r2-2/3) items include GPT, Opus and Gemini votes; see criterion_acc_excl_r2_2of3.",
+  "Gemini wrote the synthetic student answers with intended labels; reported as reference only (DESIGN 4-1).",
+  "Grok 4.7 (round-1 panelist) was not evaluated as a model.",
+  "RuBio-27B is the RuIso-v6 retrain (main since 2026-11-04), run on a RunPod A100 Pod with vLLM 0.32.0 under the same generation settings; RuBio-4B v2 was run on RunPod Serverless (worker-v1-vllm, vLLM 0.32.0). The decision to replace the 27B main weights used this same dev split."
+ ],
+ "models": {
+  "4b": {
+   "key": "4b",
+   "label": "RuBio-4B v2",
+   "hf": "mulkzz/RuBio-4B@ba9e00c8",
+   "role": "finetuned",
+   "size": "4B",
+   "name": "RuBio-4B v2",
+   "answers": 200,
+   "missing_output": 0,
+   "json_invalid": 0,
+   "criterion_acc": {
+    "mean": 0.7716666666666667,
+    "ci95": [
+     0.7339481076683567,
+     0.8093852256649767
+    ],
+    "n_answers": 200
+   },
+   "criterion_acc_excl_r2_2of3": {
+    "mean": 0.79875,
+    "ci95": [
+     0.759669235815335,
+     0.837830764184665
+    ],
+    "n_answers": 200
+   },
+   "criterion_micro": {
+    "hit": 467,
+    "n": 604
+   },
+   "misconception": {
+    "k": 106,
+    "n": 133,
+    "p": 0.7969924812030075,
+    "ci95": [
+     0.7207474553277116,
+     0.8565630300901761
+    ]
+   },
+   "misconception_with_id": {
+    "k": 65,
+    "n": 79,
+    "p": 0.8227848101265823,
+    "ci95": [
+     0.7242382665030478,
+     0.8913956243248582
+    ]
+   },
+   "json_valid": {
+    "k": 200,
+    "n": 200,
+    "p": 1.0,
+    "ci95": [
+     0.981154748802476,
+     1.0
+    ]
+   },
+   "unit_criterion_micro": {
+    "I": {
+     "k": 156,
+     "n": 206,
+     "p": 0.7572815533980582,
+     "ci95": [
+      0.6943735885025178,
+      0.8107697158073134
+     ]
+    },
+    "II": {
+     "k": 154,
+     "n": 200,
+     "p": 0.77,
+     "ci95": [
+      0.7069170932155301,
+      0.822906471137807
+     ]
+    },
+    "III": {
+     "k": 157,
+     "n": 198,
+     "p": 0.7929292929292929,
+     "ci95": [
+      0.7311760804230768,
+      0.8435324539592243
+     ]
+    }
+   },
+   "other_issues": {
+    "k": 153,
+    "n": 200,
+    "p": 0.765,
+    "ci95": [
+     0.7015864484736013,
+     0.8184255683917109
+    ]
+   },
+   "answer_status": {
+    "k": 124,
+    "n": 200,
+    "p": 0.62,
+    "ci95": [
+     0.5510673223171103,
+     0.684409817395484
+    ]
+   },
+   "gold_tier_criterion_acc": {
+    "r1-3/3": {
+     "k": 431,
+     "n": 521,
+     "p": 0.8272552783109405,
+     "ci95": [
+      0.7924302993782009,
+      0.8572897340091328
+     ]
+    },
+    "r2-3/3": {
+     "k": 10,
+     "n": 23,
+     "p": 0.43478260869565216,
+     "ci95": [
+      0.2563466486442807,
+      0.6318858887246053
+     ]
+    },
+    "r2-2/3": {
+     "k": 23,
+     "n": 51,
+     "p": 0.45098039215686275,
+     "ci95": [
+      0.32267542174059904,
+      0.5861526522832785
+     ]
+    },
+    "human": {
+     "k": 3,
+     "n": 9,
+     "p": 0.3333333333333333,
+     "ci95": [
+      0.12058406396498972,
+      0.6457973346907151
+     ]
+    }
+   },
+   "other_issues_split": {
+    "has_error_recall": {
+     "k": 10,
+     "n": 53,
+     "p": 0.18867924528301888,
+     "ci95": [
+      0.10585337627628057,
+      0.3135843061386674
+     ]
+    },
+    "no_error_specificity": {
+     "k": 143,
+     "n": 147,
+     "p": 0.9727891156462585,
+     "ci95": [
+      0.9321289915585997,
+      0.9893684235398896
+     ]
+    }
+   },
+   "confusion": {
+    "contradicted->contradicted": 121,
+    "contradicted->missing": 5,
+    "contradicted->supported": 3,
+    "contradicted->unclear": 4,
+    "missing->contradicted": 12,
+    "missing->missing": 92,
+    "missing->supported": 18,
+    "missing->unclear": 20,
+    "supported->contradicted": 14,
+    "supported->missing": 9,
+    "supported->supported": 232,
+    "supported->unclear": 28,
+    "unclear->contradicted": 8,
+    "unclear->missing": 7,
+    "unclear->supported": 9,
+    "unclear->unclear": 22
+   },
+   "cli_seconds_p50": null
+  },
+  "qwen4b": {
+   "key": "qwen4b",
+   "label": "Qwen3.5-4B (base)",
+   "hf": "Qwen/Qwen3.5-4B",
+   "role": "base",
+   "size": "4B",
+   "name": "Qwen3.5-4B (기본)",
+   "answers": 200,
+   "missing_output": 0,
+   "json_invalid": 0,
+   "criterion_acc": {
+    "mean": 0.7091666666666667,
+    "ci95": [
+     0.673031876586196,
+     0.7453014567471374
+    ],
+    "n_answers": 200
+   },
+   "criterion_acc_excl_r2_2of3": {
+    "mean": 0.7454166666666667,
+    "ci95": [
+     0.7095199602308042,
+     0.7813133731025292
+    ],
+    "n_answers": 200
+   },
+   "criterion_micro": {
+    "hit": 427,
+    "n": 604
+   },
+   "misconception": {
+    "k": 78,
+    "n": 133,
+    "p": 0.5864661654135338,
+    "ci95": [
+     0.501491500687543,
+     0.666586235573361
+    ]
+   },
+   "misconception_with_id": {
+    "k": 55,
+    "n": 79,
+    "p": 0.6962025316455697,
+    "ci95": [
+     0.5876536225817802,
+     0.7865552130194958
+    ]
+   },
+   "json_valid": {
+    "k": 200,
+    "n": 200,
+    "p": 1.0,
+    "ci95": [
+     0.981154748802476,
+     1.0
+    ]
+   },
+   "unit_criterion_micro": {
+    "I": {
+     "k": 144,
+     "n": 206,
+     "p": 0.6990291262135923,
+     "ci95": [
+      0.633218801872325,
+      0.7575524335371671
+     ]
+    },
+    "II": {
+     "k": 142,
+     "n": 200,
+     "p": 0.71,
+     "ci95": [
+      0.6436253597813291,
+      0.7684596347157107
+     ]
+    },
+    "III": {
+     "k": 141,
+     "n": 198,
+     "p": 0.7121212121212122,
+     "ci95": [
+      0.6454906562919455,
+      0.770677592743514
+     ]
+    }
+   },
+   "other_issues": {
+    "k": 150,
+    "n": 200,
+    "p": 0.75,
+    "ci95": [
+     0.6856591553575013,
+     0.8049182190437367
+    ]
+   },
+   "answer_status": {
+    "k": 89,
+    "n": 200,
+    "p": 0.445,
+    "ci95": [
+     0.37780614202227114,
+     0.5142668356094566
+    ]
+   },
+   "gold_tier_criterion_acc": {
+    "r1-3/3": {
+     "k": 402,
+     "n": 521,
+     "p": 0.7715930902111324,
+     "ci95": [
+      0.7336347625660318,
+      0.8055757050252159
+     ]
+    },
+    "r2-3/3": {
+     "k": 4,
+     "n": 23,
+     "p": 0.17391304347826086,
+     "ci95": [
+      0.06978666088359675,
+      0.37137602596083263
+     ]
+    },
+    "r2-2/3": {
+     "k": 15,
+     "n": 51,
+     "p": 0.29411764705882354,
+     "ci95": [
+      0.1870874425221019,
+      0.4299904683781837
+     ]
+    },
+    "human": {
+     "k": 6,
+     "n": 9,
+     "p": 0.6666666666666666,
+     "ci95": [
+      0.35420266530928485,
+      0.8794159360350102
+     ]
+    }
+   },
+   "other_issues_split": {
+    "has_error_recall": {
+     "k": 4,
+     "n": 53,
+     "p": 0.07547169811320754,
+     "ci95": [
+      0.029739608356547,
+      0.17858450402747283
+     ]
+    },
+    "no_error_specificity": {
+     "k": 146,
+     "n": 147,
+     "p": 0.9931972789115646,
+     "ci95": [
+      0.9624761437110765,
+      0.9987981382261968
+     ]
+    }
+   },
+   "confusion": {
+    "contradicted->contradicted": 100,
+    "contradicted->missing": 20,
+    "contradicted->supported": 10,
+    "contradicted->unclear": 3,
+    "missing->contradicted": 22,
+    "missing->missing": 79,
+    "missing->supported": 37,
+    "missing->unclear": 4,
+    "supported->contradicted": 12,
+    "supported->missing": 16,
+    "supported->supported": 244,
+    "supported->unclear": 11,
+    "unclear->contradicted": 7,
+    "unclear->missing": 9,
+    "unclear->supported": 26,
+    "unclear->unclear": 4
+   },
+   "cli_seconds_p50": null
+  },
+  "27b": {
+   "key": "27b",
+   "src": "27b-v6-s3407",
+   "label": "RuBio-27B",
+   "hf": "mulkzz/RuBio-27B@c1cfefdb",
+   "role": "finetuned",
+   "size": "27B",
+   "name": "RuBio-27B RuIso-v6 (시드 3407)",
+   "answers": 200,
+   "missing_output": 0,
+   "json_invalid": 0,
+   "criterion_acc": {
+    "mean": 0.86875,
+    "ci95": [
+     0.8381287620959553,
+     0.8993712379040447
+    ],
+    "n_answers": 200
+   },
+   "criterion_acc_excl_r2_2of3": {
+    "mean": 0.9,
+    "ci95": [
+     0.8714503967862466,
+     0.9285496032137535
+    ],
+    "n_answers": 200
+   },
+   "criterion_micro": {
+    "hit": 524,
+    "n": 604
+   },
+   "misconception": {
+    "k": 118,
+    "n": 133,
+    "p": 0.8872180451127819,
+    "ci95": [
+     0.8222450524506153,
+     0.9304508968916938
+    ]
+   },
+   "misconception_with_id": {
+    "k": 73,
+    "n": 79,
+    "p": 0.9240506329113924,
+    "ci95": [
+     0.8440460366216044,
+     0.9647278983876056
+    ]
+   },
+   "json_valid": {
+    "k": 200,
+    "n": 200,
+    "p": 1.0,
+    "ci95": [
+     0.981154748802476,
+     1.0
+    ]
+   },
+   "unit_criterion_micro": {
+    "I": {
+     "k": 185,
+     "n": 206,
+     "p": 0.8980582524271845,
+     "ci95": [
+      0.8491894424118119,
+      0.9323530284071725
+     ]
+    },
+    "II": {
+     "k": 166,
+     "n": 200,
+     "p": 0.83,
+     "ci95": [
+      0.7718412936135373,
+      0.8757208405960968
+     ]
+    },
+    "III": {
+     "k": 173,
+     "n": 198,
+     "p": 0.8737373737373737,
+     "ci95": [
+      0.8202540537968194,
+      0.9129947659323233
+     ]
+    }
+   },
+   "other_issues": {
+    "k": 173,
+    "n": 200,
+    "p": 0.865,
+    "ci95": [
+     0.810708740295804,
+     0.9055342263300035
+    ]
+   },
+   "answer_status": {
+    "k": 154,
+    "n": 200,
+    "p": 0.77,
+    "ci95": [
+     0.7069170932155301,
+     0.822906471137807
+    ]
+   },
+   "gold_tier_criterion_acc": {
+    "r1-3/3": {
+     "k": 483,
+     "n": 521,
+     "p": 0.927063339731286,
+     "ci95": [
+      0.9014725811198147,
+      0.9464025357639493
+     ]
+    },
+    "r2-3/3": {
+     "k": 11,
+     "n": 23,
+     "p": 0.4782608695652174,
+     "ci95": [
+      0.29237189265791597,
+      0.670372286465046
+     ]
+    },
+    "r2-2/3": {
+     "k": 23,
+     "n": 51,
+     "p": 0.45098039215686275,
+     "ci95": [
+      0.32267542174059904,
+      0.5861526522832785
+     ]
+    },
+    "human": {
+     "k": 7,
+     "n": 9,
+     "p": 0.7777777777777778,
+     "ci95": [
+      0.45258959147532657,
+      0.9367747440984988
+     ]
+    }
+   },
+   "other_issues_split": {
+    "has_error_recall": {
+     "k": 46,
+     "n": 53,
+     "p": 0.8679245283018868,
+     "ci95": [
+      0.7515977273649117,
+      0.9345213752356045
+     ]
+    },
+    "no_error_specificity": {
+     "k": 127,
+     "n": 147,
+     "p": 0.8639455782312925,
+     "ci95": [
+      0.7991850746700077,
+      0.9101690506216352
+     ]
+    }
+   },
+   "confusion": {
+    "contradicted->contradicted": 127,
+    "contradicted->missing": 4,
+    "contradicted->supported": 1,
+    "contradicted->unclear": 1,
+    "missing->contradicted": 8,
+    "missing->missing": 125,
+    "missing->supported": 2,
+    "missing->unclear": 7,
+    "supported->contradicted": 4,
+    "supported->missing": 20,
+    "supported->supported": 253,
+    "supported->unclear": 6,
+    "unclear->contradicted": 4,
+    "unclear->missing": 15,
+    "unclear->supported": 8,
+    "unclear->unclear": 19
+   },
+   "cli_seconds_p50": null
+  },
+  "qwen27b": {
+   "key": "qwen27b",
+   "label": "Qwen3.8-27B (base)",
+   "hf": "Qwen/Qwen3.8-27B",
+   "role": "base",
+   "size": "27B",
+   "name": "Qwen3.8-27B (기본)",
+   "answers": 200,
+   "missing_output": 0,
+   "json_invalid": 5,
+   "criterion_acc": {
+    "mean": 0.8179166666666667,
+    "ci95": [
+     0.7816876269997497,
+     0.8541457063335838
+    ],
+    "n_answers": 200
+   },
+   "criterion_acc_excl_r2_2of3": {
+    "mean": 0.8495833333333332,
+    "ci95": [
+     0.8142417703596521,
+     0.8849248963070144
+    ],
+    "n_answers": 200
+   },
+   "criterion_micro": {
+    "hit": 495,
+    "n": 604
+   },
+   "misconception": {
+    "k": 107,
+    "n": 133,
+    "p": 0.8045112781954887,
+    "ci95": [
+     0.7289698915278552,
+     0.8629560492170677
+    ]
+   },
+   "misconception_with_id": {
+    "k": 68,
+    "n": 79,
+    "p": 0.8607594936708861,
+    "ci95": [
+     0.7676269042027796,
+     0.9204345031931153
+    ]
+   },
+   "json_valid": {
+    "k": 195,
+    "n": 200,
+    "p": 0.975,
+    "ci95": [
+     0.9428217485145032,
+     0.989275262847849
+    ]
+   },
+   "unit_criterion_micro": {
+    "I": {
+     "k": 165,
+     "n": 206,
+     "p": 0.8009708737864077,
+     "ci95": [
+      0.7411592356541727,
+      0.8497631203309128
+     ]
+    },
+    "II": {
+     "k": 167,
+     "n": 200,
+     "p": 0.835,
+     "ci95": [
+      0.777342313757747,
+      0.8800313679399118
+     ]
+    },
+    "III": {
+     "k": 163,
+     "n": 198,
+     "p": 0.8232323232323232,
+     "ci95": [
+      0.764095715325512,
+      0.8700654260618547
+     ]
+    }
+   },
+   "other_issues": {
+    "k": 156,
+    "n": 200,
+    "p": 0.78,
+    "ci95": [
+     0.7176121367571497,
+     0.8318345225722368
+    ]
+   },
+   "answer_status": {
+    "k": 120,
+    "n": 200,
+    "p": 0.6,
+    "ci95": [
+     0.530836862213719,
+     0.6653940875467761
+    ]
+   },
+   "gold_tier_criterion_acc": {
+    "r1-3/3": {
+     "k": 461,
+     "n": 521,
+     "p": 0.8848368522072937,
+     "ci95": [
+      0.854565276901407,
+      0.9094749969646364
+     ]
+    },
+    "r2-3/3": {
+     "k": 5,
+     "n": 23,
+     "p": 0.21739130434782608,
+     "ci95": [
+      0.09663993783788177,
+      0.4190343907606237
+     ]
+    },
+    "r2-2/3": {
+     "k": 24,
+     "n": 51,
+     "p": 0.47058823529411764,
+     "ci95": [
+      0.3405301441444071,
+      0.6047667002699194
+     ]
+    },
+    "human": {
+     "k": 5,
+     "n": 9,
+     "p": 0.5555555555555556,
+     "ci95": [
+      0.26665173025527866,
+      0.8112211368594864
+     ]
+    }
+   },
+   "other_issues_split": {
+    "has_error_recall": {
+     "k": 29,
+     "n": 53,
+     "p": 0.5471698113207547,
+     "ci95": [
+      0.41454001247278066,
+      0.6734239750401059
+     ]
+    },
+    "no_error_specificity": {
+     "k": 127,
+     "n": 147,
+     "p": 0.8639455782312925,
+     "ci95": [
+      0.7991850746700077,
+      0.9101690506216352
+     ]
+    }
+   },
+   "confusion": {
+    "contradicted->(없음)": 6,
+    "contradicted->contradicted": 112,
+    "contradicted->missing": 7,
+    "contradicted->supported": 5,
+    "contradicted->unclear": 3,
+    "missing->(없음)": 8,
+    "missing->contradicted": 3,
+    "missing->missing": 95,
+    "missing->supported": 20,
+    "missing->unclear": 16,
+    "supported->(없음)": 3,
+    "supported->contradicted": 4,
+    "supported->missing": 1,
+    "supported->supported": 269,
+    "supported->unclear": 6,
+    "unclear->(없음)": 1,
+    "unclear->missing": 3,
+    "unclear->supported": 23,
+    "unclear->unclear": 19
+   },
+   "cli_seconds_p50": 9.7
+  },
+  "gpt": {
+   "key": "gpt",
+   "label": "GPT-6 Astra (xhigh)",
+   "hf": null,
+   "role": "general",
+   "size": null,
+   "name": "GPT-6 Astra (xhigh)",
+   "answers": 200,
+   "missing_output": 0,
+   "json_invalid": 0,
+   "criterion_acc": {
+    "mean": 0.9116666666666667,
+    "ci95": [
+     0.8858637301001623,
+     0.9374696032331712
+    ],
+    "n_answers": 200
+   },
+   "criterion_acc_excl_r2_2of3": {
+    "mean": 0.9558333333333333,
+    "ci95": [
+     0.9362898427686828,
+     0.9753768238979839
+    ],
+    "n_answers": 200
+   },
+   "criterion_micro": {
+    "hit": 550,
+    "n": 604
+   },
+   "misconception": {
+    "k": 129,
+    "n": 133,
+    "p": 0.9699248120300752,
+    "ci95": [
+     0.9252225048958136,
+     0.9882434530438822
+    ]
+   },
+   "misconception_with_id": {
+    "k": 78,
+    "n": 79,
+    "p": 0.9873417721518988,
+    "ci95": [
+     0.9317244628529116,
+     0.9977619997696132
+    ]
+   },
+   "json_valid": {
+    "k": 200,
+    "n": 200,
+    "p": 1.0,
+    "ci95": [
+     0.981154748802476,
+     1.0
+    ]
+   },
+   "unit_criterion_micro": {
+    "I": {
+     "k": 184,
+     "n": 206,
+     "p": 0.8932038834951457,
+     "ci95": [
+      0.8436020225775144,
+      0.928409442499775
+     ]
+    },
+    "II": {
+     "k": 185,
+     "n": 200,
+     "p": 0.925,
+     "ci95": [
+      0.8799564990407998,
+      0.9540250374413048
+     ]
+    },
+    "III": {
+     "k": 181,
+     "n": 198,
+     "p": 0.9141414141414141,
+     "ci95": [
+      0.866814758905608,
+      0.9457042034969555
+     ]
+    }
+   },
+   "other_issues": {
+    "k": 177,
+    "n": 200,
+    "p": 0.885,
+    "ci95": [
+     0.8333529508707886,
+     0.9221362057071179
+    ]
+   },
+   "answer_status": {
+    "k": 176,
+    "n": 200,
+    "p": 0.88,
+    "ci95": [
+     0.8276575999704266,
+     0.9180200091194551
+    ]
+   },
+   "gold_tier_criterion_acc": {
+    "r1-3/3": {
+     "k": 510,
+     "n": 521,
+     "p": 0.9788867562380038,
+     "ci95": [
+      0.9625927664904859,
+      0.9881705668241167
+     ]
+    },
+    "r2-3/3": {
+     "k": 14,
+     "n": 23,
+     "p": 0.6086956521739131,
+     "ci95": [
+      0.4078556031551567,
+      0.7784235012300337
+     ]
+    },
+    "r2-2/3": {
+     "k": 22,
+     "n": 51,
+     "p": 0.43137254901960786,
+     "ci95": [
+      0.30501259121603025,
+      0.5673467124173983
+     ]
+    },
+    "human": {
+     "k": 4,
+     "n": 9,
+     "p": 0.4444444444444444,
+     "ci95": [
+      0.18877886314051356,
+      0.7333482697447213
+     ]
+    }
+   },
+   "other_issues_split": {
+    "has_error_recall": {
+     "k": 33,
+     "n": 53,
+     "p": 0.6226415094339622,
+     "ci95": [
+      0.48806923125215484,
+      0.7406371362813504
+     ]
+    },
+    "no_error_specificity": {
+     "k": 144,
+     "n": 147,
+     "p": 0.9795918367346939,
+     "ci95": [
+      0.9417208980015789,
+      0.9930354727098384
+     ]
+    }
+   },
+   "confusion": {
+    "contradicted->contradicted": 129,
+    "contradicted->missing": 3,
+    "contradicted->unclear": 1,
+    "missing->contradicted": 1,
+    "missing->missing": 135,
+    "missing->supported": 2,
+    "missing->unclear": 4,
+    "supported->contradicted": 3,
+    "supported->missing": 11,
+    "supported->supported": 269,
+    "unclear->contradicted": 2,
+    "unclear->missing": 23,
+    "unclear->supported": 4,
+    "unclear->unclear": 17
+   },
+   "cli_seconds_p50": 575.6
+  },
+  "opus": {
+   "key": "opus",
+   "label": "Claude Opus 5.5 (xhigh)",
+   "hf": null,
+   "role": "general",
+   "size": null,
+   "name": "Claude Opus 5.5 (xhigh)",
+   "answers": 200,
+   "missing_output": 0,
+   "json_invalid": 0,
+   "criterion_acc": {
+    "mean": 0.9420833333333333,
+    "ci95": [
+     0.9203542491963614,
+     0.9638124174703051
+    ],
+    "n_answers": 200
+   },
+   "criterion_acc_excl_r2_2of3": {
+    "mean": 0.9608333333333333,
+    "ci95": [
+     0.9417409178385642,
+     0.9799257488281025
+    ],
+    "n_answers": 200
+   },
+   "criterion_micro": {
+    "hit": 570,
+    "n": 604
+   },
+   "misconception": {
+    "k": 127,
+    "n": 133,
+    "p": 0.9548872180451128,
+    "ci95": [
+     0.9050722874953598,
+     0.9791627597902657
+    ]
+   },
+   "misconception_with_id": {
+    "k": 77,
+    "n": 79,
+    "p": 0.9746835443037974,
+    "ci95": [
+     0.9123143405749654,
+     0.9930296165248962
+    ]
+   },
+   "json_valid": {
+    "k": 200,
+    "n": 200,
+    "p": 1.0,
+    "ci95": [
+     0.981154748802476,
+     1.0
+    ]
+   },
+   "unit_criterion_micro": {
+    "I": {
+     "k": 193,
+     "n": 206,
+     "p": 0.9368932038834952,
+     "ci95": [
+      0.895037892168079,
+      0.9627526245844646
+     ]
+    },
+    "II": {
+     "k": 189,
+     "n": 200,
+     "p": 0.945,
+     "ci95": [
+      0.9042131036406227,
+      0.9690146227935807
+     ]
+    },
+    "III": {
+     "k": 188,
+     "n": 198,
+     "p": 0.9494949494949495,
+     "ci95": [
+      0.9095419269028148,
+      0.9723384103389922
+     ]
+    }
+   },
+   "other_issues": {
+    "k": 195,
+    "n": 200,
+    "p": 0.975,
+    "ci95": [
+     0.9428217485145032,
+     0.989275262847849
+    ]
+   },
+   "answer_status": {
+    "k": 184,
+    "n": 200,
+    "p": 0.92,
+    "ci95": [
+     0.8740106225357106,
+     0.9501593664583693
+    ]
+   },
+   "gold_tier_criterion_acc": {
+    "r1-3/3": {
+     "k": 510,
+     "n": 521,
+     "p": 0.9788867562380038,
+     "ci95": [
+      0.9625927664904859,
+      0.9881705668241167
+     ]
+    },
+    "r2-3/3": {
+     "k": 18,
+     "n": 23,
+     "p": 0.782608695652174,
+     "ci95": [
+      0.5809656092393763,
+      0.9033600621621182
+     ]
+    },
+    "r2-2/3": {
+     "k": 37,
+     "n": 51,
+     "p": 0.7254901960784313,
+     "ci95": [
+      0.5905337813556237,
+      0.8288570781345395
+     ]
+    },
+    "human": {
+     "k": 5,
+     "n": 9,
+     "p": 0.5555555555555556,
+     "ci95": [
+      0.26665173025527866,
+      0.8112211368594864
+     ]
+    }
+   },
+   "other_issues_split": {
+    "has_error_recall": {
+     "k": 52,
+     "n": 53,
+     "p": 0.9811320754716981,
+     "ci95": [
+      0.9005711355345968,
+      0.9966615370968476
+     ]
+    },
+    "no_error_specificity": {
+     "k": 143,
+     "n": 147,
+     "p": 0.9727891156462585,
+     "ci95": [
+      0.9321289915585997,
+      0.9893684235398896
+     ]
+    }
+   },
+   "confusion": {
+    "contradicted->contradicted": 128,
+    "contradicted->missing": 3,
+    "contradicted->supported": 1,
+    "contradicted->unclear": 1,
+    "missing->contradicted": 1,
+    "missing->missing": 123,
+    "missing->supported": 7,
+    "missing->unclear": 11,
+    "supported->missing": 3,
+    "supported->supported": 279,
+    "supported->unclear": 1,
+    "unclear->supported": 6,
+    "unclear->unclear": 40
+   },
+   "cli_seconds_p50": 41.3
+  },
+  "gemini": {
+   "key": "gemini",
+   "label": "Gemini 4 Argon (High)",
+   "hf": null,
+   "role": "reference",
+   "size": null,
+   "name": "Gemini 4 Argon (High)",
+   "answers": 200,
+   "missing_output": 0,
+   "json_invalid": 2,
+   "criterion_acc": {
+    "mean": 0.9125,
+    "ci95": [
+     0.8850333281496484,
+     0.9399666718503515
+    ],
+    "n_answers": 200
+   },
+   "criterion_acc_excl_r2_2of3": {
+    "mean": 0.9358333333333333,
+    "ci95": [
+     0.9098731921940174,
+     0.9617934744726492
+    ],
+    "n_answers": 200
+   },
+   "criterion_micro": {
+    "hit": 551,
+    "n": 604
+   },
+   "misconception": {
+    "k": 122,
+    "n": 133,
+    "p": 0.9172932330827067,
+    "ci95": [
+     0.8579662495837602,
+     0.9531915210666896
+    ]
+   },
+   "misconception_with_id": {
+    "k": 75,
+    "n": 79,
+    "p": 0.9493670886075949,
+    "ci95": [
+     0.8769230243999817,
+     0.980135921654554
+    ]
+   },
+   "json_valid": {
+    "k": 198,
+    "n": 200,
+    "p": 0.99,
+    "ci95": [
+     0.9642783183248235,
+     0.997253335501603
+    ]
+   },
+   "unit_criterion_micro": {
+    "I": {
+     "k": 187,
+     "n": 206,
+     "p": 0.9077669902912622,
+     "ci95": [
+      0.8604469235517486,
+      0.9401575587506256
+     ]
+    },
+    "II": {
+     "k": 184,
+     "n": 200,
+     "p": 0.92,
+     "ci95": [
+      0.8740106225357106,
+      0.9501593664583693
+     ]
+    },
+    "III": {
+     "k": 180,
+     "n": 198,
+     "p": 0.9090909090909091,
+     "ci95": [
+      0.8608883910901738,
+      0.9417218034782121
+     ]
+    }
+   },
+   "other_issues": {
+    "k": 190,
+    "n": 200,
+    "p": 0.95,
+    "ci95": [
+     0.9104219523342212,
+     0.9726173215880072
+    ]
+   },
+   "answer_status": {
+    "k": 179,
+    "n": 200,
+    "p": 0.895,
+    "ci95": [
+     0.8448197976153835,
+     0.9302924539385726
+    ]
+   },
+   "gold_tier_criterion_acc": {
+    "r1-3/3": {
+     "k": 496,
+     "n": 521,
+     "p": 0.9520153550863724,
+     "ci95": [
+      0.9301245809465594,
+      0.9672893068113121
+     ]
+    },
+    "r2-3/3": {
+     "k": 19,
+     "n": 23,
+     "p": 0.8260869565217391,
+     "ci95": [
+      0.6286239740391674,
+      0.9302133391164032
+     ]
+    },
+    "r2-2/3": {
+     "k": 32,
+     "n": 51,
+     "p": 0.6274509803921569,
+     "ci95": [
+      0.49025241212438464,
+      0.7467945954135337
+     ]
+    },
+    "human": {
+     "k": 4,
+     "n": 9,
+     "p": 0.4444444444444444,
+     "ci95": [
+      0.18877886314051356,
+      0.7333482697447213
+     ]
+    }
+   },
+   "other_issues_split": {
+    "has_error_recall": {
+     "k": 45,
+     "n": 53,
+     "p": 0.8490566037735849,
+     "ci95": [
+      0.7294577896504801,
+      0.9214757179448814
+     ]
+    },
+    "no_error_specificity": {
+     "k": 145,
+     "n": 147,
+     "p": 0.9863945578231292,
+     "ci95": [
+      0.9517544022691328,
+      0.9962609240552124
+     ]
+    }
+   },
+   "confusion": {
+    "contradicted->contradicted": 125,
+    "contradicted->missing": 5,
+    "contradicted->supported": 1,
+    "contradicted->unclear": 2,
+    "missing->(없음)": 2,
+    "missing->missing": 125,
+    "missing->supported": 10,
+    "missing->unclear": 5,
+    "supported->(없음)": 3,
+    "supported->contradicted": 2,
+    "supported->missing": 6,
+    "supported->supported": 267,
+    "supported->unclear": 5,
+    "unclear->(없음)": 1,
+    "unclear->contradicted": 1,
+    "unclear->missing": 9,
+    "unclear->supported": 1,
+    "unclear->unclear": 34
+   },
+   "cli_seconds_p50": 87.9
+  }
+ },
+ "paired": {
+  "qwen4b-minus-4b": {
+   "all": {
+    "diff": -0.06250000000000001,
+    "ci95": [
+     -0.09929215430834135,
+     -0.02570784569165868
+    ],
+    "n": 200,
+    "verdict": "악화"
+   },
+   "no_majority": {
+    "diff": -0.05333333333333334,
+    "ci95": [
+     -0.0949195943790324,
+     -0.011747072287634275
+    ],
+    "n": 200,
+    "verdict": "악화"
+   }
+  },
+  "27b-minus-4b": {
+   "all": {
+    "diff": 0.09708333333333334,
+    "ci95": [
+     0.06123184381518337,
+     0.1329348228514833
+    ],
+    "n": 200,
+    "verdict": "개선"
+   },
+   "no_majority": {
+    "diff": 0.10125,
+    "ci95": [
+     0.06521017094468781,
+     0.1372898290553122
+    ],
+    "n": 200,
+    "verdict": "개선"
+   }
+  },
+  "qwen27b-minus-4b": {
+   "all": {
+    "diff": 0.04625,
+    "ci95": [
+     0.007906503441078273,
+     0.08459349655892173
+    ],
+    "n": 200,
+    "verdict": "개선"
+   },
+   "no_majority": {
+    "diff": 0.05083333333333334,
+    "ci95": [
+     0.008389559707535299,
+     0.09327710695913138
+    ],
+    "n": 200,
+    "verdict": "개선"
+   }
+  },
+  "gpt-minus-4b": {
+   "all": {
+    "diff": 0.14,
+    "ci95": [
+     0.1028811816212637,
+     0.17711881837873633
+    ],
+    "n": 200,
+    "verdict": "개선"
+   },
+   "no_majority": {
+    "diff": 0.15708333333333335,
+    "ci95": [
+     0.11841813107448726,
+     0.19574853559217945
+    ],
+    "n": 200,
+    "verdict": "개선"
+   }
+  },
+  "opus-minus-4b": {
+   "all": {
+    "diff": 0.1704166666666667,
+    "ci95": [
+     0.1349501475307221,
+     0.20588318580261128
+    ],
+    "n": 200,
+    "verdict": "개선"
+   },
+   "no_majority": {
+    "diff": 0.16208333333333336,
+    "ci95": [
+     0.12777146352786023,
+     0.19639520313880648
+    ],
+    "n": 200,
+    "verdict": "개선"
+   }
+  },
+  "gemini-minus-4b": {
+   "all": {
+    "diff": 0.14083333333333334,
+    "ci95": [
+     0.10129028178518129,
+     0.18037638488148539
+    ],
+    "n": 200,
+    "verdict": "개선"
+   },
+   "no_majority": {
+    "diff": 0.13708333333333333,
+    "ci95": [
+     0.0986490412756628,
+     0.17551762539100385
+    ],
+    "n": 200,
+    "verdict": "개선"
+   }
+  },
+  "4b-minus-qwen4b": {
+   "all": {
+    "diff": 0.06250000000000001,
+    "ci95": [
+     0.02570784569165868,
+     0.09929215430834135
+    ],
+    "n": 200,
+    "verdict": "개선"
+   },
+   "no_majority": {
+    "diff": 0.05333333333333334,
+    "ci95": [
+     0.011747072287634275,
+     0.0949195943790324
+    ],
+    "n": 200,
+    "verdict": "개선"
+   }
+  },
+  "27b-minus-qwen4b": {
+   "all": {
+    "diff": 0.15958333333333333,
+    "ci95": [
+     0.12437326122557398,
+     0.1947934054410927
+    ],
+    "n": 200,
+    "verdict": "개선"
+   },
+   "no_majority": {
+    "diff": 0.15458333333333335,
+    "ci95": [
+     0.11312043866157052,
+     0.1960462280050962
+    ],
+    "n": 200,
+    "verdict": "개선"
+   }
+  },
+  "qwen27b-minus-qwen4b": {
+   "all": {
+    "diff": 0.10875,
+    "ci95": [
+     0.06948238748995589,
+     0.1480176125100441
+    ],
+    "n": 200,
+    "verdict": "개선"
+   },
+   "no_majority": {
+    "diff": 0.10416666666666669,
+    "ci95": [
+     0.06221595810731039,
+     0.14611737522602297
+    ],
+    "n": 200,
+    "verdict": "개선"
+   }
+  },
+  "gpt-minus-qwen4b": {
+   "all": {
+    "diff": 0.2025,
+    "ci95": [
+     0.16509784766297997,
+     0.23990215233702006
+    ],
+    "n": 200,
+    "verdict": "개선"
+   },
+   "no_majority": {
+    "diff": 0.21041666666666667,
+    "ci95": [
+     0.1727924563143645,
+     0.24804087701896885
+    ],
+    "n": 200,
+    "verdict": "개선"
+   }
+  },
+  "opus-minus-qwen4b": {
+   "all": {
+    "diff": 0.2329166666666667,
+    "ci95": [
+     0.1962971271528449,
+     0.26953620618048846
+    ],
+    "n": 200,
+    "verdict": "개선"
+   },
+   "no_majority": {
+    "diff": 0.21541666666666667,
+    "ci95": [
+     0.17708051297608593,
+     0.25375282035724744
+    ],
+    "n": 200,
+    "verdict": "개선"
+   }
+  },
+  "gemini-minus-qwen4b": {
+   "all": {
+    "diff": 0.20333333333333337,
+    "ci95": [
+     0.16215851002079762,
+     0.2445081566458691
+    ],
+    "n": 200,
+    "verdict": "개선"
+   },
+   "no_majority": {
+    "diff": 0.19041666666666668,
+    "ci95": [
+     0.14755506490751477,
+     0.23327826842581859
+    ],
+    "n": 200,
+    "verdict": "개선"
+   }
+  },
+  "4b-minus-27b": {
+   "all": {
+    "diff": -0.09708333333333334,
+    "ci95": [
+     -0.1329348228514833,
+     -0.06123184381518337
+    ],
+    "n": 200,
+    "verdict": "악화"
+   },
+   "no_majority": {
+    "diff": -0.10125,
+    "ci95": [
+     -0.1372898290553122,
+     -0.06521017094468781
+    ],
+    "n": 200,
+    "verdict": "악화"
+   }
+  },
+  "qwen4b-minus-27b": {
+   "all": {
+    "diff": -0.15958333333333333,
+    "ci95": [
+     -0.1947934054410927,
+     -0.12437326122557398
+    ],
+    "n": 200,
+    "verdict": "악화"
+   },
+   "no_majority": {
+    "diff": -0.15458333333333335,
+    "ci95": [
+     -0.1960462280050962,
+     -0.11312043866157052
+    ],
+    "n": 200,
+    "verdict": "악화"
+   }
+  },
+  "qwen27b-minus-27b": {
+   "all": {
+    "diff": -0.05083333333333334,
+    "ci95": [
+     -0.09092183669301501,
+     -0.010744829973651678
+    ],
+    "n": 200,
+    "verdict": "악화"
+   },
+   "no_majority": {
+    "diff": -0.05041666666666667,
+    "ci95": [
+     -0.0903058270121205,
+     -0.01052750632121284
+    ],
+    "n": 200,
+    "verdict": "악화"
+   }
+  },
+  "gpt-minus-27b": {
+   "all": {
+    "diff": 0.04291666666666667,
+    "ci95": [
+     0.01078839936599886,
+     0.07504493396733448
+    ],
+    "n": 200,
+    "verdict": "개선"
+   },
+   "no_majority": {
+    "diff": 0.05583333333333334,
+    "ci95": [
+     0.02437694687489072,
+     0.08728971979177597
+    ],
+    "n": 200,
+    "verdict": "개선"
+   }
+  },
+  "opus-minus-27b": {
+   "all": {
+    "diff": 0.07333333333333333,
+    "ci95": [
+     0.04152834968415308,
+     0.10513831698251358
+    ],
+    "n": 200,
+    "verdict": "개선"
+   },
+   "no_majority": {
+    "diff": 0.060833333333333336,
+    "ci95": [
+     0.030981954682252844,
+     0.09068471198441383
+    ],
+    "n": 200,
+    "verdict": "개선"
+   }
+  },
+  "gemini-minus-27b": {
+   "all": {
+    "diff": 0.04375,
+    "ci95": [
+     0.009717845060980758,
+     0.07778215493901924
+    ],
+    "n": 200,
+    "verdict": "개선"
+   },
+   "no_majority": {
+    "diff": 0.035833333333333335,
+    "ci95": [
+     0.001730887172231424,
+     0.06993577949443525
+    ],
+    "n": 200,
+    "verdict": "개선"
+   }
+  },
+  "4b-minus-qwen27b": {
+   "all": {
+    "diff": -0.04625,
+    "ci95": [
+     -0.08459349655892173,
+     -0.007906503441078273
+    ],
+    "n": 200,
+    "verdict": "악화"
+   },
+   "no_majority": {
+    "diff": -0.05083333333333334,
+    "ci95": [
+     -0.09327710695913138,
+     -0.008389559707535299
+    ],
+    "n": 200,
+    "verdict": "악화"
+   }
+  },
+  "qwen4b-minus-qwen27b": {
+   "all": {
+    "diff": -0.10875,
+    "ci95": [
+     -0.1480176125100441,
+     -0.06948238748995589
+    ],
+    "n": 200,
+    "verdict": "악화"
+   },
+   "no_majority": {
+    "diff": -0.10416666666666669,
+    "ci95": [
+     -0.14611737522602297,
+     -0.06221595810731039
+    ],
+    "n": 200,
+    "verdict": "악화"
+   }
+  },
+  "27b-minus-qwen27b": {
+   "all": {
+    "diff": 0.05083333333333334,
+    "ci95": [
+     0.010744829973651678,
+     0.09092183669301501
+    ],
+    "n": 200,
+    "verdict": "개선"
+   },
+   "no_majority": {
+    "diff": 0.05041666666666667,
+    "ci95": [
+     0.01052750632121284,
+     0.0903058270121205
+    ],
+    "n": 200,
+    "verdict": "개선"
+   }
+  },
+  "gpt-minus-qwen27b": {
+   "all": {
+    "diff": 0.09375,
+    "ci95": [
+     0.05730219163645278,
+     0.13019780836354722
+    ],
+    "n": 200,
+    "verdict": "개선"
+   },
+   "no_majority": {
+    "diff": 0.10625,
+    "ci95": [
+     0.07279390330302032,
+     0.13970609669697967
+    ],
+    "n": 200,
+    "verdict": "개선"
+   }
+  },
+  "opus-minus-qwen27b": {
+   "all": {
+    "diff": 0.12416666666666668,
+    "ci95": [
+     0.09197561056896073,
+     0.15635772276437263
+    ],
+    "n": 200,
+    "verdict": "개선"
+   },
+   "no_majority": {
+    "diff": 0.11125,
+    "ci95": [
+     0.07880409709964073,
+     0.14369590290035927
+    ],
+    "n": 200,
+    "verdict": "개선"
+   }
+  },
+  "gemini-minus-qwen27b": {
+   "all": {
+    "diff": 0.09458333333333334,
+    "ci95": [
+     0.0551882445624855,
+     0.13397842210418118
+    ],
+    "n": 200,
+    "verdict": "개선"
+   },
+   "no_majority": {
+    "diff": 0.08625,
+    "ci95": [
+     0.04819286835030037,
+     0.12430713164969961
+    ],
+    "n": 200,
+    "verdict": "개선"
+   }
+  },
+  "4b-minus-gpt": {
+   "all": {
+    "diff": -0.14,
+    "ci95": [
+     -0.17711881837873633,
+     -0.1028811816212637
+    ],
+    "n": 200,
+    "verdict": "악화"
+   },
+   "no_majority": {
+    "diff": -0.15708333333333335,
+    "ci95": [
+     -0.19574853559217945,
+     -0.11841813107448726
+    ],
+    "n": 200,
+    "verdict": "악화"
+   }
+  },
+  "qwen4b-minus-gpt": {
+   "all": {
+    "diff": -0.2025,
+    "ci95": [
+     -0.23990215233702006,
+     -0.16509784766297997
+    ],
+    "n": 200,
+    "verdict": "악화"
+   },
+   "no_majority": {
+    "diff": -0.21041666666666667,
+    "ci95": [
+     -0.24804087701896885,
+     -0.1727924563143645
+    ],
+    "n": 200,
+    "verdict": "악화"
+   }
+  },
+  "27b-minus-gpt": {
+   "all": {
+    "diff": -0.04291666666666667,
+    "ci95": [
+     -0.07504493396733448,
+     -0.01078839936599886
+    ],
+    "n": 200,
+    "verdict": "악화"
+   },
+   "no_majority": {
+    "diff": -0.05583333333333334,
+    "ci95": [
+     -0.08728971979177597,
+     -0.02437694687489072
+    ],
+    "n": 200,
+    "verdict": "악화"
+   }
+  },
+  "qwen27b-minus-gpt": {
+   "all": {
+    "diff": -0.09375,
+    "ci95": [
+     -0.13019780836354722,
+     -0.05730219163645278
+    ],
+    "n": 200,
+    "verdict": "악화"
+   },
+   "no_majority": {
+    "diff": -0.10625,
+    "ci95": [
+     -0.13970609669697967,
+     -0.07279390330302032
+    ],
+    "n": 200,
+    "verdict": "악화"
+   }
+  },
+  "opus-minus-gpt": {
+   "all": {
+    "diff": 0.030416666666666668,
+    "ci95": [
+     0.004444457390938891,
+     0.056388875942394445
+    ],
+    "n": 200,
+    "verdict": "개선"
+   },
+   "no_majority": {
+    "diff": 0.005000000000000001,
+    "ci95": [
+     -0.015406897160101511,
+     0.025406897160101513
+    ],
+    "n": 200,
+    "verdict": "차이 없음"
+   }
+  },
+  "gemini-minus-gpt": {
+   "all": {
+    "diff": 0.0008333333333333328,
+    "ci95": [
+     -0.029734282082786483,
+     0.03140094874945315
+    ],
+    "n": 200,
+    "verdict": "차이 없음"
+   },
+   "no_majority": {
+    "diff": -0.02,
+    "ci95": [
+     -0.048200934523245254,
+     0.008200934523245256
+    ],
+    "n": 200,
+    "verdict": "차이 없음"
+   }
+  },
+  "4b-minus-opus": {
+   "all": {
+    "diff": -0.1704166666666667,
+    "ci95": [
+     -0.20588318580261128,
+     -0.1349501475307221
+    ],
+    "n": 200,
+    "verdict": "악화"
+   },
+   "no_majority": {
+    "diff": -0.16208333333333336,
+    "ci95": [
+     -0.19639520313880648,
+     -0.12777146352786023
+    ],
+    "n": 200,
+    "verdict": "악화"
+   }
+  },
+  "qwen4b-minus-opus": {
+   "all": {
+    "diff": -0.2329166666666667,
+    "ci95": [
+     -0.26953620618048846,
+     -0.1962971271528449
+    ],
+    "n": 200,
+    "verdict": "악화"
+   },
+   "no_majority": {
+    "diff": -0.21541666666666667,
+    "ci95": [
+     -0.25375282035724744,
+     -0.17708051297608593
+    ],
+    "n": 200,
+    "verdict": "악화"
+   }
+  },
+  "27b-minus-opus": {
+   "all": {
+    "diff": -0.07333333333333333,
+    "ci95": [
+     -0.10513831698251358,
+     -0.04152834968415308
+    ],
+    "n": 200,
+    "verdict": "악화"
+   },
+   "no_majority": {
+    "diff": -0.060833333333333336,
+    "ci95": [
+     -0.09068471198441383,
+     -0.030981954682252844
+    ],
+    "n": 200,
+    "verdict": "악화"
+   }
+  },
+  "qwen27b-minus-opus": {
+   "all": {
+    "diff": -0.12416666666666668,
+    "ci95": [
+     -0.15635772276437263,
+     -0.09197561056896073
+    ],
+    "n": 200,
+    "verdict": "악화"
+   },
+   "no_majority": {
+    "diff": -0.11125,
+    "ci95": [
+     -0.14369590290035927,
+     -0.07880409709964073
+    ],
+    "n": 200,
+    "verdict": "악화"
+   }
+  },
+  "gpt-minus-opus": {
+   "all": {
+    "diff": -0.030416666666666668,
+    "ci95": [
+     -0.056388875942394445,
+     -0.004444457390938891
+    ],
+    "n": 200,
+    "verdict": "악화"
+   },
+   "no_majority": {
+    "diff": -0.005000000000000001,
+    "ci95": [
+     -0.025406897160101513,
+     0.015406897160101511
+    ],
+    "n": 200,
+    "verdict": "차이 없음"
+   }
+  },
+  "gemini-minus-opus": {
+   "all": {
+    "diff": -0.029583333333333336,
+    "ci95": [
+     -0.056084893999743986,
+     -0.003081772666922687
+    ],
+    "n": 200,
+    "verdict": "악화"
+   },
+   "no_majority": {
+    "diff": -0.025,
+    "ci95": [
+     -0.04693792318061361,
+     -0.003062076819386396
+    ],
+    "n": 200,
+    "verdict": "악화"
+   }
+  },
+  "4b-minus-gemini": {
+   "all": {
+    "diff": -0.14083333333333334,
+    "ci95": [
+     -0.18037638488148539,
+     -0.10129028178518129
+    ],
+    "n": 200,
+    "verdict": "악화"
+   },
+   "no_majority": {
+    "diff": -0.13708333333333333,
+    "ci95": [
+     -0.17551762539100385,
+     -0.0986490412756628
+    ],
+    "n": 200,
+    "verdict": "악화"
+   }
+  },
+  "qwen4b-minus-gemini": {
+   "all": {
+    "diff": -0.20333333333333337,
+    "ci95": [
+     -0.2445081566458691,
+     -0.16215851002079762
+    ],
+    "n": 200,
+    "verdict": "악화"
+   },
+   "no_majority": {
+    "diff": -0.19041666666666668,
+    "ci95": [
+     -0.23327826842581859,
+     -0.14755506490751477
+    ],
+    "n": 200,
+    "verdict": "악화"
+   }
+  },
+  "27b-minus-gemini": {
+   "all": {
+    "diff": -0.04375,
+    "ci95": [
+     -0.07778215493901924,
+     -0.009717845060980758
+    ],
+    "n": 200,
+    "verdict": "악화"
+   },
+   "no_majority": {
+    "diff": -0.035833333333333335,
+    "ci95": [
+     -0.06993577949443525,
+     -0.001730887172231424
+    ],
+    "n": 200,
+    "verdict": "악화"
+   }
+  },
+  "qwen27b-minus-gemini": {
+   "all": {
+    "diff": -0.09458333333333334,
+    "ci95": [
+     -0.13397842210418118,
+     -0.0551882445624855
+    ],
+    "n": 200,
+    "verdict": "악화"
+   },
+   "no_majority": {
+    "diff": -0.08625,
+    "ci95": [
+     -0.12430713164969961,
+     -0.04819286835030037
+    ],
+    "n": 200,
+    "verdict": "악화"
+   }
+  },
+  "gpt-minus-gemini": {
+   "all": {
+    "diff": -0.0008333333333333328,
+    "ci95": [
+     -0.03140094874945315,
+     0.029734282082786483
+    ],
+    "n": 200,
+    "verdict": "차이 없음"
+   },
+   "no_majority": {
+    "diff": 0.02,
+    "ci95": [
+     -0.008200934523245256,
+     0.048200934523245254
+    ],
+    "n": 200,
+    "verdict": "차이 없음"
+   }
+  },
+  "opus-minus-gemini": {
+   "all": {
+    "diff": 0.029583333333333336,
+    "ci95": [
+     0.003081772666922687,
+     0.056084893999743986
+    ],
+    "n": 200,
+    "verdict": "개선"
+   },
+   "no_majority": {
+    "diff": 0.025,
+    "ci95": [
+     0.003062076819386396,
+     0.04693792318061361
+    ],
+    "n": 200,
+    "verdict": "개선"
+   }
+  }
+ },
+ "supplementary": {
+  "qwen27b_system_prompt_only": {
+   "label": "Qwen3.8-27B (base, SYSTEM_PROMPT only)",
+   "criterion_acc": {
+    "mean": 0.8433333333333333,
+    "ci95": [
+     0.8119954335270959,
+     0.8746712331395706
+    ],
+    "n_answers": 200
+   },
+   "criterion_acc_excl_r2_2of3": {
+    "mean": 0.8729166666666667,
+    "ci95": [
+     0.8426809286496472,
+     0.9031524046836862
+    ],
+    "n_answers": 200
+   },
+   "json_invalid": 0,
+   "paired_27b_minus_this": {
+    "all": {
+     "diff": 0.02541666666666667,
+     "ci95": [
+      -0.009341613239777467,
+      0.06017494657311081
+     ],
+     "n": 200,
+     "verdict": "차이 없음"
+    },
+    "no_majority": {
+     "diff": 0.027083333333333334,
+     "ci95": [
+      -0.008110744018988085,
+      0.062277410685654754
+     ],
+     "n": 200,
+     "verdict": "차이 없음"
+    }
+   }
+  }
+ }
+}

figures/rubench_v1_dev200_all.png CHANGED Viewed

Git LFS Details

SHA256: fcb1b2b973ec930134bac04de65a542e3934e0f23da191e73b23abeba64de84e
Pointer size: 131 Bytes
Size of remote file: 969 kB

Git LFS Details

SHA256: 5cb3a51eec33861415e1eb1a11de982a4cb28709c06cbef229a2346d91271b16
Pointer size: 131 Bytes
Size of remote file: 965 kB

figures/rubench_v1_dev200_all.svg CHANGED Viewed