main 가중치를 RuIso-v6 재학습본(ruiso-v6@f61ac9cb, 시드 3408)으로 교체하고 카드·그림 갱신(Gemini 4 Argon High 작성)
Browse files- README.md +118 -91
- README_en.md +118 -91
- figures/data/rubench_v1_dev200_9b.json +89 -87
- figures/data/rubench_v1_dev200_all_9b.json +360 -355
- figures/rubench_v1_dev200_9b.png +2 -2
- figures/rubench_v1_dev200_9b.svg +0 -0
- figures/rubench_v1_dev200_all_9b.png +2 -2
- figures/rubench_v1_dev200_all_9b.svg +0 -0
- figures/rubio_9b_header.svg +4 -4
- figures/rubio_9b_nature_fig1.png +2 -2
- figures/rubio_9b_nature_fig1.svg +0 -0
- model.safetensors-00001-of-00004.safetensors +1 -1
- model.safetensors-00002-of-00004.safetensors +1 -1
- model.safetensors-00003-of-00004.safetensors +1 -1
- model.safetensors-00004-of-00004.safetensors +1 -1
README.md
CHANGED
|
@@ -21,70 +21,95 @@ tags:
|
|
| 21 |
|
| 22 |
## 1. 모델 요약
|
| 23 |
|
| 24 |
-
`mulkzz/RuBio-9B`는 고등학교 2022 개정 『생명과학』 대단원 Ⅰ–Ⅲ 서술형 문항의 학생 답안을 평가 기준별로 진단하고, 교사용 진단 정보와 학생용 피드백을 단일 JSON 객체로 출력하는 모델입니다. [Qwen/Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B)에 LoRA를 학습한 뒤 가중치를 병합한 **16비트(`bf16`) 전체 모델**(LoRA 어댑터가 아닌 병합 가중치, `safetensors` 4개 약 `19.3GB` / `18GiB`
|
| 25 |
|
| 26 |
-
|
|
|
|
|
|
|
| 27 |
|
| 28 |
- **시범 서비스**: https://mulk.codes/rubio (`RuBio-4B` · `RuBio-27B`만 운영하며, `RuBio-9B`는 올리지 않음)
|
| 29 |
-
- **
|
| 30 |
-
- **상위 모델**: [mulkzz/RuBio-27B](https://huggingface.co/mulkzz/RuBio-27B)
|
| 31 |
|
| 32 |
### RuBench v1 벤치마크 성능 요약 (Out-of-Distribution Dev, n = 200)
|
| 33 |
|
| 34 |
<img src="./figures/rubench_v1_dev200_9b.svg" width="100%" style="width: 100%; height: auto;" alt="Fig. 1 | RuBench v1 (dev n = 200) diagnostic evaluation of RuBio-9B vs. its base model Qwen3.5-9B" />
|
| 35 |
|
| 36 |
-
> **Fig. 1 | RuBench v1 (dev 분할, n = 200 답안 · 604 평가 기준)에서 RuBio-9B(`mulkzz/RuBio-9B@
|
| 37 |
|
| 38 |
<img src="./figures/rubench_v1_dev200_all_9b.svg" width="100%" style="width: 100%; height: auto;" alt="Fig. 2 | RuBench v1 (dev n = 200) criterion accuracy of RuBio, base Qwen models, and general-purpose LLMs" />
|
| 39 |
|
| 40 |
-
> **Fig. 2 | RuBench v1 (dev 분할, n = 200 답안 · 604 평가 기준)에서 RuBio-9B를 포함한 9개 모델(미세조정 RuBio 3종, 기본 Qwen 3종, 범용 프론티어 모델 2종, 참고 모델 1종)의 진단 성능 비
|
| 41 |
|
| 42 |
-
**RuBench v1**은 새 문항·새 상황·가상 학생 페르소나로 구축한 2022 개정 『생명과학』 서술형 진단 벤치마크입니다(19개 성취기준, 대단원 Ⅰ–Ⅲ 298문항 중 `dev = 200` / `test = 98`, 가상 학생 30명 × 성취수준 A–E 층화 배정; `test` 분할은 열지 않았습니다). 학습 문항과의 5-gram 중복(`< 0.40`) 및 장면·사례 중복을 차단하고, 모범답안·허용 표현을 가린 채 생성한 학생 답안을 3개 프론티어 모델(`GPT-6 Astra xhigh` · `Opus 5.5 xhigh` · `Grok 4.7 xhigh`) 블라인드 채점, 2차 블라인드 재채점, 인간 검수로 확정했습니다. 모든 모델은
|
| 43 |
|
| 44 |
-
| 구분 | 평가 지표 (RuBench v1 `dev`) | 표본 수 | RuBio-9B (`@
|
| 45 |
|---|---|---:|---:|---:|---|
|
| 46 |
-
| **주 지표** | **기준 판정 정확도** (답안 단위 평균, 95% 군집 CI)<sup>†</sup> | 200 답안 (604 기준) | **
|
| 47 |
-
| | **기준 판정 정확도** (2차 2/3 다수결 항목 제외)<sup>‡</sup> | 200 답안 (553 기준) | **
|
| 48 |
-
| | **기준 판정 미시 일치율** (`checks` 전체 합산) | 604 기준 | **
|
| 49 |
-
| **보조 지표** | **오개념 진단 일치** (정답 `contradicted` + ID 일치) | 133 기준 | **
|
| 50 |
-
| | **기준 밖 오류(`other_issues`) 유무 일치** | 200 답안 |
|
| 51 |
-
| | **종합 판정(`answer_status`) 규칙 일치** | 200 답안 | **
|
| 52 |
-
| **정답 출처별** | 1차 3/3 만장일치 확정 기준 (`r1-3/3`) | 521 기준 | **
|
| 53 |
-
| | 2차 3/3 만장일치 확정 기준 (`r2-3/3`) | 23 기준 | **
|
| 54 |
-
| | 2차 2/3 다수결 확정 기준 (`r2-2/3`, 경계 사례) | 51 기준 | **
|
| 55 |
| | 사용자 직접 판정 기준 (`human`) | 9 기준 | **6/9 (66.7%)** | 3/9 (33.3%) | +33.3%p (3개 모델 모두 갈린 고난도 항목) |
|
| 56 |
-
| **대단원별** | Ⅰ. 생명과학의 이해 (`checks` 미시 일치) | 206 기준 | **
|
| 57 |
-
| | Ⅱ. 사람의 물질대사 (`checks` 미시 일치) | 200 기준 | **
|
| 58 |
-
| | Ⅲ. 항상성과 몸의 조절 (`checks` 미시 일치) | 198 기준 | **
|
| 59 |
-
| **출력 안정성** | **JSON 유효 응답률** (스키마 강제 없음) | 200 답안 |
|
| 60 |
|
| 61 |
<sup>†</sup> **벤치마크 설계 때 정한 판정 규칙**: 같은 답안에서의 쌍대 차이(paired difference) 95% 군집 신뢰구간 하한이 `0`보다 크면 **개선**, 상한이 `0`보다 작으면 **악화**, `0`을 포함하면 **차이 없음**으로 판정합니다(Miller 2024, arXiv:2411.00640).<br/>
|
| 62 |
<sup>‡</sup> 2차 채점의 2/3 다수결 확정분(`r2-2/3`, 51개 기준)을 제외하고 만장일치·사용자 판정 항목만으로 계산한 민감도 분석입니다.<br/>
|
| 63 |
-
<sup>§</sup> **평가 조건 및 서빙 경로**:
|
| 64 |
-
|
| 65 |
-
###
|
| 66 |
-
|
| 67 |
-
- **
|
| 68 |
-
- **
|
| 69 |
-
|
| 70 |
-
|
| 71 |
-
|
| 72 |
-
|
| 73 |
-
|
| 74 |
-
|
| 75 |
-
|
| 76 |
-
|
| 77 |
-
-
|
| 78 |
-
|
| 79 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 80 |
|
| 81 |
## 2. 학습 설정
|
| 82 |
|
| 83 |
-
- **베이스 모델 및 템플릿 검증**: 기본 모델은 [Qwen/Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B)(Apache-2.0)입니다. 실제 학습에는 `unsloth/Qwen3.5-9B`를 불러왔으며 가중치 `sha256`
|
| 84 |
-
- **
|
| 85 |
-
- **
|
| 86 |
-
- **
|
| 87 |
-
- **
|
|
|
|
|
|
|
|
|
|
|
|
|
| 88 |
|
| 89 |
## 3. 입력 및 출력 규격
|
| 90 |
|
|
@@ -136,7 +161,7 @@ user 메시지는 JSON이다: 문항(question)·자료(materials)·평가 기준
|
|
| 136 |
|
| 137 |
## 4. 서빙 및 호출
|
| 138 |
|
| 139 |
-
병합된 `bf16` 가중치(`safetensors` 4개)는 약 `19.3GB`(`18GiB`)입니다. 확인한 서빙 환경은 `NVIDIA A40
|
| 140 |
|
| 141 |
```bash
|
| 142 |
vllm serve mulkzz/RuBio-9B --served-model-name bio-feedback \
|
|
@@ -164,12 +189,12 @@ result = json.loads(response.choices[0].message.content)
|
|
| 164 |
|
| 165 |
## 5. 한계 및 라이선스
|
| 166 |
|
| 167 |
-
- **사용 범위 및 한계**: 학습 보조용 모델이므로 고부담 성적 채점에 직접 사용하면 안 됩니다. 『생명과학』 대단원 Ⅰ–Ⅲ 범위 밖의 문항이나 구조화된 `rubric`·`evidence`가 없는 일반 질의응답에는 적합하지 않습니다. 벤치마크의 학생 답은 합성 자료이며, **실제 학생 답에서의 성능은 재지 않았습니다**. 또한 RuBench v1 `
|
| 168 |
-
- **
|
| 169 |
-
- **
|
|
|
|
| 170 |
- `answer_status`(위에서부터 처음 맞는 것): 모두 `supported`이고 `other_issues`가 없으면 `correct` → `supported`가 있으면 `contradicted`나 `other_issues`�� 있을 때 `mixed`, 아니면 `partial` → `contradicted`나 `other_issues`가 있으면 `incorrect` → `unclear`가 있으면 `unclear` → 나머지 `no_attempt`
|
| 171 |
- `next_action`(위에서부터, 고친 `answer_status` 기준): `reveal_requested`면 `explain` → `correct`면 `complete` → `message_type`이 `other`면 `refocus` → `unclear`면 `clarify` → 나머지 `revise`
|
| 172 |
-
- 옛 dev 300건 재채점(스키마 없는 출력): 종합 판정 일치 285→291, 다음 조치 일치 289→290, 정책 통과 293→297.
|
| 173 |
- **라이선스**: 기본 모델과 동일하게 [Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0)을 따릅니다.
|
| 174 |
|
| 175 |
---
|
|
@@ -178,52 +203,54 @@ result = json.loads(response.choices[0].message.content)
|
|
| 178 |
|
| 179 |
<img src="./figures/rubio_9b_nature_fig1.svg" width="100%" style="width: 100%; height: auto;" alt="Fig. 3 | Training dynamics, optimization stability, and in-distribution dev set diagnostic performance of RuBio-9B" />
|
| 180 |
|
| 181 |
-
> **Fig. 3 | RuBio-9B 학습 동역학 및 학습 분포 내 옛 Dev 분할(n = 300) 진단 성능.** **a**, 판정 토큰 가중 손실(`W=10`) 기준 스텝별 학습 손실(원시 및 EMA α = 0.32, 유효 배치 `2 × 8 = 16`, 누적 토큰 `
|
| 182 |
|
| 183 |
-
### 1. 스텝별 학습 수렴 및 교사 강제 판정 지표
|
| 184 |
|
| 185 |
-
`RuBio-9B`는 스텝별 빠른 점검(quick-probe) 생성을 끄고 학습하여
|
| 186 |
|
| 187 |
-
|
|
| 188 |
-
|---:|---:|---:|---:|---:|---:|---:|---:|
|
| 189 |
-
|
|
| 190 |
-
|
|
| 191 |
-
|
|
| 192 |
-
|
|
| 193 |
-
|
|
| 194 |
-
|
|
| 195 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 196 |
|
| 197 |
### 2. 옛 Dev 300건 핵심 평가 지표 비교 (학습과 같은 분포)
|
| 198 |
|
| 199 |
-
|
| 200 |
-
|
| 201 |
-
|
|
| 202 |
-
|
|
| 203 |
-
| **
|
| 204 |
-
| **
|
| 205 |
-
| **
|
| 206 |
-
| **
|
| 207 |
-
| **
|
| 208 |
-
| **
|
| 209 |
-
| **
|
| 210 |
-
| **
|
| 211 |
-
|
| 212 |
-
|
| 213 |
-
|
| 214 |
-
### 3.
|
| 215 |
-
|
| 216 |
-
|
| 217 |
-
|
| 218 |
-
|
|
| 219 |
-
|
|
| 220 |
-
|
|
| 221 |
-
| **정
|
| 222 |
-
|
|
| 223 |
-
|
|
| 224 |
-
|
|
| 225 |
-
|
|
| 226 |
-
|
|
| 227 |
-
| **잔여 정책 위반 유형** (중복 포함) | `answer_status` 불일치 | 300 | **5건** | 후처리 시 해소 |
|
| 228 |
-
| | `student_span` 변형 | 300 | **3건** | 원문 발췌 미세 변형 |
|
| 229 |
-
| | 기준 상태 허용 값 위반 | 300 | **0건** | 허용된 4개 상태만 출력 |
|
|
|
|
| 21 |
|
| 22 |
## 1. 모델 요약
|
| 23 |
|
| 24 |
+
`mulkzz/RuBio-9B`는 고등학교 2022 개정 『생명과학』 대단원 Ⅰ–Ⅲ 서술형 문항의 학생 답안을 평가 기준별로 진단하고, 교사용 진단 정보와 학생용 피드백을 단일 JSON 객체로 출력하는 모델입니다. [Qwen/Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B)에 LoRA를 학습한 뒤 가중치를 병합한 **16비트(`bf16`) 전체 모델**(LoRA 어댑터가 아닌 병합 가중치, `safetensors` 4개 약 `19.3GB` / `18GiB`)이며, [mulkzz/RuBio-4B](https://huggingface.co/mulkzz/RuBio-4B)·[mulkzz/RuBio-8B](https://huggingface.co/mulkzz/RuBio-8B)와 상위 모델 [mulkzz/RuBio-27B](https://huggingface.co/mulkzz/RuBio-27B) 사이의 중간 규모 모델입니다.
|
| 25 |
|
| 26 |
+
**2026-11-04부터 `main` 브랜치 가중치는 보강 데이터(`RuIso-v6`, 3,579행)로 재학습한 `9b-v6-s3408`(시드 `3408`, 리비전 `f61ac9cb`, `ruiso-v6` 브랜치와 동일)로 바뀌었습니다.** 이전 `main` 가중치(`9b-wt`, 시드 `3407`, 최초 업로드 커밋 `e19404e3`)는 `revision="a0da6701"`로 계속 받을 수 있습니다.
|
| 27 |
+
|
| 28 |
+
출력 형식·`system` 프롬프트는 `RuBio-4B`·`RuBio-8B`·`RuBio-27B`와 같고, **학습 방법이 4B·27B와 한 가지 다릅니다**: 기준별 `status`·`misconception_id` 값 토큰을 손실에서 10번 세는 **판정 토큰 가중 손실(`W=10`)**을 적용했습니다(`RuBio-8B`도 일반 손실 `W=1`을 사용합니다). 이 때문에 `eval_loss`에 `W=10` 가중이 섞여 있어 `RuBio-4B`·`RuBio-8B`·`RuBio-27B`의 `eval_loss`와 바로 비교할 수 없으며, 상단 헤더 이미지의 `eval_loss*`(`0.4248`)가 이 뜻입니다. 시범 서비스(https://mulk.codes/rubio)는 `RuBio-4B`와 `RuBio-27B`만 운영하며, `RuBio-9B`는 서비스에 올리지 않았습니다.
|
| 29 |
|
| 30 |
- **시범 서비스**: https://mulk.codes/rubio (`RuBio-4B` · `RuBio-27B`만 운영하며, `RuBio-9B`는 올리지 않음)
|
| 31 |
+
- **관련 모델**: [mulkzz/RuBio-4B](https://huggingface.co/mulkzz/RuBio-4B) · [mulkzz/RuBio-8B](https://huggingface.co/mulkzz/RuBio-8B) · [mulkzz/RuBio-27B](https://huggingface.co/mulkzz/RuBio-27B)
|
|
|
|
| 32 |
|
| 33 |
### RuBench v1 벤치마크 성능 요약 (Out-of-Distribution Dev, n = 200)
|
| 34 |
|
| 35 |
<img src="./figures/rubench_v1_dev200_9b.svg" width="100%" style="width: 100%; height: auto;" alt="Fig. 1 | RuBench v1 (dev n = 200) diagnostic evaluation of RuBio-9B vs. its base model Qwen3.5-9B" />
|
| 36 |
|
| 37 |
+
> **Fig. 1 | RuBench v1 (dev 분할, n = 200 답안 · 604 평가 기준)에서 RuBio-9B(`mulkzz/RuBio-9B@f61ac9cb`, `9b-v6-s3408`)와 기본 모델 Qwen3.5-9B(`Qwen/Qwen3.5-9B`)의 진단 성능 비교.** **a**, 핵심 지표 5종 및 95% 신뢰구간(답안 군집 CI 및 Wilson CI). 주 지표인 기준 판정 정확도는 `RuBio-9B` `84.9% [81.6, 88.1]` 대 `Qwen3.5-9B` `76.6% [73.0, 80.2]`로 쌍대 차이 `+8.2%p [+4.9, +11.6]`(개선)이며, 2차 2/3 다수결 제외 시 `87.7% [84.6, 90.8]` 대 `81.4% [77.9, 84.8]`, 쌍대 차이 `+6.3%p [+2.7, +10.0]`(개선)임. 보조 지표는 오개념 일치 `111/133 = 83.5%` 대 `86/133 = 64.7%`, 기준 밖 오류 일치 `153/200 = 76.5%` 대 `153/200 = 76.5%`, 전체 상태 일치 `144/200 = 72.0%` 대 `96/200 = 48.0%`임. 점선은 `RuBio-9B`의 학습 분포 내 옛 dev(300건) 기준 일치율(`759/779 = 97.4%`). **b**, 정답 확정 출처별 기준 판정 정확도(`R1 3/3` `467/521 = 89.6%` 대 `440/521 = 84.5%`, `R2 3/3` `13/23 = 56.5%` 대 `7/23 = 30.4%`, `R2 2/3` `26/51 = 51.0%` 대 `13/51 = 25.5%`) 및 기준 밖 과학 오류(`other_issues`)의 오류 있음 재현율(TP)·오류 없음 특이도(TN) 상충 관계로, `RuBio-9B`가 오류 있음 재현율은 더 높으나(`42/53 = 79.2%` 대 `7/53 = 13.2%`) 오류 없음 특이도는 더 낮음(`111/147 = 75.5%`, 오탐 36건 대 `146/147 = 99.3%`, 오탐 1건). **c, d**, 미세조정 모델 `RuBio-9B`(기준 micro `512/604 = 84.8%`, JSON 오류 `1/200`)와 기본 모델 `Qwen3.5-9B`(기준 micro `463/604 = 76.7%`, JSON 오류 `0/200`)의 평가 기준별 판정(`checks`) 행 정규화 혼동 행렬(예측 열의 `invalid/none`은 깨진 JSON 또는 판정 누락으로, `RuBio-9B`에서 `finish_reason=length`로 잘린 답안 1건의 3개 기준이 `contradicted` 1건·`unclear` 2건에서 발생함). 두 모델 모두 `vLLM 0.32.0, bf16`(`RuBio-9B`는 `RunPod RTX A6000 48GB Pod`, `Qwen3.5-9B`는 `RunPod A40 Pod`)에서 첫 턴·`temperature=0`·`max_tokens=1024`·`thinking` 끔·JSON 스키마 강제 없음(깨진 JSON은 오답 처리) 조건에서 평가하되, `RuBio-9B`는 학습 때와 동일한 `SYSTEM_PROMPT`만 입력받고 기본 모델 `Qwen3.5-9B`는 `SYSTEM_PROMPT`에 판정·응답 정책(`POLICY`)을 추가해 입력받음.
|
| 38 |
|
| 39 |
<img src="./figures/rubench_v1_dev200_all_9b.svg" width="100%" style="width: 100%; height: auto;" alt="Fig. 2 | RuBench v1 (dev n = 200) criterion accuracy of RuBio, base Qwen models, and general-purpose LLMs" />
|
| 40 |
|
| 41 |
+
> **Fig. 2 | RuBench v1 (dev 분할, n = 200 답안 · 604 평가 기준)에서 RuBio-9B를 포함한 9개 모델(미세조정 RuBio 3종, 기본 Qwen 3종, 범용 프론티어 모델 2종, 참고 모델 1종)의 진단 성능 비��.** `RuBio-9B`(`84.9% [81.6, 88.1]`, 2차 2/3 제외 `87.7% [84.6, 90.8]`)는 경량 모델 `RuBio-4B v2`(`77.2% [73.4, 80.9]`) 및 기본 모델 `Qwen3.5-9B`(`76.6% [73.0, 80.2]`)보다 높고, 상위 모델 `RuBio-27B`(`c1cfefdb`, `86.9% [83.8, 89.9]`)와는 95% 신뢰구간이 0을 포함하여 유의한 차이가 없으며, 범용 모델(`GPT-6 Astra` `91.2% [88.6, 93.7]`, `Claude Opus 5.5` `94.2% [92.0, 96.4]`)보다는 낮음. **a**, 9개 모델의 기준 판정 정확도(답안 단위 군집 95% CI, 막대, `n=604`)와 2차 2/3 다수결 제외 값(빈 마름모, `n=553`)으로, 4B 쌍(`RuBio-4B v2` `77.2% [73.4, 80.9]` · 제외 `79.9% [76.0, 83.8]` · 기준 micro `467/604`; `Qwen3.5-4B` `70.9% [67.3, 74.5]` · 제외 `74.5% [71.0, 78.1]` · micro `427/604`), 9B 쌍(`RuBio-9B` `84.9% [81.6, 88.1]` · 제외 `87.7% [84.6, 90.8]` · micro `512/604`; `Qwen3.5-9B` `76.6% [73.0, 80.2]` · 제외 `81.4% [77.9, 84.8]` · micro `463/604`), 27B 쌍(`RuBio-27B` `86.9% [83.8, 89.9]` · 제외 `90.0% [87.1, 92.9]` · micro `524/604`; `Qwen3.8-27B` `81.8% [78.2, 85.4]` · 제외 `85.0% [81.4, 88.5]` · micro `495/604`), 범용(`GPT-6 Astra (xhigh)`† `91.2% [88.6, 93.7]` · 제외 `95.6% [93.6, 97.5]` · micro `550/604`; `Claude Opus 5.5 (xhigh)`† `94.2% [92.0, 96.4]` · 제외 `96.1% [94.2, 98.0]` · micro `570/604`), 참고(`Gemini 4 Argon (High)`, 빗금, `91.2% [88.5, 94.0]` · 제외 `93.6% [91.0, 96.2]` · micro `551/604`) 순서임. **b**, 답안 단위 짝 차이(%p, 95% 군집 CI; 하한 > 0 개선, 상한 < 0 악화, 그 밖 차이 없음) forest plot(전체 = 채운 원, 2차 2/3 다수결 제외 = 빈 원). `RuBio-9B − Qwen3.5-9B`는 전체 `+8.2%p [+4.9, +11.6]`·제외 `+6.3%p [+2.7, +10.0]`(개선), `RuBio-4B − Qwen3.5-4B`는 전체 `+6.3%p [+2.6, +9.9]`·제외 `+5.3%p [+1.2, +9.5]`(개선), `RuBio-27B − Qwen3.8-27B`는 전체 `+5.1%p [+1.1, +9.1]`·제외 `+5.0%p [+1.1, +9.0]`(개선)임. 규모 간 비교는 `RuBio-9B − RuBio-4B`가 전체 `+7.7%p [+4.7, +10.8]`·제외 `+7.8%p [+4.3, +11.3]`(개선), `RuBio-27B − RuBio-9B`가 전체 `+2.0%p [-1.4, +5.4]`·제외 `+2.3%p [-1.1, +5.7]`(차이 없음)임. 범용 모델 대비는 `GPT − RuBio-9B`가 전체 `+6.3%p [+3.1, +9.5]`·제외 `+7.9%p [+4.9, +10.9]`(개선), `Opus − RuBio-9B`가 전체 `+9.3%p [+6.2, +12.5]`·제외 `+8.4%p [+5.6, +11.2]`(개선)이고, `Opus − GPT`는 전체 `+3.0%p [+0.4, +5.6]`(개선)·제외 `+0.5%p [-1.5, +2.5]`(차이 없음)임. 회색 보조 행 `RuBio-9B − Qwen3.5-9B, same prompt*`는 `Qwen3.5-9B`에 `SYSTEM_PROMPT`만 준 비공식 참고 비교로, 이 모델이 200개 중 189개에서 `checks`를 `{criterion_id: status}` 사전으로 내 엄격 채점 시 `3.8% [1.3, 6.2]`(제외 `4.2% [1.6, 6.8]`, 미시 `21/604`)이고 사전을 목록으로 바꾼 관대 채점(`76.0% [72.3, 79.7]`, 제외 `80.1% [76.4, 83.7]`, 미시 `458/604`) 대비 차이는 전체 `+8.9%p [+5.1, +12.6]`·제외 `+7.6%p [+3.6, +11.7]`(개선)임. **c**, 보조 지표 5종 히트맵(`RuBio-4B v2`, `Qwen3.5-4B`, `RuBio-9B`, `Qwen3.5-9B`, `RuBio-27B`, `Qwen3.8-27B`, `GPT`, `Opus`, `Gemini` 순)으로, 오개념 ID 일치(`133`기준: `106/133 = 79.7%`, `78/133 = 58.6%`, `111/133 = 83.5%`, `86/133 = 64.7%`, `118/133 = 88.7%`, `107/133 = 80.5%`, `129/133 = 97.0%`, `127/133 = 95.5%`, `122/133 = 91.7%`), 기준 밖 오류 재현율(TP, `53`답안: `10/53 = 18.9%`, `4/53 = 7.5%`, `42/53 = 79.2%`, `7/53 = 13.2%`, `46/53 = 86.8%`, `29/53 = 54.7%`, `33/53 = 62.3%`, `52/53 = 98.1%`, `45/53 = 84.9%`), 오류 없음 특이도(TN, `147`답안: `143/147 = 97.3%`, `146/147 = 99.3%`, `111/147 = 75.5%`, `146/147 = 99.3%`, `127/147 = 86.4%`, `127/147 = 86.4%`, `144/147 = 98.0%`, `143/147 = 97.3%`, `145/147 = 98.6%`), `answer_status` 규칙 일치(`200`답안: `124/200 = 62.0%`, `89/200 = 44.5%`, `144/200 = 72.0%`, `96/200 = 48.0%`, `154/200 = 77.0%`, `120/200 = 60.0%`, `176/200 = 88.0%`, `184/200 = 92.0%`, `179/200 = 89.5%`), 유효 JSON(`RuBio-9B` `199/200 = 99.5%`, `Qwen3.8-27B` `195/200 = 97.5%`, `Gemini` `198/200 = 99.0%`, 나머지 6개 모델 `200/200 = 100.0%`)을 제시함. **d**, 정답 확정 출처별 기준 정확도와 Wilson 95% CI로, 같은 9개 모델 순서대로 `R1 3/3`(`521`기준: `431/521 = 82.7%`, `402/521 = 77.2%`, `467/521 = 89.6%`, `440/521 = 84.5%`, `483/521 = 92.7%`, `461/521 = 88.5%`, `510/521 = 97.9%`, `510/521 = 97.9%`, `496/521 = 95.2%`), `R2 3/3`(`23`기준: `10/23 = 43.5%`, `4/23 = 17.4%`, `13/23 = 56.5%`, `7/23 = 30.4%`, `11/23 = 47.8%`, `5/23 = 21.7%`, `14/23 = 60.9%`, `18/23 = 78.3%`, `19/23 = 82.6%`), `R2 2/3`(`51`기준: `23/51 = 45.1%`, `15/51 = 29.4%`, `26/51 = 51.0%`, `13/51 = 25.5%`, `23/51 = 45.1%`, `24/51 = 47.1%`, `22/51 = 43.1%`, `37/51 = 72.5%`, `32/51 = 62.7%`), 사람 판정(`human`, `9`기준: `3/9 = 33.3%`, `6/9 = 66.7%`, `6/9 = 66.7%`, `3/9 = 33.3%`, `7/9 = 77.8%`, `5/9 = 55.6%`, `4/9 = 44.4%`, `5/9 = 55.6%`, `4/9 = 44.4%`)임. †`GPT-6 Astra (xhigh)`와 `Opus 5.5 (xhigh)`는 1차 정답 패널(`GPT-6 Astra xhigh`·`Opus 5.5 xhigh`·`Grok 4.7 xhigh`)이므로 `R1 3/3` 항목(`604`개 중 `521`개)에 자기 블라인드 판정이 들어 있어 점수가 자기 일치를 포함한 상한이며, 2차 다수결(`R2 2/3`)에도 2차 패널인 `GPT`·`Opus`·`Gemini`의 투표가 참여함. `Gemini 4 Argon (High)`은 의도된 라벨로 가상 학생 답을 쓴 모델이라 참고값으로만 제시하고 비교 판정에서 제외하며(`Gemini − RuBio-9B` 단순 차이는 전체 `+6.4%p [+2.6, +10.2]`·제외 `+5.9%p [+2.4, +9.3]`), 1차 패널인 `Grok 4.7`은 모델로 평가하지 않음. 평가 조건은 첫 턴(`feedback_mode="hint_1"`, `reveal_requested=false`, `history=[]`)·JSON 스키마 강제 없음·깨진 JSON은 오답 처리로 동일하되, `RuBio`는 `SYSTEM_PROMPT`만, 기본·범용 모델은 `SYSTEM_PROMPT+POLICY`를 입력받으며, `RuBio`와 `Qwen` 기본은 `temperature=0`·`max_tokens=1024`·`thinking` 끔으로, 범용 모델은 표기한 추론 강도로 CLI 1회 실행해 출력을 고정함(응답 시간 중앙값 `GPT` `575.6초`, `Opus` `41.3초`, `Gemini` `87.9초`).
|
| 42 |
|
| 43 |
+
**RuBench v1**은 새 문항·새 상황·가상 학생 페르소나로 구축한 2022 개정 『생명과학』 서술형 진단 벤치마크입니다(19개 성취기준, 대단원 Ⅰ–Ⅲ 298문항 중 `dev = 200` / `test = 98`, 가상 학생 30명 × 성취수준 A–E 층화 배정; `test` 분할은 열지 않았습니다). 학습 문항과의 5-gram 중복(`< 0.40`) 및 장면·사례 중복을 차단하고, 모범답안·허용 표현을 가린 채 생성한 학생 답안을 3개 프론티어 모델(`GPT-6 Astra xhigh` · `Opus 5.5 xhigh` · `Grok 4.7 xhigh`) 블라인드 채점, 2차 블라인드 재채점, 인간 검수로 확정했습니다. 모든 모델은 첫 턴 `feedback_mode="hint_1"`, `reveal_requested=false`, `history=[]`, `temperature=0`, `max_tokens=1024`, `enable_thinking=false`, **JSON 스키마 강제 없음**(깨진 JSON은 오답 처리) 조건에서 평가합니다.
|
| 44 |
|
| 45 |
+
| 구분 | 평가 지표 (RuBench v1 `dev`) | 표본 수 | RuBio-9B (`@f61ac9cb`) | Qwen3.5-9B (`Qwen/Qwen3.5-9B`)<sup>§</sup> | 쌍대 차이 Δ (미세조정 − 기본) 및 비고 |
|
| 46 |
|---|---|---:|---:|---:|---|
|
| 47 |
+
| **주 지표** | **기준 판정 정확도** (답안 단위 평균, 95% 군집 CI)<sup>†</sup> | 200 답안 (604 기준) | **84.9%** `[81.6, 88.1]` | 76.6% `[73.0, 80.2]` | +8.2%p `[+4.9, +11.6]` (**개선**) |
|
| 48 |
+
| | **기준 판정 정확도** (2차 2/3 다수결 항목 제외)<sup>‡</sup> | 200 답안 (553 기준) | **87.7%** `[84.6, 90.8]` | 81.4% `[77.9, 84.8]` | +6.3%p `[+2.7, +10.0]` (**개선**) |
|
| 49 |
+
| | **기준 판정 미시 일치율** (`checks` 전체 합산) | 604 기준 | **512/604 (84.8%)** | 463/604 (76.7%) | +8.1%p (+49 기준) |
|
| 50 |
+
| **보조 지표** | **오개념 진단 일치** (정답 `contradicted` + ID 일치) | 133 기준 | **111/133 (83.5%)** | 86/133 (64.7%) | +18.8%p (ID 지정 79건: **65/79 (82.3%)** vs 62/79 (78.5%)) |
|
| 51 |
+
| | **기준 밖 오류(`other_issues`) 유무 일치** | 200 답안 | **153/200 (76.5%)** | **153/200 (76.5%)** | 0.0%p (오류 있음 53건 재현율 TP: **42/53 (79.2%)** vs 7/53 (13.2%) / 오류 없음 147건 특이도 TN: 111/147 (75.5%, 오탐 36건) vs **146/147 (99.3%, 오탐 1건)**) |
|
| 52 |
+
| | **종합 판정(`answer_status`) 규칙 일치** | 200 답안 | **144/200 (72.0%)** | 96/200 (48.0%) | +24.0%p (+48 답안) |
|
| 53 |
+
| **정답 출처별** | 1차 3/3 만장일치 확정 기준 (`r1-3/3`) | 521 기준 | **467/521 (89.6%)** | 440/521 (84.5%) | +5.2%p |
|
| 54 |
+
| | 2차 3/3 만장일치 확정 기준 (`r2-3/3`) | 23 기준 | **13/23 (56.5%)** | 7/23 (30.4%) | +26.1%p |
|
| 55 |
+
| | 2차 2/3 다수결 확정 기준 (`r2-2/3`, 경계 사례) | 51 기준 | **26/51 (51.0%)** | 13/51 (25.5%) | +25.5%p (채점 패널 간 갈린 모호한 경계 답안) |
|
| 56 |
| | 사용자 직접 판정 기준 (`human`) | 9 기준 | **6/9 (66.7%)** | 3/9 (33.3%) | +33.3%p (3개 모델 모두 갈린 고난도 항목) |
|
| 57 |
+
| **대단원별** | Ⅰ. 생명과학의 이해 (`checks` 미시 일치) | 206 기준 | **170/206 (82.5%)** | 152/206 (73.8%) | +8.7%p |
|
| 58 |
+
| | Ⅱ. 사람의 물질대사 (`checks` 미시 일치) | 200 기준 | **171/200 (85.5%)** | 156/200 (78.0%) | +7.5%p |
|
| 59 |
+
| | Ⅲ. 항상성과 몸의 조절 (`checks` 미시 일치) | 198 기준 | **171/198 (86.4%)** | 155/198 (78.3%) | +8.1%p |
|
| 60 |
+
| **출력 안정성** | **JSON 유효 응답률** (스키마 강제 없음) | 200 답안 | 199/200 (99.5%) | **200/200 (100.0%)** | -0.5%p (JSON 파싱 오류 1건 `finish_reason=length` vs **0건**) |
|
| 61 |
|
| 62 |
<sup>†</sup> **벤치마크 설계 때 정한 판정 규칙**: 같은 답안에서의 쌍대 차이(paired difference) 95% 군집 신뢰구간 하한이 `0`보다 크면 **개선**, 상한이 `0`보다 작으면 **악화**, `0`을 포함하면 **차이 없음**으로 판정합니다(Miller 2024, arXiv:2411.00640).<br/>
|
| 63 |
<sup>‡</sup> 2차 채점의 2/3 다수결 확정분(`r2-2/3`, 51개 기준)을 제외하고 만장일치·사용자 판정 항목만으로 계산한 민감도 분석입니다.<br/>
|
| 64 |
+
<sup>§</sup> **평가 조건 및 서빙 경로**: `RuBio-9B`(`9b-v6-s3408`, `@f61ac9cb`)는 학습 Pod인 `RunPod RTX A6000 48GB`의 `vLLM 0.32.0`(`bf16`, `--language-model-only --max-model-len 8192 --max-num-seqs 32`)에서, 기본 모델(`Qwen/Qwen3.5-9B`)은 `RunPod A40 Pod`의 `vLLM 0.32.0`(`bf16`)에서 첫 턴·`temperature=0`·`max_tokens=1024`·`thinking` 끔·JSON 스키마 강제 없음(깨진 JSON은 오답 처리) 조건으로 평가하되, `RuBio-9B`는 학습 때와 같은 `SYSTEM_PROMPT`만 받으며 기본 모델(`Qwen/Qwen3.5-9B`)은 `SYSTEM_PROMPT`+판정·응답 정책(`POLICY`)을 받습니다. Fig. 2의 `RuBio-27B`도 RuIso-v6 재학습 새 `main` 가중치(`27b-v6-s3407`, `@c1cfefdb`)입니다.
|
| 65 |
+
|
| 66 |
+
### RuIso-v6 재학습과 가중치 교체
|
| 67 |
+
|
| 68 |
+
- **재학습한 까닭**: 최소쌍 진단에서 이전 `RuBio`가 학생 답의 추측 말투("~인 것 같아요")와 용어 나열 문체를 오답 신호로 쓰고, 답안에 평가 기준과 무관한 기준 밖 오류가 있으면 이미 충족한 기준까지 깎는 경향이 확인되었습니다.
|
| 69 |
+
- **RuIso-v6 데이터 보강 (`train = 3,579`행, `dev = 300`행)**: 기존 학습 데이터 `2,400`행에 새 합성 데이터 `1,179`행을 더해 총 `3,579`행으로 재학습했습니다(`dev = 300`행은 그대로 유지).
|
| 70 |
+
- **문체 변형 (`708`행)**: 같은 문항 계열의 첫 턴 답에서 내용·오류·누락은 그대로 두고 문��만 바꾼 변형입니다(추측 말투 `235`행, 압축 나열 `236`행, 1–2문장으로 짧게 `237`행; 기준 라벨은 원본으로 고정하고 블라인드 재채점으로 확인).
|
| 71 |
+
- **기준 밖 과학 오류 주입 (`471`행)**: 맞거나 부분적으로 맞는 답에 평가 기준과 무관한 명백한 과학 오류를 한 군데 넣은 행입니다(기준별 판정은 원본으로 고정하고 `other_issues`만 추가했으며, `answer_status`는 규칙으로 다시 계산).
|
| 72 |
+
- **합성 및 검수 절차**: 학생 답 다시 쓰기는 `Claude Opus 5.5 (xhigh)`, 블라인드 채점과 피드백 작성은 `Gemini 4 Argon (High)`이 수행했습니다. 계획한 `1,200`행 중 고정 라벨과 블라인드 라벨이 갈린 행은 버리거나 검수 대기열에 보내 `1,179`행이 남았습니다. RuBench 문항·답과 최소쌍 변형은 학습에 넣지 않았으며, 모든 학생 답은 합성 자료이고 실제 학생 자료가 아닙니다.
|
| 73 |
+
- **결과를 보기 전에 정한 규칙**:
|
| 74 |
+
1. 시드가 둘인 규모는 RuBench 채점 전에 `best_eval_loss`가 낮은 시드를 공개 후보로 고릅니다(`RuBio-9B`는 시드 `3408`의 `0.4248` < 시드 `3407`의 `0.4262`이므로 `9b-v6-s3408`을 선정).
|
| 75 |
+
2. 같은 Pod·같은 `vLLM 0.32.0`에서 현재 HF `main`(리비전 `a0da6701` 고정)을 다시 돌린 대조군과 비교해, RuBench v1 `dev` 200 주 지표(기준 판정 정확도) 짝 차이 점추정이 `0`보다 크고 정답 `missing`·`contradicted` 기준의 재현율이 각각 대조군 이상이면 `main` 가중치를 바꿉니다. 아니면 `ruiso-v6` 브랜치에만 둡니다. 이 규칙은 95% 신뢰구간 하한을 요구하지 않으므로, 교체됐더라도 신뢰구간이 `0`을 포함하면 그렇게 밝힙니다(`RuBio-9B`는 짝 차이 `+3.8%p [+1.3, +6.3]`으로 95% 신뢰구간 하한도 `0`보다 큽니다).
|
| 76 |
+
|
| 77 |
+
| 모델 키 | 모델 및 실행 조건 | 주 지표 (전체, 95% CI) | R2 2/3 제외 | 기준 미시 | `supported` (283) | `contradicted` (133) | `missing` (142) | `unclear` (46) | 기준 밖 TP/FP (오류 53) | `answer_status` (200) | JSON 오류 |
|
| 78 |
+
|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
|
| 79 |
+
| **`9b-v6-s3408` (선정 `main`)** | `RuBio-9B RuIso-v6` (시드 `3408`, `@f61ac9cb`, RTX A6000 Pod) | **84.9%** `[81.6, 88.1]` | 87.7% | **512/604** | **263/283 (92.9%)** | **124/133 (93.2%)** | 102/142 (71.8%) | 23/46 (50.0%) | **42** / 36 | **144/200** | 1 |
|
| 80 |
+
| `9b-pub-pod-p3408` (대조군) | `RuBio-9B 공개판` (시드 `3408` Pod, `@a0da6701`) | 81.0% `[77.5, 84.6]` | 83.7% | 489/604 | 240/283 (84.8%) | 122/133 (91.7%) | 101/142 (71.1%) | **26/46 (56.5%)** | 16 / **10** | 131/200 | **0** |
|
| 81 |
+
| `9b-v6-s3407` | `RuBio-9B RuIso-v6` (시드 `3407`, `local rubio-9b-v6-s3407`, A40 Pod) | **84.9%** `[81.7, 88.1]` | **88.0%** | 511/604 | 261/283 (92.2%) | 121/133 (91.0%) | **110/142 (77.5%)** | 19/46 (41.3%) | 41 / 41 | 134/200 | **0** |
|
| 82 |
+
| `9b-pub-pod-p3407` (대조군) | `RuBio-9B 공개판` (시드 `3407` Pod, `@a0da6701`) | 81.0% `[77.5, 84.6]` | 83.7% | 489/604 | 239/283 (84.5%) | 122/133 (91.7%) | 102/142 (71.8%) | **26/46 (56.5%)** | 17 / **10** | 131/200 | **0** |
|
| 83 |
+
|
| 84 |
+
- **같은 Pod 대조군 대비 짝 차이 및 Pod 간 재현성**: 선정 모델 `9b-v6-s3408`과 같은 Pod(`RTX A6000 48GB`)에서 다시 돌린 옛 `main` 대조군 `9b-pub-pod-p3408`의 주 지표 짝 차이는 전체 `+3.8%p [+1.3, +6.3]`(**개선**), 2차 2/3 제외 `+4.0%p [+1.2, +6.8]`(**개선**)입니다. 시드 `3407` 재학습본(`9b-v6-s3407`)도 대조군 대비 전체 `+3.8%p [+1.0, +6.7]`(**개선**), 2차 2/3 제외 `+4.3%p [+1.2, +7.4]`(**개선**)입니다. 서로 다른 두 Pod(`RTX A6000 48GB`의 `9b-pub-pod-p3408`과 `A40 48GB`의 `9b-pub-pod-p3407`)에서 돌린 옛 `main`(`@a0da6701`) 대조군끼리의 차이는 전체 `+0.0%p [-0.5, +0.5]`(**차이 없음**), 2차 2/3 제외 `+0.0%p [-0.5, +0.5]`(**차이 없음**), 기준 미시 `489/604` 동일로 Pod 간 재현성이 확인되었습니다.
|
| 85 |
+
- **최소쌍 A–E 진단 결과**: 정답이 `supported`인 기준만 대상으로 문체 변형 전후 `supported` 유지 여부를 비교한 표입니다(변형 뒤에도 `supported`로 남아야 이상적이며, 표본이 작아 경향으로만 해석합니다). 옛 공개판 대조군은 추측 말투가 있는 원본(A)에서 `45/74`(`p3408`)·`44/74`(`p3407`)만 `supported`로 인정하고 추측 말투를 빼면 `60/74`(`+16 −1` 및 `+17 −1`)로 뛰었으나, `9b-v6-s3408`은 원본에서도 `61/74`를 `supported`로 판정해 변형 후 `64/74`(`+4 −1`)와 격차가 줄었습니다.
|
| 86 |
+
|
| 87 |
+
| 최소쌍 변형 (정답 `supported` 기준만) | 표본 수 | `9b-v6-s3408` (선정 `main`) | `9b-pub-pod-p3408` (옛 `main` 대조군) | `9b-v6-s3407` (시드 3407) | `9b-pub-pod-p3407` (옛 `main` 대조군) |
|
| 88 |
+
|---|---:|---:|---:|---:|---:|
|
| 89 |
+
| **A. 추측 말투 제거** ("~인 것 같아요" → 평서문) | 36답 · 74기준 | `61→64/74 (+4 −1)` | `45→60/74 (+16 −1)` | `62→63/74 (+3 −2)` | `44→60/74 (+17 −1)` |
|
| 90 |
+
| **B. 추측 말투 삽입** (평��문 → 추측 말투) | 12답 · 28기준 | `26→27/28 (+1 −0)` | `24→25/28 (+2 −1)` | `27→27/28 (+0 −0)` | `24→25/28 (+2 −1)` |
|
| 91 |
+
| **C. 교과서식 문장 정돈** | 12답 · 27기준 | `27→27/27 (+0 −0)` | `27→27/27 (+0 −0)` | `27→27/27 (+0 −0)` | `27→27/27 (+0 −0)` |
|
| 92 |
+
| **D. 나열 → 관계 문장** (A 변형과 비교) | 10답 · 25기준 | `22→25/25 (+3 −0)` | `21→25/25 (+4 −0)` | `22→25/25 (+3 −0)` | `21→25/25 (+4 −0)` |
|
| 93 |
+
| **E. 문장 → 용어 나열** | 12답 · 26기준 | `25→25/26 (+1 −1)` | `25→24/26 (+1 −2)` | `26→26/26 (+0 −0)` | `25→24/26 (+1 −2)` |
|
| 94 |
+
|
| 95 |
+
- **가중치 교체 판정과 그 한계**: `9b-v6-s3408`은 같은 Pod 대조군(`9b-pub-pod-p3408`) 대비 주 지표 짝 차이 점추정이 `+3.8%p > 0`이고, 정답 `missing` 재현율(`102/142 = 71.8% >= 101/142 = 71.1%`)과 정답 `contradicted` 재현율(`124/133 = 93.2% >= 122/133 = 91.7%`)이 모두 대조군 이상이므로 결과를 보기 전에 정한 규칙에 따라 `main` 가중치를 교체했습니다(`RuBio-27B`도 규칙을 충족해 `main`을 `c1cfefdb`로 바꿨고, `RuBio-4B`는 `contradicted` 재현율 `117/133 < 122/133`으로 낮아 `ruiso-v6` 브랜치 `553fb15b`에만 두었으며 `main` `6f3e415a`를 유지했습니다. `RuBio-8B`는 새 모델 저장소 `21e6bf3d`라 교체 규칙이 없습니다). 다만 **같은 RuBench v1 `dev` 분할을 보고 가중치 교체를 정했으므로, 옛 `main` 가중치와의 비교는 독립 검증이 아닙니다**(`test` 분할은 열지 않았습니다).
|
| 96 |
+
- **이전 판(`revision="a0da6701"`) 이력**: 초기 공개판(`revision="a0da6701"`, 가중치 커밋 `e19404e3`)은 기존 2,400행으로 학습한 9B 후보 3종(`r16` 시드 3407·`W=1` `79.0%`, `s3408` 시드 3408·`W=1` `80.5%`, `wt` 시드 3407·`W=10` `80.7% [77.2, 84.2]`) 중 RuBench v1 `dev` 주 지표가 가장 높았던 `wt` 모델이었습니다.
|
| 97 |
+
- **기본 모델(`Qwen3.5-9B`) 대비 해석 및 같은 프롬프트 보조 비교(참고값)**:
|
| 98 |
+
- **공식 비교(`SYSTEM_PROMPT+POLICY` 제공 기본 모델 대비)**: Fig. 1 데이터 기준 `RuBio-9B`(`84.9% [81.6, 88.1]`)와 기본 모델 `Qwen3.5-9B`(`76.6% [73.0, 80.2]`)의 기준 판정 정확도 짝 차이는 전체 기준에서 `+8.2%p [+4.9, +11.6]`(**개선**), 2차 2/3 다수결 제외 시 `+6.3%p [+2.7, +10.0]`(**개선**)입니다. 혼동 행렬(정답 `supported` 283, `contradicted` 133, `missing` 142, `unclear` 46)에서 기본 모델은 정답 `missing` 142건 중 `47건`(`33.1%`)과 `unclear` 46건 중 `28건`(`60.9%`)을 `supported`로 과다 판정했으나(`missing->missing` `72/142 = 50.7%`, `unclear->unclear` `8/46 = 17.4%`), `RuBio-9B`는 `missing->supported`가 `22건`(`15.5%`), `unclear->supported`가 `12건`(`26.1%`)으로 줄고 `missing->missing` `102/142`(`71.8%`), `unclear->unclear` `23/46`(`50.0%`), `contradicted->contradicted` `124/133`(`93.2%`, 기본 모델 `113/133 = 85.0%`)로 높으며 `supported->supported`도 `263/283`(`92.9%`, 옛 `main` 대조군 `240/283 = 84.8%`, 기본 모델 `270/283 = 95.4%`)입니다.
|
| 99 |
+
- **같은 프롬프트 보조 비교(`Qwen3.5-9B`에 `SYSTEM_PROMPT`만 제공, Fig. 2 b 회색 행)**: 기본 모델 `Qwen3.5-9B`에 `POLICY` 없이 `RuBio`와 동일한 `SYSTEM_PROMPT`만 준 경우, JSON 파싱 오류는 `0건`이나 200개 답안 중 `189개`에서 `checks`를 배열이 아닌 `{criterion_id: status}` 사전으로 출력하여 **엄격 채점 시 `3.8% [1.3, 6.2]`(2차 2/3 제외 `4.2% [1.6, 6.8]`, 미시 `21/604`, `RuBio-9B` 대비 짝 차이 전체 `+81.1%p [+77.1, +85.2]` · 제외 `+83.5%p [+79.5, +87.6]`)**에 그칩니다. 이 사전을 목록으로 바꾼 **관대 채점 기준 판정 정확도는 `76.0% [72.3, 79.7]`(2차 2/3 제외 `80.1% [76.4, 83.7]`, 미시 `458/604`)**이며, `RuBio-9B`에서 이 관대 채점 값을 뺀 짝 차이는 전체 `+8.9%p [+5.1, +12.6]`(**개선**), 2차 2/3 제외 `+7.6%p [+3.6, +11.7]`(**개선**)입니다. 이는 공식 비교가 아닌 참고값입니다.
|
| 100 |
+
- **기준 밖 과학 오류(`other_issues`) 탐지·오탐 상충**: 기준 밖 오류 유무 전체 일치율은 `RuBio-9B` `153/200`(`76.5%`)과 기본 모델 `Qwen3.5-9B` `153/200`(`76.5%`)이 같으나, 오류가 있는 답안 53건의 재현율(TP)은 `RuBio-9B`가 `42/53`(`79.2%`)으로 옛 `main` 대조군(`16/53 = 30.2%`) 및 기본 모델(`7/53 = 13.2%`)보다 크게 높아진 대신, 오류 없는 답안 147건에서의 특이도(TN)는 `111/147`(`75.5%`, 오탐 36건)로 옛 `main` 대조군(`137/147 = 93.2%`, 오탐 10건) 및 기본 모델(`146/147 = 99.3%`, 오탐 1건)보다 낮아졌습니다.
|
| 101 |
|
| 102 |
## 2. 학습 설정
|
| 103 |
|
| 104 |
+
- **베이스 모델 및 템플릿 검증**: 기본 모델은 [Qwen/Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B)(Apache-2.0)입니다. 실제 학습에는 `unsloth/Qwen3.5-9B`를 불러왔으며 가중치 `sha256`과 chat template 렌더 결과가 공식 `Qwen/Qwen3.5-9B`와 같습니다.
|
| 105 |
+
- **학습 데이터 (`RuIso-v6`)**: 기존 학습 데이터 `2,400`행(300 문항 계열 × 10 케이스)에 문체 변형 `708`행(추측 말투 `235`, 압축 나열 `236`, 1–2문장 요약 `237`)과 기준 밖 과학 오류 주입 `471`행을 더한 **`train = 3,579`행**이며, 검증 분할은 기존 **`dev = 300`행**(문항 계열 단위 분리)을 그대로 사용했습니다.
|
| 106 |
+
- **LoRA 및 최적화 설정**: 데이터 외 하이퍼파라미터는 이전 공개판과 같습니다. Unsloth `bf16` LoRA(`r=16`, `alpha=32`, 대상 모듈 `q·k·v·o·gate·up·down`)를 2 에폭(`448`스텝, 1에폭 = `224`스텝), 배치 `2 × 누적 8`(유효 배치 `16`), 학습률 `1e-4` cosine 스케줄러, 예열 `20`스텝(`4%`), 최대 길이 `4,096`, `thinking` 끔, assistant JSON 구간에만 손실 적용으로 학습하고, 50스텝마다(마지막 step `448` 포함) `dev eval_loss`를 재 최저 손실 체크포인트를 선택한 뒤 16비트(`bf16`)로 병합했습니다.
|
| 107 |
+
- **소프트웨어 환경**: `unsloth` · `unsloth_zoo 2026.10.11`, `transformers 5.17.0`, `trl 1.15.0`, `torch 2.14.1+cu132`, `fla 0.6.0`.
|
| 108 |
+
- **판정 토큰 가중 손실(`W=10`, 4B·27B와 다른 점) 및 `eval_loss*` 주의**: 기준별 `status`·`misconception_id` 값 토큰을 손실에서 10번 셉니다(`W=10`). 따라서 `eval_loss`는 `W=10` 가중이 섞여 있어 일반 손실(`W=1`)로 학습한 `RuBio-4B`·`RuBio-8B`·`RuBio-27B`와 바로 비교할 수 없으며, 상단 헤더의 `eval_loss*`가 이 뜻입니다.
|
| 109 |
+
- **시드 2종(`3407`·`3408`) 학습 및 체크포인트 선정**: 두 시드를 별도 Pod에서 학습했습니다.
|
| 110 |
+
- **시드 `3408` (선정, `9b-v6-s3408`)**: `NVIDIA RTX A6000 48GB` Pod에서 `train_runtime = 9,548초`(약 `159분`, 약 `1,443 tok/s`) 동안 학습했으며, 평균 `train_loss = 0.3822`, **step 400에서 `best_eval_loss = 0.4248`**(step 400 교사 강제 `status_acc = 0.9756`, `ce = 0.0156`; 마지막 step 448도 `eval_loss = 0.4248`)을 기록했습니다.
|
| 111 |
+
- **시드 `3407` (`9b-v6-s3407`)**: `NVIDIA A40 48GB` Pod에서 `train_runtime = 9,744초` 동안 학습했으며, 평균 `train_loss = 0.3830`, step 448에서 `best_eval_loss = 0.4262`를 기록했습니다.
|
| 112 |
+
- 결과를 보기 전에 정한 규칙(`best_eval_loss` 비교: `0.4248 < 0.4262`)에 따라 시드 `3408`(`step 400` 체크포인트)을 골라 `bf16` `safetensors` 4개(약 `19.3GB` / `19.31GB` / `18GiB`, 리비전 `f61ac9cb`)로 병합해 업로드했습니다.
|
| 113 |
|
| 114 |
## 3. 입력 및 출력 규격
|
| 115 |
|
|
|
|
| 161 |
|
| 162 |
## 4. 서빙 및 호출
|
| 163 |
|
| 164 |
+
병합된 `bf16` 가중치(`safetensors` 4개)는 약 `19.3GB`(`18GiB`)입니다. 확인한 서빙 환경은 `NVIDIA RTX A6000 48GB` 및 `NVIDIA A40 48GB`와 `vLLM 0.32.0`이며(더 작은 GPU는 시험하지 않았습니다), 기본 chat template의 추론 강도 지시문이 붙지 않도록 `enable_thinking: false`를 지정합니다. 시범 서비스(https://mulk.codes/rubio)는 `RuBio-4B`·`RuBio-27B`만 운영하며 `RuBio-9B`는 서비스에 올리지 않았습니다.
|
| 165 |
|
| 166 |
```bash
|
| 167 |
vllm serve mulkzz/RuBio-9B --served-model-name bio-feedback \
|
|
|
|
| 189 |
|
| 190 |
## 5. 한계 및 라이선스
|
| 191 |
|
| 192 |
+
- **사용 범위 및 한계**: 학습 보조용 모델이므로 고부담 성적 채점에 직접 사용하면 안 됩니다. 『생명과학』 대단원 Ⅰ–Ⅲ 범위 밖의 문항이나 구조화된 `rubric`·`evidence`가 없는 일반 질의응답에는 적합하지 않습니다. RuBench 벤치마크와 `RuIso-v6` 보강 데이터의 학생 답은 모두 합성 자료(보강 데이터의 학생 답 다시 쓰기는 `Claude Opus 5.5 (xhigh)`, 블라인드 채점과 피드백 작성은 `Gemini 4 Argon (High)`)이며, **실제 학생 답에서의 성능은 재지 않았습니다**. 또한 RuBench v1 `test` 분할(98건)은 열지 않았고, 같은 `dev` 분할(200건) 결과로 `main` 가중치 교체를 결정했으므로 옛 `main` 가중치와의 비교는 독립 검증이 아닙니다.
|
| 193 |
+
- **기준 밖 오류 탐지와 특이도의 상충**: `RuIso-v6`에서 기준 밖 과학 오류 주입 데이터(471행)를 보강하면서 RuBench v1 `dev`의 기준 밖 오류 재현율(TP)은 옛 `main` 대조군 `16/53`(`30.2%`)에서 `42/53`(`79.2%`)으로 높아졌으나, 오류 없는 답안 147건에 대한 특이도(TN)는 옛 `main` 대조군 `137/147`(`93.2%`, 오탐 10건) 및 기본 모델 `146/147`(`99.3%`, 오탐 1건)에서 `111/147`(`75.5%`, 오탐 36건)으로 낮아졌습니다. 학습 분포 옛 `dev` 300건에서도 완전 정답 답안을 틀렸다고 한 오경보(`false_error_on_correct`)가 `0/57`(`0.0%`)에서 `5/57`(`8.8%`)로 늘었습니다.
|
| 194 |
+
- **재현성**: 두 학습 Pod(`NVIDIA RTX A6000 48GB`와 `NVIDIA A40 48GB`)에서 옛 `main` 가중치(`@a0da6701`)를 같은 `vLLM 0.32.0` 조건으로 각각 돌렸을 때 주 지표는 두 Pod 모두 `81.0% [77.5, 84.6]`(짝 차이 `+0.0%p [-0.5, +0.5]`, 기준 미시 `489/604` 동일)이었습니다. 다만 `temperature=0`이어도 `bf16` 배치 순서와 하드웨어에 따라 출력 문자열 전체가 글자 단위로 고정되지는 않습니다.
|
| 195 |
+
- **서빙 후처리(`answer_status`·`next_action` 재계산) 권장**: 잔여 정책 위반(옛 dev 300건 중 20건)은 종합 판정(`answer_status`)이 기준별 판정(`checks`)과 어긋나는 경우(14건), 원문 발췌(`student_span`) 미세 변형(5건), `checks` 순서(1건), 정답 누설(1건)이므로, 서비스에서는 두 값을 모델 출력 대신 아래 규칙으로 다시 계산하는 것을 권장합니다. 시범 서비스(mulk.codes/rubio)는 2026-11-02부터 이렇게 합니다. 이 카드의 다른 수치와 그림은 모두 후처리 없는 모델 원본 출력 기준입니다.
|
| 196 |
- `answer_status`(위에서부터 처음 맞는 것): 모두 `supported`이고 `other_issues`가 없으면 `correct` → `supported`가 있으면 `contradicted`나 `other_issues`�� 있을 때 `mixed`, 아니면 `partial` → `contradicted`나 `other_issues`가 있으면 `incorrect` → `unclear`가 있으면 `unclear` → 나머지 `no_attempt`
|
| 197 |
- `next_action`(위에서부터, 고친 `answer_status` 기준): `reveal_requested`면 `explain` → `correct`면 `complete` → `message_type`이 `other`면 `refocus` → `unclear`면 `clarify` → 나머지 `revise`
|
|
|
|
| 198 |
- **라이선스**: 기본 모델과 동일하게 [Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0)을 따릅니다.
|
| 199 |
|
| 200 |
---
|
|
|
|
| 203 |
|
| 204 |
<img src="./figures/rubio_9b_nature_fig1.svg" width="100%" style="width: 100%; height: auto;" alt="Fig. 3 | Training dynamics, optimization stability, and in-distribution dev set diagnostic performance of RuBio-9B" />
|
| 205 |
|
| 206 |
+
> **Fig. 3 | RuBio-9B (`9b-v6-s3408`) 학습 동역학 및 학습 분포 내 옛 Dev 분할(n = 300) 진단 성능.** **a**, 판정 토큰 가중 손실(`W=10`) 기준 스텝별 학습 손실(원시 및 EMA α = 0.32, 유효 배치 `2 × 8 = 16`, 총 `448`스텝, 1에폭 = `224`스텝, 누적 토큰 `13.77M`)과 50스텝 간격(마지막 step `448` 포함) Dev 평가 손실(step 400 최저 `0.4248`, 교사 강제 판정 정확도 `0.9756` = `97.6%`; `eval_loss`는 `W=10` 가중이 섞여 `RuBio-4B`·`RuBio-8B`·`RuBio-27B`와 직접 비교할 수 없으며 상단 헤더의 `eval_loss*`가 이 뜻임). **b**, 코사인 학습률 스케줄(정점 `10⁻⁴`, 4% 예열 = 20스텝)과 경사 L₂ 노름(‖g‖₂ 중앙값 `0.46`, 사분위 범위 `0.41–0.51`; `RTX A6000 48GB` 약 `1,443 tok/s`, 약 `159분`). **c**, 학습 분포 내 옛 Dev 300건(스키마 없음)에서 4개 규모(`RuBio-4B v2`, `RuBio-8B`, `RuBio-9B`, `RuBio-27B`)의 핵심 지표 6종과 95% Wilson 신뢰구간(CI)으로, 강조된 `RuBio-9B` 값은 `JSON valid` `300/300 = 100.0%`, `Policy compliance` `280/300 = 93.3%`, `Overall status match` `273/300 = 91.0%`, `Next-action match` `287/300 = 95.7%`, `Criterion match` `759/779 = 97.4%`(1건에서 기준 1개 누락으로 분모 779), `Misconception ID` `87/97 = 89.7%`임. **d**, `RuBio-9B`의 옛 Dev 300건 기준별 판정(`checks`) 행 정규화 혼동 행렬(`759/779 = 97.4%` 일치: 정답 `supported` `378/379 = 99.7%`, `contradicted` `163/166 = 98.2%`, `missing` `173/185 = 93.5%`, `unclear` `45/49 = 91.8%`).
|
| 207 |
|
| 208 |
+
### 1. 스텝별 학습 수렴 및 교사 강제 판정 지표 (`RuBio-9B v6 s3408`)
|
| 209 |
|
| 210 |
+
`RuBio-9B`는 스텝별 빠른 점검(quick-probe) 생성을 끄고 학습하여 `training-s3408.json`의 교사 강제 판정 평가(`judge status_acc`, `judge mis_acc`, `judge ce`)를 기록했습니다(`judge mis_acc`는 별도 산출하지 않아 `—`로 표기). `eval_loss`는 판정 토큰 가중(`W=10`)이 포함된 값으로 `RuBio-4B`·`RuBio-8B`·`RuBio-27B`의 `eval_loss`와 직접 비교할 수 없습니다.
|
| 211 |
|
| 212 |
+
| Step | `loss` | `grad_norm` | `lr` | `tokens` | `eval_loss` | `judge status_acc` | `judge mis_acc` | `judge ce` |
|
| 213 |
+
|---:|---:|---:|---:|---:|---:|---:|---:|---:|
|
| 214 |
+
| 5 | 0.8829 | 0.8665 | 1.739e-05 | 155,137 | — | — | — | — |
|
| 215 |
+
| 50 | 0.4378 | 0.2977 | 9.908e-05 | 1,544,711 | 0.5116 | 0.9423 | — | 0.0380 |
|
| 216 |
+
| 100 | 0.4068 | 0.3849 | 9.232e-05 | 3,075,345 | 0.4656 | 0.9718 | — | 0.0229 |
|
| 217 |
+
| 150 | 0.3743 | 0.4155 | 7.984e-05 | 4,617,707 | 0.4492 | 0.9756 | — | 0.0177 |
|
| 218 |
+
| 200 | 0.3736 | 0.6215 | 6.333e-05 | 6,159,720 | 0.4380 | 0.9731 | — | 0.0221 |
|
| 219 |
+
| 225 | 0.3715 | 0.4184 | 5.425e-05 | 6,914,229 | — | — | — | — |
|
| 220 |
+
| 250 | 0.3387 | 0.4580 | 4.502e-05 | 7,682,315 | 0.4328 | 0.9731 | — | 0.0162 |
|
| 221 |
+
| 300 | 0.3228 | 0.4766 | 2.738e-05 | 9,215,246 | 0.4300 | 0.9808 | — | 0.0145 |
|
| 222 |
+
| 350 | 0.3371 | 0.5377 | 1.280e-05 | 10,754,759 | 0.4263 | 0.9769 | — | 0.0158 |
|
| 223 |
+
| **400 (선정)** | **0.2959** | **0.6641** | **3.244e-06** | **12,304,134** | **0.4248** | **0.9756** | **—** | **0.0156** |
|
| 224 |
+
| 445 | 0.3040 | 0.5526 | 2.185e-08 | 13,681,536 | — | — | — | — |
|
| 225 |
+
| 448 (최종 평가) | — | — | — | — | 0.4248 | 0.9756 | — | 0.0151 |
|
| 226 |
|
| 227 |
### 2. 옛 Dev 300건 핵심 평가 지표 비교 (학습과 같은 분포)
|
| 228 |
|
| 229 |
+
2026-11-04 학습 Pod에서 `serve/api_eval.py`로 측정한 옛 `dev` 300건(스키마 없음 · 스키마 강제) 지표와 이전 `main` 가중치(`9b-wt`, `@a0da6701`) 및 다른 규모의 비교입니다. 정책 통과는 `validate_record` 기준입니다.
|
| 230 |
+
|
| 231 |
+
| 평가 지표 | RuBio-4B v2 (`@6f3e415a`, 스키마 없음) | RuBio-8B (`@21e6bf3d`, 스키마 없음) | RuBio-27B v6 (`@c1cfefdb`, 스키마 없음) | 옛 RuBio-9B (`9b-wt`, `@a0da6701`, 스키마 없음) | **RuBio-9B v6 (`@f61ac9cb`, 스키마 없음)** | **RuBio-9B v6 (`@f61ac9cb`, 스키마 강제)** |
|
| 232 |
+
|---|---:|---:|---:|---:|---:|---:|
|
| 233 |
+
| **JSON 유효율** | 300/300 (100.0%) | 299/300 (99.7%) | 300/300 (100.0%) | 300/300 (100.0%) | **300/300 (100.0%)** | **300/300 (100.0%)** |
|
| 234 |
+
| **정책 통과율** (`validate_record`) | 283/300 (94.3%) | 270/300 (90.0%) | 293/300 (97.7%) | 293/300 (97.7%) | **280/300 (93.3%)** | **280/300 (93.3%)** |
|
| 235 |
+
| **종합 판정(`answer_status`) 일치율** | 278/300 (92.7%) | 257/300 (85.7%) | 290/300 (96.7%) | 285/300 (95.0%) | **273/300 (91.0%)** | **273/300 (91.0%)** |
|
| 236 |
+
| **다음 조치(`next_action`) 일치율** | 293/300 (97.7%) | 284/300 (94.7%) | 294/300 (98.0%) | 289/300 (96.3%) | **287/300 (95.7%)** | **287/300 (95.7%)** |
|
| 237 |
+
| **평가 기준별(`checks`) 판정 일치율** | 749/780 (96.0%) | 730/777 (94.0%) | 762/780 (97.7%) | 758/780 (97.2%) | **759/779 (97.4%)** | **759/779 (97.4%)** |
|
| 238 |
+
| **오개념 ID(`misconception_id`) 일치율** | 87/97 (89.7%) | 83/97 (85.6%) | 93/97 (95.9%) | 90/97 (92.8%) | **87/97 (89.7%)** | **87/97 (89.7%)** |
|
| 239 |
+
| **정답을 틀렸다고 함 (`false_error_on_correct`)** | 0/57 (0.0%) | 7/56 (12.5%) | 1/57 (1.8%) | 0/57 (0.0%) | **5/57 (8.8%)** | **5/57 (8.8%)** |
|
| 240 |
+
| **정답 누설 의심** (힌트 단계 행) | 0/213 (0.0%) | 0/213 (0.0%) | 0/213 (0.0%) | 0/213 (0.0%) | **1/213 (0.5%)** | **1/213 (0.5%)** |
|
| 241 |
+
| **최저 Dev 평가 손실 (`best_eval_loss`)** | — | 0.3815 | 0.3686 | 0.4281 (`W=10`)* | **0.4248 (`W=10`)*** | **0.4248 (`W=10`)*** |
|
| 242 |
+
| **응답 지연 p50 / p95 (출력 처리량)** | 10.29s / 13.12s (665.5 tok/s) | 21.08s / 28.30s (400.6 tok/s) | 20.21s / 26.69s (334.0 tok/s) | 6.38s / 8.37s (1,050.5 tok/s) | **16.53s / 21.51s (414.4 tok/s)** | **15.93s / 20.24s (432.4 tok/s)** |
|
| 243 |
+
|
| 244 |
+
### 3. 옛 Dev 300건 정책 위반 종류 비교 (스키마 없음)
|
| 245 |
+
|
| 246 |
+
새 가중치(`RuBio-9B v6`, `@f61ac9cb`)는 옛 `9b-wt`(`@a0da6701`) 대비 옛 `dev` 300건의 기준별 판정 일치율이 `758/780 (97.2%)` → `759/779 (97.4%)`(1개 행 `U1-01-q13-F3`에서 `checks` 항목 `c2`가 빠져 평가된 기준 수가 779개이며, `checks` 순서 위반 1건으로 집계됨)인 반면, **정책 통과율은 `293/300 (97.7%)` → `280/300 (93.3%)`으로 13건 줄었습니다.** 주된 원인은 `RuIso-v6`에서 기준 밖 오류(`other_issues`) 탐지를 보강하면서 `other_issues` 출력이 늘어 종합 판정(`answer_status`) 규칙 불일치가 `5건` → `14건`으로 증가한 것입니다.
|
| 247 |
+
|
| 248 |
+
| 정책 위반 항목 (중복 포함, n = 300) | 옛 RuBio-9B (`9b-wt`, `@a0da6701`) | **RuBio-9B v6 (`9b-v6-s3408`, `@f61ac9cb`)** | RuBio-4B v2 (`@6f3e415a`) | RuBio-8B (`@21e6bf3d`) | RuBio-27B v6 (`@c1cfefdb`) |
|
| 249 |
+
|---|---:|---:|---:|---:|---:|
|
| 250 |
+
| **정책 통과 행 수** | 293/300 (97.7%) | **280/300 (93.3%)** | 283/300 (94.3%) | 270/300 (90.0%) | 293/300 (97.7%) |
|
| 251 |
+
| **정책 위반 행 수 합계** | 7건 | **20건** | 17건 | 30건 | 7건 |
|
| 252 |
+
| `answer_status` 규칙 불일치 | 5건 | **14건** | 15건 | 25건 | 6건 |
|
| 253 |
+
| `student_span` 원문 발췌 불일치 | 3건 | **5건** | 2건 | 3건 | 1건 |
|
| 254 |
+
| `checks` 기준 목록/순서 불일치 | 0건 | **1건** | 0건 | 1건 | 0건 |
|
| 255 |
+
| 힌트 단계 정답 누설 의심 | 0건 | **1건** | 0건 | 0건 | 0건 |
|
| 256 |
+
| JSON 파싱 오류 | 0건 | **0건** | 0건 | 1건 | 0건 |
|
|
|
|
|
|
|
|
|
README_en.md
CHANGED
|
@@ -6,70 +6,95 @@
|
|
| 6 |
|
| 7 |
## 1. Model Summary
|
| 8 |
|
| 9 |
-
`mulkzz/RuBio-9B` is a rubric-aligned formative feedback language model for Korean high-school biology (*Life Science*, 2022 Revised National Curriculum, Units I–III). Given a constructed-response question, grading rubric, textbook evidence, and misconception candidates, it evaluates a student's answer criterion by criterion and outputs both teacher-facing diagnostic metadata and student-facing coaching feedback as a single JSON object. It is a **full 16-bit (`bf16`) merged model** (not a LoRA adapter; 4 `safetensors` shards totaling ~`19.3 GB` / `18 GiB`
|
| 10 |
|
| 11 |
-
|
|
|
|
|
|
|
| 12 |
|
| 13 |
- **Live Demo Service**: https://mulk.codes/rubio (operates `RuBio-4B` and `RuBio-27B` only; `RuBio-9B` is not deployed on the service)
|
| 14 |
-
- **
|
| 15 |
-
- **Larger Model**: [mulkzz/RuBio-27B](https://huggingface.co/mulkzz/RuBio-27B)
|
| 16 |
|
| 17 |
### RuBench v1 Benchmark Evaluation (Out-of-Distribution Dev, n = 200)
|
| 18 |
|
| 19 |
<img src="./figures/rubench_v1_dev200_9b.svg" width="100%" style="width: 100%; height: auto;" alt="Fig. 1 | RuBench v1 (dev n = 200) diagnostic evaluation of RuBio-9B vs. its base model Qwen3.5-9B" />
|
| 20 |
|
| 21 |
-
> **Fig. 1 | Diagnostic evaluation of RuBio-9B (`mulkzz/RuBio-9B@
|
| 22 |
|
| 23 |
<img src="./figures/rubench_v1_dev200_all_9b.svg" width="100%" style="width: 100%; height: auto;" alt="Fig. 2 | RuBench v1 (dev n = 200) criterion accuracy of RuBio, base Qwen models, and general-purpose LLMs" />
|
| 24 |
|
| 25 |
-
> **Fig. 2 | Cross-model diagnostic evaluation on RuBench v1 (dev split, n = 200 answers · 604 criteria) comparing RuBio-9B against its base model, the 4B and 27B pairs, frontier general-purpose LLMs, and the answer-generator reference model (9 models total).** `RuBio-9B` (`
|
| 26 |
|
| 27 |
-
**RuBench v1** is an out-of-distribution (OOD) benchmark for 2022 Revised high-school *Life Science* constructed-response diagnosis (`19` achievement standards across Units I–III, `298` validated question slots split into `dev = 200` and `test = 98`, answered by `30` synthetic student personas stratified across achievement levels A–E; the `test` split was not opened). Every question enforces strict n-gram (`5-gram overlap < 0.40`) and scenario deduplication against the training set; student answers are generated without access to exemplar answers or accepted expressions; and gold labels (`604` criteria, `133` contradicted criteria, and `200` out-of-rubric error indicators in `dev`) are established via a 3-model blind grading panel (`GPT-6 Astra xhigh`, `Opus 5.5 xhigh`, `Grok 4.7 xhigh`), a 2nd-round blind adjudication panel, and human review. All models are evaluated under
|
| 28 |
|
| 29 |
-
| Category | Evaluation Metric (RuBench v1 `dev`) | Sample Size | RuBio-9B (`@
|
| 30 |
|---|---|---:|---:|---:|---|
|
| 31 |
-
| **Primary Metric** | **Criterion Status Accuracy** (answer-clustered mean, 95% CI)<sup>†</sup> | 200 answers (604 criteria) | **
|
| 32 |
-
| | **Criterion Status Accuracy** (excl. R2 2/3 majority items)<sup>‡</sup> | 200 answers (553 criteria) | **
|
| 33 |
-
| | **Criterion Micro-Accuracy** (all `checks` pooled) | 604 criteria | **
|
| 34 |
-
| **Secondary Metrics** | **Misconception Diagnosis** (`contradicted` + exact ID match) | 133 criteria | **
|
| 35 |
-
| | **Out-of-Rubric Error (`other_issues`) Match** | 200 answers |
|
| 36 |
-
| | **Overall Status (`answer_status`) Rule Match** | 200 answers | **
|
| 37 |
-
| **By Gold Tier** | Round 1 Unanimous (`r1-3/3`) | 521 criteria | **
|
| 38 |
-
| | Round 2 Unanimous (`r2-3/3`) | 23 criteria | **
|
| 39 |
-
| | Round 2 Majority (`r2-2/3`, borderline items) | 51 criteria | **
|
| 40 |
| | Human Adjudicated (`human`) | 9 criteria | **6/9 (66.7%)** | 3/9 (33.3%) | +33.3%p (3-way panel split items) |
|
| 41 |
-
| **By Curriculum Unit** | I. Understanding Life Science (`checks` micro) | 206 criteria | **
|
| 42 |
-
| | II. Human Metabolism (`checks` micro) | 200 criteria | **
|
| 43 |
-
| | III. Homeostasis & Body Regulation (`checks` micro) | 198 criteria | **
|
| 44 |
-
| **Format Stability** | **Valid JSON Output Rate** (no schema enforcement) | 200 answers |
|
| 45 |
|
| 46 |
<sup>†</sup> **Decision rule fixed at benchmark design**: A model is declared **Improved** if the lower bound of the 95% answer-clustered paired confidence interval on criterion status accuracy exceeds `0`, **Worse** if the upper bound is below `0`, and **No difference** if the interval spans `0` (Miller 2024, arXiv:2411.00640).<br/>
|
| 47 |
<sup>‡</sup> Sensitivity analysis excluding the 51 criteria resolved by 2/3 majority vote in Round 2 (`r2-2/3`).<br/>
|
| 48 |
-
<sup>§</sup> **Evaluation conditions & serving path**:
|
| 49 |
-
|
| 50 |
-
###
|
| 51 |
-
|
| 52 |
-
- **Why
|
| 53 |
-
- **
|
| 54 |
-
|
| 55 |
-
|
| 56 |
-
|
| 57 |
-
|
| 58 |
-
|
| 59 |
-
|
| 60 |
-
|
| 61 |
-
|
| 62 |
-
-
|
| 63 |
-
|
| 64 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 65 |
|
| 66 |
## 2. Training Configuration
|
| 67 |
|
| 68 |
-
- **Base Model & Template Verification**: Base model is [Qwen/Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B) (Apache-2.0). Training loaded `unsloth/Qwen3.5-9B`, whose weight `sha256` digests match the official `Qwen/Qwen3.5-9B` release
|
| 69 |
-
- **
|
| 70 |
-
- **
|
| 71 |
-
- **
|
| 72 |
-
- **
|
|
|
|
|
|
|
|
|
|
|
|
|
| 73 |
|
| 74 |
## 3. Input & Output Specification
|
| 75 |
|
|
@@ -124,7 +149,7 @@ user 메시지는 JSON이다: 문항(question)·자료(materials)·평가 기준
|
|
| 124 |
|
| 125 |
## 4. Serving & Inference
|
| 126 |
|
| 127 |
-
The merged `bf16` weights (4 `safetensors` shards) total ~`19.3 GB` (`18 GiB`). The verified serving
|
| 128 |
|
| 129 |
```bash
|
| 130 |
vllm serve mulkzz/RuBio-9B --served-model-name bio-feedback \
|
|
@@ -152,12 +177,12 @@ result = json.loads(response.choices[0].message.content)
|
|
| 152 |
|
| 153 |
## 5. Limitations & License
|
| 154 |
|
| 155 |
-
- **Scope & Limitations**: Designed as a formative learning aid; do not use for autonomous high-stakes summative grading. Not intended for general QA outside Units I–III of the 2022 Revised *Life Science* curriculum or inputs lacking structured `rubric` and `evidence` fields.
|
| 156 |
-
- **
|
| 157 |
-
- **
|
|
|
|
| 158 |
- `answer_status` (first match wins): all `supported` and no `other_issues` → `correct`; otherwise if any `supported` → `mixed` when `contradicted` or `other_issues` is present, else `partial`; otherwise `contradicted` or `other_issues` → `incorrect`; otherwise any `unclear` → `unclear`; else `no_attempt`
|
| 159 |
- `next_action` (first match wins, using the recomputed `answer_status`): `reveal_requested` → `explain`; `correct` → `complete`; `message_type` = `other` → `refocus`; `unclear` → `clarify`; else `revise`
|
| 160 |
-
- Re-scoring the 300 old dev outputs (no-schema): `answer_status` match 285→291, `next_action` match 289→290, policy compliance 293→297.
|
| 161 |
- **License**: Released under [Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0), matching the base model.
|
| 162 |
|
| 163 |
---
|
|
@@ -166,52 +191,54 @@ result = json.loads(response.choices[0].message.content)
|
|
| 166 |
|
| 167 |
<img src="./figures/rubio_9b_nature_fig1.svg" width="100%" style="width: 100%; height: auto;" alt="Fig. 3 | Training dynamics, optimization stability, and in-distribution dev set diagnostic performance of RuBio-9B" />
|
| 168 |
|
| 169 |
-
> **Fig. 3 | Training dynamics and in-distribution old Dev split (n = 300) diagnostic performance of RuBio-9B.** **a**, Step-wise judge-weighted (`W=10`) training loss (raw and EMA α = 0.32, effective batch `2 × 8 = 16`, cumulative tokens `
|
| 170 |
|
| 171 |
-
### 1. Step-by-Step Training Convergence & Teacher-Forced Judge Metrics
|
| 172 |
|
| 173 |
-
Because quick-probe generation was disabled during training, the step table reports teacher-forced judge evaluation metrics (`
|
| 174 |
|
| 175 |
-
|
|
| 176 |
-
|---:|---:|---:|---:|---:|---:|---:|---:|
|
| 177 |
-
|
|
| 178 |
-
|
|
| 179 |
-
|
|
| 180 |
-
|
|
| 181 |
-
|
|
| 182 |
-
|
|
| 183 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 184 |
|
| 185 |
### 2. In-Distribution Old Dev 300 Core Evaluation Metrics
|
| 186 |
|
| 187 |
-
|
| 188 |
-
|
| 189 |
-
|
|
| 190 |
-
|
|
| 191 |
-
| **
|
| 192 |
-
| **
|
| 193 |
-
| **
|
| 194 |
-
| **
|
| 195 |
-
| **
|
| 196 |
-
| **
|
| 197 |
-
| **
|
| 198 |
-
| **
|
| 199 |
-
|
| 200 |
-
|
| 201 |
-
|
| 202 |
-
### 3.
|
| 203 |
-
|
| 204 |
-
|
| 205 |
-
|
| 206 |
-
|
|
| 207 |
-
|
|
| 208 |
-
|
|
| 209 |
-
| **
|
| 210 |
-
|
|
| 211 |
-
|
|
| 212 |
-
|
|
| 213 |
-
|
|
| 214 |
-
|
|
| 215 |
-
| **Residual Policy Violations** (with overlap) | `answer_status` mismatch | 300 | **5** | Resolved by post-processing |
|
| 216 |
-
| | `student_span` variation | 300 | **3** | Minor span inflection differences |
|
| 217 |
-
| | Invalid criterion status value | 300 | **0** | All outputs use valid canonical status values |
|
|
|
|
| 6 |
|
| 7 |
## 1. Model Summary
|
| 8 |
|
| 9 |
+
`mulkzz/RuBio-9B` is a rubric-aligned formative feedback language model for Korean high-school biology (*Life Science*, 2022 Revised National Curriculum, Units I–III). Given a constructed-response question, grading rubric, textbook evidence, and misconception candidates, it evaluates a student's answer criterion by criterion and outputs both teacher-facing diagnostic metadata and student-facing coaching feedback as a single JSON object. It is a **full 16-bit (`bf16`) merged model** (not a LoRA adapter; 4 `safetensors` shards totaling ~`19.3 GB` / `18 GiB`) fine-tuned with LoRA on [Qwen/Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B), positioned between [mulkzz/RuBio-4B](https://huggingface.co/mulkzz/RuBio-4B) / [mulkzz/RuBio-8B](https://huggingface.co/mulkzz/RuBio-8B) and the larger [mulkzz/RuBio-27B](https://huggingface.co/mulkzz/RuBio-27B).
|
| 10 |
|
| 11 |
+
**Starting 2026-11-04, the `main` branch weights have been replaced with the `RuIso-v6` retrained model `9b-v6-s3408` (seed `3408`, revision `f61ac9cb`, identical to the `ruiso-v6` branch).** The previous `main` weights (`9b-wt`, seed `3407`, initial upload commit `e19404e3`) remain accessible via `revision="a0da6701"`.
|
| 12 |
+
|
| 13 |
+
While sharing the same output schema and `system` prompt as `RuBio-4B`, `RuBio-8B`, and `RuBio-27B`, **its training method differs from 4B and 27B in one respect**: **judge-token weighted loss (`W=10`)**, which counts per-criterion `status` and `misconception_id` value tokens 10 times in the cross-entropy loss (`RuBio-8B` also uses standard unweighted loss `W=1`). Because `eval_loss` incorporates this `W=10` weighting, it cannot be directly compared with the unweighted `eval_loss` of `RuBio-4B`, `RuBio-8B`, or `RuBio-27B`; the asterisk in `eval_loss*` (`0.4248`) in the header image denotes this weighted loss. The live demo service (https://mulk.codes/rubio) operates only `RuBio-4B` and `RuBio-27B`; `RuBio-9B` has not been deployed to the demo service.
|
| 14 |
|
| 15 |
- **Live Demo Service**: https://mulk.codes/rubio (operates `RuBio-4B` and `RuBio-27B` only; `RuBio-9B` is not deployed on the service)
|
| 16 |
+
- **Related Models**: [mulkzz/RuBio-4B](https://huggingface.co/mulkzz/RuBio-4B) · [mulkzz/RuBio-8B](https://huggingface.co/mulkzz/RuBio-8B) · [mulkzz/RuBio-27B](https://huggingface.co/mulkzz/RuBio-27B)
|
|
|
|
| 17 |
|
| 18 |
### RuBench v1 Benchmark Evaluation (Out-of-Distribution Dev, n = 200)
|
| 19 |
|
| 20 |
<img src="./figures/rubench_v1_dev200_9b.svg" width="100%" style="width: 100%; height: auto;" alt="Fig. 1 | RuBench v1 (dev n = 200) diagnostic evaluation of RuBio-9B vs. its base model Qwen3.5-9B" />
|
| 21 |
|
| 22 |
+
> **Fig. 1 | Diagnostic evaluation of RuBio-9B (`mulkzz/RuBio-9B@f61ac9cb`, `9b-v6-s3408`) vs. its base model Qwen3.5-9B (`Qwen/Qwen3.5-9B`) on RuBench v1 (dev split, n = 200 answers · 604 criteria).** **a**, Five core diagnostic metrics and 95% confidence intervals (answer-clustered CI and Wilson CI). On the primary criterion status accuracy metric, `RuBio-9B` (`84.9% [81.6, 88.1]`) vs. `Qwen3.5-9B` (`76.6% [73.0, 80.2]`) yields a paired Δ of `+8.2%p [+4.9, +11.6]` (Improved), and excluding R2 2/3 majority items yields `87.7% [84.6, 90.8]` vs. `81.4% [77.9, 84.8]`, paired Δ = `+6.3%p [+2.7, +10.0]` (Improved). Secondary metrics are `111/133 = 83.5%` vs. `86/133 = 64.7%` for misconception match, `153/200 = 76.5%` vs. `153/200 = 76.5%` for out-of-rubric error match, and `144/200 = 72.0%` vs. `96/200 = 48.0%` for overall status match. Dotted line marks `RuBio-9B`'s in-distribution old dev (`n = 300`) criterion match (`759/779 = 97.4%`). **b**, Criterion accuracy stratified by gold consensus source (`R1 3/3` `467/521 = 89.6%` vs. `440/521 = 84.5%`, `R2 3/3` `13/23 = 56.5%` vs. `7/23 = 30.4%`, `R2 2/3` `26/51 = 51.0%` vs. `13/51 = 25.5%`) alongside the out-of-rubric scientific error (`other_issues`) sensitivity–specificity trade-off, where `RuBio-9B` achieves higher error-present recall (TP `42/53 = 79.2%` vs. `7/53 = 13.2%`) but lower error-free specificity (TN `111/147 = 75.5%`, 36 false positives vs. `146/147 = 99.3%`, 1 false positive). **c, d**, Row-normalized per-criterion (`checks`) confusion matrices for fine-tuned `RuBio-9B` (criterion micro `512/604 = 84.8%`, invalid JSON `1/200`) and base `Qwen3.5-9B` (criterion micro `463/604 = 76.7%`, invalid JSON `0/200`), where the `invalid/none` column denotes malformed JSON or missing criterion decisions (`RuBio-9B` has 1 truncated output with `finish_reason=length` accounting for 3 criteria: 1 gold `contradicted` and 2 gold `unclear`). Both models are evaluated with `vLLM 0.32.0` (`bf16`; `RuBio-9B` on a `RunPod RTX A6000 48GB Pod`, `Qwen3.5-9B` on a `RunPod A40 Pod`) under identical first-turn, `temperature=0`, `max_tokens=1024`, thinking-disabled, and unconstrained JSON settings (malformed JSON counted as wrong), except `RuBio-9B` receives only the training `SYSTEM_PROMPT` whereas base `Qwen3.5-9B` receives `SYSTEM_PROMPT` augmented with the grading and response policy (`POLICY`).
|
| 23 |
|
| 24 |
<img src="./figures/rubench_v1_dev200_all_9b.svg" width="100%" style="width: 100%; height: auto;" alt="Fig. 2 | RuBench v1 (dev n = 200) criterion accuracy of RuBio, base Qwen models, and general-purpose LLMs" />
|
| 25 |
|
| 26 |
+
> **Fig. 2 | Cross-model diagnostic evaluation on RuBench v1 (dev split, n = 200 answers · 604 criteria) comparing RuBio-9B against its base model, the 4B and 27B pairs, frontier general-purpose LLMs, and the answer-generator reference model (9 models total).** `RuBio-9B` (`84.9% [81.6, 88.1]`, or `87.7% [84.6, 90.8]` excluding R2 2/3 majority items) scores higher than `RuBio-4B v2` (`77.2% [73.4, 80.9]`) and base `Qwen3.5-9B` (`76.6% [73.0, 80.2]`), shows no statistically significant difference from `RuBio-27B` (`c1cfefdb`, `86.9% [83.8, 89.9]`), and scores lower than frontier general-purpose models (`GPT-6 Astra` `91.2% [88.6, 93.7]`, `Claude Opus 5.5` `94.2% [92.0, 96.4]`). **a**, Criterion status accuracy (answer-clustered 95% CI, bars, `n=604`) and accuracy excluding R2 2/3 majority items (open diamonds, `n=553`) across nine models in order: the 4B pair (`RuBio-4B v2` `77.2% [73.4, 80.9]`, excl. R2 2/3 `79.9% [76.0, 83.8]`, criterion micro `467/604`; `Qwen3.5-4B` `70.9% [67.3, 74.5]`, excl. `74.5% [71.0, 78.1]`, micro `427/604`), the 9B pair (`RuBio-9B` `84.9% [81.6, 88.1]`, excl. `87.7% [84.6, 90.8]`, micro `512/604`; `Qwen3.5-9B` `76.6% [73.0, 80.2]`, excl. `81.4% [77.9, 84.8]`, micro `463/604`), the 27B pair (`RuBio-27B` `86.9% [83.8, 89.9]`, excl. `90.0% [87.1, 92.9]`, micro `524/604`; `Qwen3.8-27B` `81.8% [78.2, 85.4]`, excl. `85.0% [81.4, 88.5]`, micro `495/604`), general-purpose models (`GPT-6 Astra (xhigh)`† `91.2% [88.6, 93.7]`, excl. `95.6% [93.6, 97.5]`, micro `550/604`; `Claude Opus 5.5 (xhigh)`† `94.2% [92.0, 96.4]`, excl. `96.1% [94.2, 98.0]`, micro `570/604`), and the reference model (`Gemini 4 Argon (High)`, hatched, `91.2% [88.5, 94.0]`, excl. `93.6% [91.0, 96.2]`, micro `551/604`). **b**, Answer-clustered paired differences (%p, 95% CI; lower bound > 0 indicates improvement, upper bound < 0 degradation, otherwise no difference) for all criteria (filled circles) and excluding R2 2/3 majority items (open circles). `RuBio-9B − Qwen3.5-9B` is `+8.2%p [+4.9, +11.6]` all and `+6.3%p [+2.7, +10.0]` excl. R2 2/3 (Improved); `RuBio-4B − Qwen3.5-4B` is `+6.3%p [+2.6, +9.9]` all and `+5.3%p [+1.2, +9.5]` excl. R2 2/3 (Improved); `RuBio-27B − Qwen3.8-27B` is `+5.1%p [+1.1, +9.1]` all and `+5.0%p [+1.1, +9.0]` excl. R2 2/3 (Improved). Across model scales, `RuBio-9B − RuBio-4B` is `+7.7%p [+4.7, +10.8]` all and `+7.8%p [+4.3, +11.3]` excl. R2 2/3 (Improved), whereas `RuBio-27B − RuBio-9B` is `+2.0%p [-1.4, +5.4]` all and `+2.3%p [-1.1, +5.7]` excl. R2 2/3 (No difference). Against frontier models, `GPT − RuBio-9B` (`+6.3%p [+3.1, +9.5]` all; `+7.9%p [+4.9, +10.9]` excl. R2 2/3) and `Opus − RuBio-9B` (`+9.3%p [+6.2, +12.5]` all; `+8.4%p [+5.6, +11.2]` excl. R2 2/3) are improved in both settings, while `Opus − GPT` is improved overall (`+3.0%p [+0.4, +5.6]`) but shows no difference excluding R2 2/3 (`+0.5%p [-1.5, +2.5]`). The gray auxiliary row `RuBio-9B − Qwen3.5-9B, same prompt*` is a non-official reference comparison where base `Qwen3.5-9B` receives only `SYSTEM_PROMPT`: it outputs `checks` as a `{criterion_id: status}` dictionary on 189/200 answers (strict score `3.8% [1.3, 6.2]`, excl. R2 2/3 `4.2% [1.6, 6.8]`, micro `21/604`), and converting that dictionary to a list yields a lenient score of `76.0% [72.3, 79.7]` (excl. R2 2/3 `80.1% [76.4, 83.7]`, micro `458/604`), giving a paired difference of `+8.9%p [+5.1, +12.6]` all and `+7.6%p [+3.6, +11.7]` excl. R2 2/3 (Improved). **c**, Secondary-metric heatmap across the nine models (`RuBio-4B v2`, `Qwen3.5-4B`, `RuBio-9B`, `Qwen3.5-9B`, `RuBio-27B`, `Qwen3.8-27B`, `GPT`, `Opus`, `Gemini`): misconception ID match (`133` criteria: `106/133 = 79.7%`, `78/133 = 58.6%`, `111/133 = 83.5%`, `86/133 = 64.7%`, `118/133 = 88.7%`, `107/133 = 80.5%`, `129/133 = 97.0%`, `127/133 = 95.5%`, `122/133 = 91.7%`), out-of-rubric error recall (TP, `53` answers: `10/53 = 18.9%`, `4/53 = 7.5%`, `42/53 = 79.2%`, `7/53 = 13.2%`, `46/53 = 86.8%`, `29/53 = 54.7%`, `33/53 = 62.3%`, `52/53 = 98.1%`, `45/53 = 84.9%`), error-free specificity (TN, `147` answers: `143/147 = 97.3%`, `146/147 = 99.3%`, `111/147 = 75.5%`, `146/147 = 99.3%`, `127/147 = 86.4%`, `127/147 = 86.4%`, `144/147 = 98.0%`, `143/147 = 97.3%`, `145/147 = 98.6%`), `answer_status` rule agreement (`200` answers: `124/200 = 62.0%`, `89/200 = 44.5%`, `144/200 = 72.0%`, `96/200 = 48.0%`, `154/200 = 77.0%`, `120/200 = 60.0%`, `176/200 = 88.0%`, `184/200 = 92.0%`, `179/200 = 89.5%`), and valid JSON rate (`199/200 = 99.5%` for `RuBio-9B`, `195/200 = 97.5%` for `Qwen3.8-27B`, `198/200 = 99.0%` for `Gemini`, and `200/200 = 100.0%` for the other six models). **d**, Criterion accuracy and Wilson 95% CIs by gold consensus source in the same nine-model order: `R1 3/3` (`521` criteria: `431/521 = 82.7%`, `402/521 = 77.2%`, `467/521 = 89.6%`, `440/521 = 84.5%`, `483/521 = 92.7%`, `461/521 = 88.5%`, `510/521 = 97.9%`, `510/521 = 97.9%`, `496/521 = 95.2%`), `R2 3/3` (`23` criteria: `10/23 = 43.5%`, `4/23 = 17.4%`, `13/23 = 56.5%`, `7/23 = 30.4%`, `11/23 = 47.8%`, `5/23 = 21.7%`, `14/23 = 60.9%`, `18/23 = 78.3%`, `19/23 = 82.6%`), `R2 2/3` (`51` criteria: `23/51 = 45.1%`, `15/51 = 29.4%`, `26/51 = 51.0%`, `13/51 = 25.5%`, `23/51 = 45.1%`, `24/51 = 47.1%`, `22/51 = 43.1%`, `37/51 = 72.5%`, `32/51 = 62.7%`), and `human` adjudication (`9` criteria: `3/9 = 33.3%`, `6/9 = 66.7%`, `6/9 = 66.7%`, `3/9 = 33.3%`, `7/9 = 77.8%`, `5/9 = 55.6%`, `4/9 = 44.4%`, `5/9 = 55.6%`, `4/9 = 44.4%`). †`GPT-6 Astra (xhigh)` and `Opus 5.5 (xhigh)` served on the Round-1 gold panel (`GPT-6 Astra xhigh`, `Opus 5.5 xhigh`, `Grok 4.7 xhigh`), so the `R1 3/3` tier (`521` of `604` criteria) contains their own blind judgments and their scores represent upper bounds that include self-agreement; Round-2 majority (`R2 2/3`) items also include votes from the Round-2 panel (`GPT`, `Opus`, `Gemini`). `Gemini 4 Argon (High)` wrote the synthetic student answers with intended labels and is reported as a reference only and excluded from comparative testing (raw difference `Gemini − RuBio-9B`: `+6.4%p [+2.6, +10.2]` all, `+5.9%p [+2.4, +9.3]` excl. R2 2/3); Round-1 panelist `Grok 4.7` was not evaluated as a model. All models are evaluated on the first turn (`feedback_mode="hint_1"`, `reveal_requested=false`, `history=[]`) without JSON schema enforcement (malformed JSON counted as wrong), with `SYSTEM_PROMPT` only for `RuBio` and `SYSTEM_PROMPT+POLICY` for base and general-purpose models; `RuBio` and base `Qwen` models run at `temperature=0`, `max_tokens=1024`, and thinking disabled, whereas general-purpose models use a single fixed vendor CLI run at the listed reasoning effort (median latency: `GPT` `575.6 s`, `Opus` `41.3 s`, `Gemini` `87.9 s`).
|
| 27 |
|
| 28 |
+
**RuBench v1** is an out-of-distribution (OOD) benchmark for 2022 Revised high-school *Life Science* constructed-response diagnosis (`19` achievement standards across Units I–III, `298` validated question slots split into `dev = 200` and `test = 98`, answered by `30` synthetic student personas stratified across achievement levels A–E; the `test` split was not opened). Every question enforces strict n-gram (`5-gram overlap < 0.40`) and scenario deduplication against the training set; student answers are generated without access to exemplar answers or accepted expressions; and gold labels (`604` criteria, `133` contradicted criteria, and `200` out-of-rubric error indicators in `dev`) are established via a 3-model blind grading panel (`GPT-6 Astra xhigh`, `Opus 5.5 xhigh`, `Grok 4.7 xhigh`), a 2nd-round blind adjudication panel, and human review. All models are evaluated under identical first-turn generation settings (`feedback_mode="hint_1"`, `reveal_requested=false`, `history=[]`, `temperature=0`, `max_tokens=1024`, `enable_thinking=false`, **no JSON schema enforcement**, malformed JSON counted as wrong).
|
| 29 |
|
| 30 |
+
| Category | Evaluation Metric (RuBench v1 `dev`) | Sample Size | RuBio-9B (`@f61ac9cb`) | Qwen3.5-9B (`Qwen/Qwen3.5-9B`)<sup>§</sup> | Paired Δ (fine-tuned − base) & Notes |
|
| 31 |
|---|---|---:|---:|---:|---|
|
| 32 |
+
| **Primary Metric** | **Criterion Status Accuracy** (answer-clustered mean, 95% CI)<sup>†</sup> | 200 answers (604 criteria) | **84.9%** `[81.6, 88.1]` | 76.6% `[73.0, 80.2]` | +8.2%p `[+4.9, +11.6]` (**Improved**) |
|
| 33 |
+
| | **Criterion Status Accuracy** (excl. R2 2/3 majority items)<sup>‡</sup> | 200 answers (553 criteria) | **87.7%** `[84.6, 90.8]` | 81.4% `[77.9, 84.8]` | +6.3%p `[+2.7, +10.0]` (**Improved**) |
|
| 34 |
+
| | **Criterion Micro-Accuracy** (all `checks` pooled) | 604 criteria | **512/604 (84.8%)** | 463/604 (76.7%) | +8.1%p (+49 criteria) |
|
| 35 |
+
| **Secondary Metrics** | **Misconception Diagnosis** (`contradicted` + exact ID match) | 133 criteria | **111/133 (83.5%)** | 86/133 (64.7%) | +18.8%p (non-null ID `n=79`: **65/79 (82.3%)** vs 62/79 (78.5%)) |
|
| 36 |
+
| | **Out-of-Rubric Error (`other_issues`) Match** | 200 answers | **153/200 (76.5%)** | **153/200 (76.5%)** | 0.0%p (Error recall TP `n=53`: **42/53 (79.2%)** vs 7/53 (13.2%) / No-error TN `n=147`: 111/147 (75.5%, 36 FP) vs **146/147 (99.3%, 1 FP)**) |
|
| 37 |
+
| | **Overall Status (`answer_status`) Rule Match** | 200 answers | **144/200 (72.0%)** | 96/200 (48.0%) | +24.0%p (+48 answers) |
|
| 38 |
+
| **By Gold Tier** | Round 1 Unanimous (`r1-3/3`) | 521 criteria | **467/521 (89.6%)** | 440/521 (84.5%) | +5.2%p |
|
| 39 |
+
| | Round 2 Unanimous (`r2-3/3`) | 23 criteria | **13/23 (56.5%)** | 7/23 (30.4%) | +26.1%p |
|
| 40 |
+
| | Round 2 Majority (`r2-2/3`, borderline items) | 51 criteria | **26/51 (51.0%)** | 13/51 (25.5%) | +25.5%p (inherently ambiguous panel-split items) |
|
| 41 |
| | Human Adjudicated (`human`) | 9 criteria | **6/9 (66.7%)** | 3/9 (33.3%) | +33.3%p (3-way panel split items) |
|
| 42 |
+
| **By Curriculum Unit** | I. Understanding Life Science (`checks` micro) | 206 criteria | **170/206 (82.5%)** | 152/206 (73.8%) | +8.7%p |
|
| 43 |
+
| | II. Human Metabolism (`checks` micro) | 200 criteria | **171/200 (85.5%)** | 156/200 (78.0%) | +7.5%p |
|
| 44 |
+
| | III. Homeostasis & Body Regulation (`checks` micro) | 198 criteria | **171/198 (86.4%)** | 155/198 (78.3%) | +8.1%p |
|
| 45 |
+
| **Format Stability** | **Valid JSON Output Rate** (no schema enforcement) | 200 answers | 199/200 (99.5%) | **200/200 (100.0%)** | -0.5%p (1 JSON parse error `finish_reason=length` vs **0**) |
|
| 46 |
|
| 47 |
<sup>†</sup> **Decision rule fixed at benchmark design**: A model is declared **Improved** if the lower bound of the 95% answer-clustered paired confidence interval on criterion status accuracy exceeds `0`, **Worse** if the upper bound is below `0`, and **No difference** if the interval spans `0` (Miller 2024, arXiv:2411.00640).<br/>
|
| 48 |
<sup>‡</sup> Sensitivity analysis excluding the 51 criteria resolved by 2/3 majority vote in Round 2 (`r2-2/3`).<br/>
|
| 49 |
+
<sup>§</sup> **Evaluation conditions & serving path**: `RuBio-9B` (`9b-v6-s3408`, `@f61ac9cb`) was evaluated on its training Pod (`RunPod RTX A6000 48GB`) with `vLLM 0.32.0` (`bf16`, `--language-model-only --max-model-len 8192 --max-num-seqs 32`), and base `Qwen3.5-9B` was evaluated on a `RunPod A40 Pod` with `vLLM 0.32.0` (`bf16`), under identical first-turn, `temperature=0`, `max_tokens=1024`, thinking-disabled, and unconstrained JSON settings (malformed JSON counted as incorrect), except `RuBio-9B` receives only `SYSTEM_PROMPT` whereas base `Qwen3.5-9B` receives `SYSTEM_PROMPT` plus `POLICY`. In Fig. 2, `RuBio-27B` also uses its new `RuIso-v6` `main` weights (`27b-v6-s3407`, `@c1cfefdb`).
|
| 50 |
+
|
| 51 |
+
### RuIso-v6 Retraining & Main Weight Replacement
|
| 52 |
+
|
| 53 |
+
- **Why We Retrained**: Minimal-pair diagnostics revealed that the previous `RuBio` models treated hedged student phrasing ("~인 것 같아요" / "I think it might be...") and compressed term-listing style as error signals, and tended to penalize satisfied rubric criteria when an answer contained an out-of-rubric error.
|
| 54 |
+
- **RuIso-v6 Data Augmentation (`train = 3,579` rows, `dev = 300` rows)**: We added `1,179` new synthetic rows to the existing `2,400` training rows (`train = 3,579` rows total; `dev = 300` rows unchanged).
|
| 55 |
+
- **Style-only variants (`708` rows)**: Rewrites of first-turn student answers from the same question families that preserve underlying content, errors, and omissions while altering only writing style (`235` hedged phrasing, `236` compressed term-listing, `237` brief 1–2 sentence answers; per-criterion labels were fixed from the source rows and verified by blind re-grading).
|
| 56 |
+
- **Out-of-rubric scientific error injection (`471` rows)**: Correct or partially correct answers into which a single clear scientific error unrelated to the rubric was injected (per-criterion judgments fixed from the source row, only `other_issues` added, and `answer_status` recomputed by rule).
|
| 57 |
+
- **Synthesis & Filtering Pipeline**: Student answer rewrites were generated by `Claude Opus 5.5 (xhigh)`, while blind grading and feedback writing were performed by `Gemini 4 Argon (High)`. Out of `1,200` planned rows, rows where fixed labels and blind labels disagreed were discarded or routed to a review queue, leaving `1,179` rows. RuBench questions/answers and minimal-pair diagnostic variants were excluded from training. All student answers are synthetic, not real student data.
|
| 58 |
+
- **Rule Fixed Before Viewing the Results**:
|
| 59 |
+
1. For model scales trained with two seeds, select the seed with the lower `best_eval_loss` as the release candidate prior to RuBench scoring (for `RuBio-9B`, seed `3408` achieved `0.4248` < seed `3407`'s `0.4262`, selecting `9b-v6-s3408`).
|
| 60 |
+
2. Compare the selected candidate against the control (the previous HF `main` pinned at `revision="a0da6701"`, re-evaluated on the same Pod and same `vLLM 0.32.0`): replace `main` weights if the RuBench v1 `dev` 200 primary metric (criterion status accuracy) paired difference point estimate is `> 0` AND recall on gold `missing` and gold `contradicted` criteria are each `>=` the control; otherwise keep the model on the `ruiso-v6` branch only. Because this replacement rule does not require the 95% CI lower bound to exceed `0`, if a model is replaced while the 95% CI spans `0`, that fact must be stated explicitly (for `RuBio-9B`, the paired difference is `+3.8%p [+1.3, +6.3]`, so the 95% CI lower bound also exceeds `0`).
|
| 61 |
+
|
| 62 |
+
| Model Key | Model & Execution Environment | Primary Metric (All, 95% CI) | Excl. R2 2/3 | Criterion Micro | `supported` (283) | `contradicted` (133) | `missing` (142) | `unclear` (46) | Out-of-Rubric TP/FP (53 err) | `answer_status` (200) | Invalid JSON |
|
| 63 |
+
|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
|
| 64 |
+
| **`9b-v6-s3408` (Selected `main`)** | `RuBio-9B RuIso-v6` (seed `3408`, `@f61ac9cb`, RTX A6000 Pod) | **84.9%** `[81.6, 88.1]` | 87.7% | **512/604** | **263/283 (92.9%)** | **124/133 (93.2%)** | 102/142 (71.8%) | 23/46 (50.0%) | **42** / 36 | **144/200** | 1 |
|
| 65 |
+
| `9b-pub-pod-p3408` (Control) | `RuBio-9B Public` (seed `3408` Pod, `@a0da6701`) | 81.0% `[77.5, 84.6]` | 83.7% | 489/604 | 240/283 (84.8%) | 122/133 (91.7%) | 101/142 (71.1%) | **26/46 (56.5%)** | 16 / **10** | 131/200 | **0** |
|
| 66 |
+
| `9b-v6-s3407` | `RuBio-9B RuIso-v6` (seed `3407`, `local rubio-9b-v6-s3407`, A40 Pod) | **84.9%** `[81.7, 88.1]` | **88.0%** | 511/604 | 261/283 (92.2%) | 121/133 (91.0%) | **110/142 (77.5%)** | 19/46 (41.3%) | 41 / 41 | 134/200 | **0** |
|
| 67 |
+
| `9b-pub-pod-p3407` (Control) | `RuBio-9B Public` (seed `3407` Pod, `@a0da6701`) | 81.0% `[77.5, 84.6]` | 83.7% | 489/604 | 239/283 (84.5%) | 122/133 (91.7%) | 102/142 (71.8%) | **26/46 (56.5%)** | 17 / **10** | 131/200 | **0** |
|
| 68 |
+
|
| 69 |
+
- **Paired Differences vs. Same-Pod Control & Cross-Pod Reproducibility**: Against the same-Pod (`RTX A6000 48GB`) control run of the previous `main` (`9b-pub-pod-p3408`), `9b-v6-s3408` achieves a paired difference of `+3.8%p [+1.3, +6.3]` overall (**Improved**) and `+4.0%p [+1.2, +6.8]` excluding R2 2/3 (**Improved**). The seed `3407` retrain (`9b-v6-s3407`) likewise achieves `+3.8%p [+1.0, +6.7]` overall (**Improved**) and `+4.3%p [+1.2, +7.4]` excluding R2 2/3 (**Improved**). Comparing the two control runs of `@a0da6701` across the two Pods (`9b-pub-pod-p3407` on `A40 48GB` minus `9b-pub-pod-p3408` on `RTX A6000 48GB`) yields `+0.0%p [-0.5, +0.5]` both overall and excluding R2 2/3 (**No difference**, with identical micro-accuracy `489/604`), confirming cross-Pod reproducibility.
|
| 70 |
+
- **Minimal-Pair Diagnostics (A–E)**: The table below tracks how many gold `supported` criteria remain `supported` before and after style transformations (`orig→var supported / criteria (+gain −loss)`; ideally all remain `supported`; interpret as directional trends only due to small sample sizes). Whereas the previous `main` control scored `45/74` (`p3408`) and `44/74` (`p3407`) on hedged originals in A and jumped to `60/74` (`+16 −1` and `+17 −1`) when hedging was removed, `9b-v6-s3408` already marks `61/74` as `supported` on the hedged originals vs. `64/74` (`+4 −1`) after removing hedging.
|
| 71 |
+
|
| 72 |
+
| Minimal-Pair Transformation (gold `supported` criteria only) | Sample Size | `9b-v6-s3408` (Selected `main`) | `9b-pub-pod-p3408` (Old `main` Control) | `9b-v6-s3407` (Seed 3407) | `9b-pub-pod-p3407` (Old `main` Control) |
|
| 73 |
+
|---|---:|---:|---:|---:|---:|
|
| 74 |
+
| **A. Remove hedging** ("~인 것 같아요" → declarative) | 36 answers · 74 criteria | `61→64/74 (+4 −1)` | `45→60/74 (+16 −1)` | `62→63/74 (+3 −2)` | `44→60/74 (+17 −1)` |
|
| 75 |
+
| **B. Insert hedging** (declarative → hedged) | 12 answers · 28 criteria | `26→27/28 (+1 −0)` | `24→25/28 (+2 −1)` | `27→27/28 (+0 −0)` | `24→25/28 (+2 −1)` |
|
| 76 |
+
| **C. Textbook-style sentence polish** | 12 answers · 27 criteria | `27→27/27 (+0 −0)` | `27→27/27 (+0 −0)` | `27→27/27 (+0 −0)` | `27→27/27 (+0 −0)` |
|
| 77 |
+
| **D. Term list → relational prose** (compared vs. A variant) | 10 answers · 25 criteria | `22→25/25 (+3 −0)` | `21→25/25 (+4 −0)` | `22→25/25 (+3 −0)` | `21→25/25 (+4 −0)` |
|
| 78 |
+
| **E. Prose → compressed term list** | 12 answers · 26 criteria | `25→25/26 (+1 −1)` | `25→24/26 (+1 −2)` | `26→26/26 (+0 −0)` | `25→24/26 (+1 −2)` |
|
| 79 |
+
|
| 80 |
+
- **Replacement Decision & Its Limitation**: `9b-v6-s3408` satisfied all conditions of the rule fixed before viewing the results—primary metric paired difference point estimate `+3.8%p > 0`, gold `missing` recall `102/142 (71.8%) >= 101/142 (71.1%)`, and gold `contradicted` recall `124/133 (93.2%) >= 122/133 (91.7%)`—so `main` was replaced (`RuBio-27B` also met the rule and replaced `main` with `c1cfefdb`; `RuBio-4B` had lower `contradicted` recall `117/133 < 122/133` than its control and therefore remains on the `ruiso-v6` branch `553fb15b` only while keeping `main` at `6f3e415a`; `RuBio-8B` is a new repository `21e6bf3d` with no replacement rule). However, **because the replacement decision was made using this same RuBench v1 `dev` split, the comparison against the previous `main` weights is not an independent validation** (the `test` split was not opened).
|
| 81 |
+
- **Previous Release (`revision="a0da6701"`) History**: The initial public release (`revision="a0da6701"`, weights commit `e19404e3`) was `wt` (seed 3407, `W=10`, `80.7% [77.2, 84.2]`), selected under a rule fixed before viewing the results as the highest-scoring model on RuBench v1 `dev` among three 2,400-row 9B candidates (`r16` seed 3407 `W=1` at `79.0%`, `s3408` seed 3408 `W=1` at `80.5%`, and `wt` at `80.7%`).
|
| 82 |
+
- **Comparison vs. Base Model (`Qwen3.5-9B`) & Same-Prompt Supplementary Comparison (Reference Only)**:
|
| 83 |
+
- **Official Comparison (vs. Base Model with `SYSTEM_PROMPT+POLICY`)**: From the Fig. 1 data, `RuBio-9B` (`84.9% [81.6, 88.1]`) outperforms base `Qwen3.5-9B` (`76.6% [73.0, 80.2]`) by `+8.2%p [+4.9, +11.6]` overall (**Improved**) and by `+6.3%p [+2.7, +10.0]` excluding R2 2/3 majority items (**Improved**). In the confusion matrix (gold `supported` 283, `contradicted` 133, `missing` 142, `unclear` 46), base `Qwen3.5-9B` over-predicts `supported` on `47/142` (`33.1%`) of gold `missing` criteria (`missing->missing` `72/142 = 50.7%`) and `28/46` (`60.9%`) of gold `unclear` criteria (`unclear->unclear` `8/46 = 17.4%`), whereas `RuBio-9B` reduces `missing->supported` to `22/142` (`15.5%`) and `unclear->supported` to `12/46` (`26.1%`), reaching `102/142` (`71.8%`) on `missing`, `23/46` (`50.0%`) on `unclear`, `124/133` (`93.2%` vs. base `113/133 = 85.0%`) on `contradicted`, and `263/283` (`92.9%` vs. old `main` control `240/283 = 84.8%` and base `270/283 = 95.4%`) on `supported`.
|
| 84 |
+
- **Same-Prompt Supplementary Comparison (`Qwen3.5-9B` with `SYSTEM_PROMPT` only; Fig. 2 b gray row)**: When base `Qwen3.5-9B` is given only `SYSTEM_PROMPT` without `POLICY`, it produces `0` JSON parse errors but outputs `checks` as a `{criterion_id: status}` dictionary instead of a list on `189` of `200` answers, resulting in a **strict score of `3.8% [1.3, 6.2]` (excl. R2 2/3 `4.2% [1.6, 6.8]`, micro `21/604`; paired Δ `RuBio-9B − strict`: `+81.1%p [+77.1, +85.2]` all / `+83.5%p [+79.5, +87.6]` excl. R2 2/3)**. Converting the dictionary to a list yields a **lenient score of `76.0% [72.3, 79.7]` (excl. R2 2/3 `80.1% [76.4, 83.7]`, micro `458/604`)**, and `RuBio-9B` minus this lenient score is `+8.9%p [+5.1, +12.6]` overall (**Improved**) and `+7.6%p [+3.6, +11.7]` excluding R2 2/3 (**Improved**). This is a non-official reference comparison.
|
| 85 |
+
- **Out-of-Rubric Error (`other_issues`) Sensitivity–Specificity Trade-Off**: Overall binary agreement on `other_issues` is identical at `153/200` (`76.5%`) for both `RuBio-9B` and base `Qwen3.5-9B`, but `RuBio-9B` achieves substantially higher recall on the 53 error-present answers (TP `42/53 = 79.2%` vs. old `main` control `16/53 = 30.2%` and base `7/53 = 13.2%`) at the cost of lower specificity on the 147 error-free answers (TN `111/147 = 75.5%`, 36 false positives vs. old `main` control `137/147 = 93.2%`, 10 false positives and base `146/147 = 99.3%`, 1 false positive).
|
| 86 |
|
| 87 |
## 2. Training Configuration
|
| 88 |
|
| 89 |
+
- **Base Model & Template Verification**: Base model is [Qwen/Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B) (Apache-2.0). Training loaded `unsloth/Qwen3.5-9B`, whose weight `sha256` digests and rendered chat templates match the official `Qwen/Qwen3.5-9B` release.
|
| 90 |
+
- **Training Data (`RuIso-v6`)**: **`train = 3,579` rows** combining the original `2,400` training rows (300 question families × 10 cases) with `708` style-only rewrites (`235` hedged phrasing, `236` compressed term-listing, `237` brief 1–2 sentence answers) and `471` out-of-rubric scientific error injection rows; the validation split remains the original **`dev = 300` rows** (split by question family).
|
| 91 |
+
- **LoRA & Optimization Settings**: All hyperparameters other than the dataset match the previous release: Unsloth `bf16` LoRA (`r=16`, `alpha=32`, target modules `q·k·v·o·gate·up·down`), 2 epochs (`448` steps; Epoch 1 at step `224`), micro-batch `2 × gradient accumulation 8` (effective batch `16`), learning rate `1e-4` with cosine scheduler, `20` warmup steps (`4%`), max sequence length `4,096`, `thinking` disabled, loss masked to assistant JSON tokens only, checkpoint selection by `dev eval_loss` evaluated every 50 steps (plus final step `448`), followed by 16-bit (`bf16`) merging.
|
| 92 |
+
- **Software Stack**: `unsloth` & `unsloth_zoo 2026.10.11`, `transformers 5.17.0`, `trl 1.15.0`, `torch 2.14.1+cu132`, `fla 0.6.0`.
|
| 93 |
+
- **Judge-Token Weighted Loss (`W=10`, difference from 4B & 27B) & Note on `eval_loss*`**: Per-criterion `status` and `misconception_id` value tokens are weighted by a factor of 10 (`W=10`) in the training and evaluation loss. Because `eval_loss` includes `W=10` weighting, it cannot be directly compared with the unweighted (`W=1`) `eval_loss` of `RuBio-4B`, `RuBio-8B`, or `RuBio-27B`; the header image denotes this as `eval_loss*`.
|
| 94 |
+
- **Two-Seed Training (`3407` & `3408`) and Checkpoint Selection**: Two seeds were trained on separate Pods:
|
| 95 |
+
- **Seed `3408` (Selected, `9b-v6-s3408`)**: Trained on an `NVIDIA RTX A6000 48GB` Pod for `train_runtime = 9,548 s` (~`159 min`, ~`1,443 tok/s`), with mean `train_loss = 0.3822` and **`best_eval_loss = 0.4248` at step 400** (step 400 teacher-forced `status_acc = 0.9756`, `ce = 0.0156`; final step 448 also had `eval_loss = 0.4248`).
|
| 96 |
+
- **Seed `3407` (`9b-v6-s3407`)**: Trained on an `NVIDIA A40 48GB` Pod for `train_runtime = 9,744 s`, with mean `train_loss = 0.3830` and `best_eval_loss = 0.4262` at step 448.
|
| 97 |
+
- Following the rule fixed before viewing the results (`best_eval_loss` `0.4248 < 0.4262`), seed `3408` (`step 400` checkpoint) was selected, merged into 4 `bf16` `safetensors` shards (~`19.3 GB` / `19.31 GB` / `18 GiB`), and uploaded at revision `f61ac9cb`.
|
| 98 |
|
| 99 |
## 3. Input & Output Specification
|
| 100 |
|
|
|
|
| 149 |
|
| 150 |
## 4. Serving & Inference
|
| 151 |
|
| 152 |
+
The merged `bf16` weights (4 `safetensors` shards) total ~`19.3 GB` (`18 GiB`). The verified serving environments are `NVIDIA RTX A6000 48GB` and `NVIDIA A40 48GB` GPUs running `vLLM 0.32.0` (smaller GPUs were not tested). Disable thinking mode (`enable_thinking: false`) so default reasoning directives are not injected into the chat template. The live demo service (https://mulk.codes/rubio) runs only `RuBio-4B` and `RuBio-27B`; `RuBio-9B` is not deployed on the service.
|
| 153 |
|
| 154 |
```bash
|
| 155 |
vllm serve mulkzz/RuBio-9B --served-model-name bio-feedback \
|
|
|
|
| 177 |
|
| 178 |
## 5. Limitations & License
|
| 179 |
|
| 180 |
+
- **Scope & Limitations**: Designed as a formative learning aid; do not use for autonomous high-stakes summative grading. Not intended for general QA outside Units I–III of the 2022 Revised *Life Science* curriculum or inputs lacking structured `rubric` and `evidence` fields. All student answers in both RuBench and the `RuIso-v6` augmentation set are synthetic (in `RuIso-v6`, student answers were rewritten by `Claude Opus 5.5 (xhigh)` and blind labels/feedback were produced by `Gemini 4 Argon (High)`), and **performance on real student answers has not been measured**. In addition, the RuBench v1 `test` split (98 answers) was not opened, and because the decision to replace `main` weights was made on the RuBench v1 `dev` split (200 answers), the comparison against the previous `main` weights is not an independent validation.
|
| 181 |
+
- **Out-of-Rubric Error Recall vs. Specificity Trade-Off**: Augmenting the training set with out-of-rubric scientific error rows (`471` rows in `RuIso-v6`) raised out-of-rubric error recall (TP) on RuBench v1 `dev` from `16/53` (`30.2%`) in the old `main` control to `42/53` (`79.2%`), but lowered specificity (TN) on the 147 error-free answers from `137/147` (`93.2%`, 10 false positives) in the old `main` control and `146/147` (`99.3%`, 1 false positive) in base `Qwen3.5-9B` to `111/147` (`75.5%`, 36 false positives). On the in-distribution old `dev` 300 split as well, false error alarms on fully correct answers (`false_error_on_correct`) increased from `0/57` (`0.0%`) to `5/57` (`8.8%`).
|
| 182 |
+
- **Reproducibility**: Re-evaluating the previous `main` weights (`@a0da6701`) under identical `vLLM 0.32.0` settings on two separate Pods (`NVIDIA RTX A6000 48GB` and `NVIDIA A40 48GB`) yielded `81.0% [77.5, 84.6]` on both Pods (paired difference `+0.0%p [-0.5, +0.5]`, identical criterion micro-accuracy `489/604`). However, even at `temperature=0`, `bf16` output is not byte-identical across batch orders and hardware.
|
| 183 |
+
- **Recommended Serving Post-processing (recompute `answer_status` and `next_action`)**: Residual policy violations on the old dev set (20 out of 300 instances) consist of `answer_status`–`checks` disagreements (14 instances), minor variations in `student_span` (5 instances), `checks` ordering/completeness (1 instance), and suspected answer leakage (1 instance); production services should recompute `answer_status` and `next_action` from the rules below instead of using the model's values. The demo service (mulk.codes/rubio) does this since 2026-11-02. All other numbers and figures in this card are raw model outputs without post-processing.
|
| 184 |
- `answer_status` (first match wins): all `supported` and no `other_issues` → `correct`; otherwise if any `supported` → `mixed` when `contradicted` or `other_issues` is present, else `partial`; otherwise `contradicted` or `other_issues` → `incorrect`; otherwise any `unclear` → `unclear`; else `no_attempt`
|
| 185 |
- `next_action` (first match wins, using the recomputed `answer_status`): `reveal_requested` → `explain`; `correct` → `complete`; `message_type` = `other` → `refocus`; `unclear` → `clarify`; else `revise`
|
|
|
|
| 186 |
- **License**: Released under [Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0), matching the base model.
|
| 187 |
|
| 188 |
---
|
|
|
|
| 191 |
|
| 192 |
<img src="./figures/rubio_9b_nature_fig1.svg" width="100%" style="width: 100%; height: auto;" alt="Fig. 3 | Training dynamics, optimization stability, and in-distribution dev set diagnostic performance of RuBio-9B" />
|
| 193 |
|
| 194 |
+
> **Fig. 3 | Training dynamics and in-distribution old Dev split (n = 300) diagnostic performance of RuBio-9B (`9b-v6-s3408`).** **a**, Step-wise judge-weighted (`W=10`) training loss (raw and EMA α = 0.32, effective batch `2 × 8 = 16`, `448` total steps, Epoch 1 = step `224`, cumulative tokens `13.77M`) and 50-step (plus final step `448`) Dev evaluation loss (minimum `0.4248` at step 400, teacher-forced judge accuracy `0.9756` = `97.6%`; `eval_loss` includes `W=10` weighting and is not directly comparable with `RuBio-4B`, `RuBio-8B`, or `RuBio-27B`, as denoted by `eval_loss*` in the header). **b**, Cosine learning rate schedule (peak `10⁻⁴`, 4% warmup = 20 steps) and gradient L₂ norm (‖g‖₂ median `0.46`, IQR `0.41–0.51`; `RTX A6000 48GB` run at ~`1,443 tok/s`, ~`159 min`). **c**, In-distribution old Dev set (`n = 300` instances, no-schema) performance comparing four model scales (`RuBio-4B v2`, `RuBio-8B`, `RuBio-9B`, `RuBio-27B`) across six metrics with 95% Wilson confidence intervals (CI); highlighted `RuBio-9B` values are `JSON valid` `300/300 = 100.0%`, `Policy compliance` `280/300 = 93.3%`, `Overall status match` `273/300 = 91.0%`, `Next-action match` `287/300 = 95.7%`, `Criterion match` `759/779 = 97.4%` (denominator 779 due to 1 omitted criterion in 1 row), and `Misconception ID` `87/97 = 89.7%`. **d**, Row-normalized per-criterion (`checks`) confusion matrix for `RuBio-9B` on the old Dev 300 split (`759/779 = 97.4%` exact match: gold `supported` `378/379 = 99.7%`, `contradicted` `163/166 = 98.2%`, `missing` `173/185 = 93.5%`, `unclear` `45/49 = 91.8%`).
|
| 195 |
|
| 196 |
+
### 1. Step-by-Step Training Convergence & Teacher-Forced Judge Metrics (`RuBio-9B v6 s3408`)
|
| 197 |
|
| 198 |
+
Because quick-probe generation was disabled during training, the step table reports teacher-forced judge evaluation metrics (`judge status_acc`, `judge mis_acc`, and `judge ce`, with `judge mis_acc` not separately logged and shown as `—`) from `training-s3408.json`. `eval_loss` includes `W=10` judge-token weighting and cannot be directly compared with the unweighted `eval_loss` of `RuBio-4B`, `RuBio-8B`, or `RuBio-27B`.
|
| 199 |
|
| 200 |
+
| Step | `loss` | `grad_norm` | `lr` | `tokens` | `eval_loss` | `judge status_acc` | `judge mis_acc` | `judge ce` |
|
| 201 |
+
|---:|---:|---:|---:|---:|---:|---:|---:|---:|
|
| 202 |
+
| 5 | 0.8829 | 0.8665 | 1.739e-05 | 155,137 | — | — | — | — |
|
| 203 |
+
| 50 | 0.4378 | 0.2977 | 9.908e-05 | 1,544,711 | 0.5116 | 0.9423 | — | 0.0380 |
|
| 204 |
+
| 100 | 0.4068 | 0.3849 | 9.232e-05 | 3,075,345 | 0.4656 | 0.9718 | — | 0.0229 |
|
| 205 |
+
| 150 | 0.3743 | 0.4155 | 7.984e-05 | 4,617,707 | 0.4492 | 0.9756 | — | 0.0177 |
|
| 206 |
+
| 200 | 0.3736 | 0.6215 | 6.333e-05 | 6,159,720 | 0.4380 | 0.9731 | — | 0.0221 |
|
| 207 |
+
| 225 | 0.3715 | 0.4184 | 5.425e-05 | 6,914,229 | — | — | — | — |
|
| 208 |
+
| 250 | 0.3387 | 0.4580 | 4.502e-05 | 7,682,315 | 0.4328 | 0.9731 | — | 0.0162 |
|
| 209 |
+
| 300 | 0.3228 | 0.4766 | 2.738e-05 | 9,215,246 | 0.4300 | 0.9808 | — | 0.0145 |
|
| 210 |
+
| 350 | 0.3371 | 0.5377 | 1.280e-05 | 10,754,759 | 0.4263 | 0.9769 | — | 0.0158 |
|
| 211 |
+
| **400 (Selected)** | **0.2959** | **0.6641** | **3.244e-06** | **12,304,134** | **0.4248** | **0.9756** | **—** | **0.0156** |
|
| 212 |
+
| 445 | 0.3040 | 0.5526 | 2.185e-08 | 13,681,536 | — | — | — | — |
|
| 213 |
+
| 448 (Final Eval) | — | — | — | — | 0.4248 | 0.9756 | — | 0.0151 |
|
| 214 |
|
| 215 |
### 2. In-Distribution Old Dev 300 Core Evaluation Metrics
|
| 216 |
|
| 217 |
+
The table below compares the new `RuBio-9B v6` weights (no-schema and schema-enforced, measured on the training Pod on 2026-11-04 via `serve/api_eval.py`) alongside the previous `main` weights (`9b-wt`, `@a0da6701`) and the other RuBio scales. Policy pass is determined by `validate_record`.
|
| 218 |
+
|
| 219 |
+
| Metric | RuBio-4B v2 (`@6f3e415a`, no schema) | RuBio-8B (`@21e6bf3d`, no schema) | RuBio-27B v6 (`@c1cfefdb`, no schema) | Old RuBio-9B (`9b-wt`, `@a0da6701`, no schema) | **RuBio-9B v6 (`@f61ac9cb`, no schema)** | **RuBio-9B v6 (`@f61ac9cb`, schema enforced)** |
|
| 220 |
+
|---|---:|---:|---:|---:|---:|---:|
|
| 221 |
+
| **Valid JSON Rate** | 300/300 (100.0%) | 299/300 (99.7%) | 300/300 (100.0%) | 300/300 (100.0%) | **300/300 (100.0%)** | **300/300 (100.0%)** |
|
| 222 |
+
| **Policy Pass Rate** (`validate_record`) | 283/300 (94.3%) | 270/300 (90.0%) | 293/300 (97.7%) | 293/300 (97.7%) | **280/300 (93.3%)** | **280/300 (93.3%)** |
|
| 223 |
+
| **Overall Status (`answer_status`) Match** | 278/300 (92.7%) | 257/300 (85.7%) | 290/300 (96.7%) | 285/300 (95.0%) | **273/300 (91.0%)** | **273/300 (91.0%)** |
|
| 224 |
+
| **Next Action (`next_action`) Match** | 293/300 (97.7%) | 284/300 (94.7%) | 294/300 (98.0%) | 289/300 (96.3%) | **287/300 (95.7%)** | **287/300 (95.7%)** |
|
| 225 |
+
| **Per-Criterion (`checks`) Status Match** | 749/780 (96.0%) | 730/777 (94.0%) | 762/780 (97.7%) | 758/780 (97.2%) | **759/779 (97.4%)** | **759/779 (97.4%)** |
|
| 226 |
+
| **Misconception ID (`misconception_id`) Match** | 87/97 (89.7%) | 83/97 (85.6%) | 93/97 (95.9%) | 90/97 (92.8%) | **87/97 (89.7%)** | **87/97 (89.7%)** |
|
| 227 |
+
| **False Error on Correct (`false_error_on_correct`)** | 0/57 (0.0%) | 7/56 (12.5%) | 1/57 (1.8%) | 0/57 (0.0%) | **5/57 (8.8%)** | **5/57 (8.8%)** |
|
| 228 |
+
| **Suspected Answer Leakage** (hint-only rows) | 0/213 (0.0%) | 0/213 (0.0%) | 0/213 (0.0%) | 0/213 (0.0%) | **1/213 (0.5%)** | **1/213 (0.5%)** |
|
| 229 |
+
| **Best Dev Evaluation Loss (`best_eval_loss`)** | — | 0.3815 | 0.3686 | 0.4281 (`W=10`)* | **0.4248 (`W=10`)*** | **0.4248 (`W=10`)*** |
|
| 230 |
+
| **Latency p50 / p95 (Output Throughput)** | 10.29s / 13.12s (665.5 tok/s) | 21.08s / 28.30s (400.6 tok/s) | 20.21s / 26.69s (334.0 tok/s) | 6.38s / 8.37s (1,050.5 tok/s) | **16.53s / 21.51s (414.4 tok/s)** | **15.93s / 20.24s (432.4 tok/s)** |
|
| 231 |
+
|
| 232 |
+
### 3. Policy Violation Breakdown on Old Dev 300 (no schema)
|
| 233 |
+
|
| 234 |
+
Comparing the new weights (`RuBio-9B v6`, `@f61ac9cb`) with the previous `9b-wt` (`@a0da6701`) on the old `dev` 300 split, per-criterion status agreement goes from `758/780 (97.2%)` to `759/779 (97.4%)` (in 1 row, `U1-01-q13-F3`, criterion `c2` was omitted from `checks`, leaving 779 evaluated criteria and counting as 1 `checks` ordering/completeness violation), whereas **policy compliance drops from `293/300 (97.7%)` to `280/300 (93.3%)` (13 fewer passing rows)**. This decrease is primarily driven by `answer_status` rule mismatches increasing from `5` to `14` as the model flags `other_issues` more frequently after `RuIso-v6` augmentation.
|
| 235 |
+
|
| 236 |
+
| Policy Violation Type (with overlap, n = 300) | Old RuBio-9B (`9b-wt`, `@a0da6701`) | **RuBio-9B v6 (`9b-v6-s3408`, `@f61ac9cb`)** | RuBio-4B v2 (`@6f3e415a`) | RuBio-8B (`@21e6bf3d`) | RuBio-27B v6 (`@c1cfefdb`) |
|
| 237 |
+
|---|---:|---:|---:|---:|---:|
|
| 238 |
+
| **Policy-Passing Rows** | 293/300 (97.7%) | **280/300 (93.3%)** | 283/300 (94.3%) | 270/300 (90.0%) | 293/300 (97.7%) |
|
| 239 |
+
| **Total Policy-Violating Rows** | 7 | **20** | 17 | 30 | 7 |
|
| 240 |
+
| `answer_status` rule mismatch | 5 | **14** | 15 | 25 | 6 |
|
| 241 |
+
| `student_span` verbatim mismatch | 3 | **5** | 2 | 3 | 1 |
|
| 242 |
+
| `checks` criterion list / order mismatch | 0 | **1** | 0 | 1 | 0 |
|
| 243 |
+
| Suspected answer leakage on hint turn | 0 | **1** | 0 | 0 | 0 |
|
| 244 |
+
| JSON parse error | 0 | **0** | 0 | 1 | 0 |
|
|
|
|
|
|
|
|
|
figures/data/rubench_v1_dev200_9b.json
CHANGED
|
@@ -3,137 +3,137 @@
|
|
| 3 |
"split": "dev",
|
| 4 |
"n_answers": 200,
|
| 5 |
"n_criteria": 604,
|
| 6 |
-
"built_at": "2026-11-
|
| 7 |
"conditions": "first turn (feedback_mode=hint_1, reveal_requested=false, history=[]), temperature 0, max_tokens 1024, enable_thinking=false, no JSON schema enforcement. RuBio: SYSTEM_PROMPT only; base: SYSTEM_PROMPT+POLICY. Invalid JSON counts as wrong.",
|
| 8 |
"finetuned": {
|
| 9 |
-
"key": "9b-
|
| 10 |
"label": "RuBio-9B",
|
| 11 |
-
"hf": "mulkzz/RuBio-9B@
|
| 12 |
-
"in_dist_train_dev_criterion_match": 0.
|
| 13 |
-
"name": "RuBio-9B
|
| 14 |
"answers": 200,
|
| 15 |
"missing_output": 0,
|
| 16 |
-
"json_invalid":
|
| 17 |
"criterion_acc": {
|
| 18 |
-
"mean": 0.
|
| 19 |
"ci95": [
|
| 20 |
-
0.
|
| 21 |
-
0.
|
| 22 |
],
|
| 23 |
"n_answers": 200
|
| 24 |
},
|
| 25 |
"criterion_acc_excl_r2_2of3": {
|
| 26 |
-
"mean": 0.
|
| 27 |
"ci95": [
|
| 28 |
-
0.
|
| 29 |
-
0.
|
| 30 |
],
|
| 31 |
"n_answers": 200
|
| 32 |
},
|
| 33 |
"criterion_micro": {
|
| 34 |
-
"hit":
|
| 35 |
"n": 604
|
| 36 |
},
|
| 37 |
"misconception": {
|
| 38 |
-
"k":
|
| 39 |
"n": 133,
|
| 40 |
-
"p": 0.
|
| 41 |
"ci95": [
|
| 42 |
-
0.
|
| 43 |
-
0.
|
| 44 |
]
|
| 45 |
},
|
| 46 |
"misconception_with_id": {
|
| 47 |
-
"k":
|
| 48 |
"n": 79,
|
| 49 |
-
"p": 0.
|
| 50 |
"ci95": [
|
| 51 |
-
0.
|
| 52 |
-
0.
|
| 53 |
]
|
| 54 |
},
|
| 55 |
"json_valid": {
|
| 56 |
-
"k":
|
| 57 |
"n": 200,
|
| 58 |
-
"p":
|
| 59 |
"ci95": [
|
| 60 |
-
0.
|
| 61 |
-
|
| 62 |
]
|
| 63 |
},
|
| 64 |
"unit_criterion_micro": {
|
| 65 |
"I": {
|
| 66 |
-
"k":
|
| 67 |
"n": 206,
|
| 68 |
-
"p": 0.
|
| 69 |
"ci95": [
|
| 70 |
-
0.
|
| 71 |
-
0.
|
| 72 |
]
|
| 73 |
},
|
| 74 |
"II": {
|
| 75 |
-
"k":
|
| 76 |
"n": 200,
|
| 77 |
-
"p": 0.
|
| 78 |
"ci95": [
|
| 79 |
-
0.
|
| 80 |
-
0.
|
| 81 |
]
|
| 82 |
},
|
| 83 |
"III": {
|
| 84 |
-
"k":
|
| 85 |
"n": 198,
|
| 86 |
-
"p": 0.
|
| 87 |
"ci95": [
|
| 88 |
-
0.
|
| 89 |
-
0.
|
| 90 |
]
|
| 91 |
}
|
| 92 |
},
|
| 93 |
"other_issues": {
|
| 94 |
-
"k":
|
| 95 |
"n": 200,
|
| 96 |
-
"p": 0.
|
| 97 |
"ci95": [
|
| 98 |
-
0.
|
| 99 |
-
0.
|
| 100 |
]
|
| 101 |
},
|
| 102 |
"answer_status": {
|
| 103 |
-
"k":
|
| 104 |
"n": 200,
|
| 105 |
-
"p": 0.
|
| 106 |
"ci95": [
|
| 107 |
-
0.
|
| 108 |
-
0.
|
| 109 |
]
|
| 110 |
},
|
| 111 |
"gold_tier_criterion_acc": {
|
| 112 |
"r1-3/3": {
|
| 113 |
-
"k":
|
| 114 |
"n": 521,
|
| 115 |
-
"p": 0.
|
| 116 |
"ci95": [
|
| 117 |
-
0.
|
| 118 |
-
0.
|
| 119 |
]
|
| 120 |
},
|
| 121 |
"r2-3/3": {
|
| 122 |
-
"k":
|
| 123 |
"n": 23,
|
| 124 |
-
"p": 0.
|
| 125 |
"ci95": [
|
| 126 |
-
0.
|
| 127 |
-
0.
|
| 128 |
]
|
| 129 |
},
|
| 130 |
"r2-2/3": {
|
| 131 |
-
"k":
|
| 132 |
"n": 51,
|
| 133 |
-
"p": 0.
|
| 134 |
"ci95": [
|
| 135 |
-
0.
|
| 136 |
-
0.
|
| 137 |
]
|
| 138 |
},
|
| 139 |
"human": {
|
|
@@ -148,41 +148,43 @@
|
|
| 148 |
},
|
| 149 |
"other_issues_split": {
|
| 150 |
"has_error_recall": {
|
| 151 |
-
"k":
|
| 152 |
"n": 53,
|
| 153 |
-
"p": 0.
|
| 154 |
"ci95": [
|
| 155 |
-
0.
|
| 156 |
-
0.
|
| 157 |
]
|
| 158 |
},
|
| 159 |
"no_error_specificity": {
|
| 160 |
-
"k":
|
| 161 |
"n": 147,
|
| 162 |
-
"p": 0.
|
| 163 |
"ci95": [
|
| 164 |
-
0.
|
| 165 |
-
0.
|
| 166 |
]
|
| 167 |
}
|
| 168 |
},
|
| 169 |
"confusion": {
|
| 170 |
-
"contradicted->
|
| 171 |
-
"contradicted->
|
|
|
|
| 172 |
"contradicted->supported": 2,
|
| 173 |
-
"contradicted->unclear":
|
| 174 |
-
"missing->contradicted":
|
| 175 |
-
"missing->missing":
|
| 176 |
-
"missing->supported":
|
| 177 |
-
"missing->unclear":
|
| 178 |
-
"supported->contradicted":
|
| 179 |
-
"supported->missing":
|
| 180 |
-
"supported->supported":
|
| 181 |
-
"supported->unclear":
|
| 182 |
-
"unclear->
|
| 183 |
-
"unclear->
|
| 184 |
-
"unclear->
|
| 185 |
-
"unclear->
|
|
|
|
| 186 |
}
|
| 187 |
},
|
| 188 |
"base": {
|
|
@@ -367,22 +369,22 @@
|
|
| 367 |
},
|
| 368 |
"paired_ft_minus_base": {
|
| 369 |
"all": {
|
| 370 |
-
"diff": 0.
|
| 371 |
"ci95": [
|
| 372 |
-
0.
|
| 373 |
-
0.
|
| 374 |
],
|
| 375 |
"n": 200,
|
| 376 |
"verdict": "개선"
|
| 377 |
},
|
| 378 |
"no_majority": {
|
| 379 |
-
"diff": 0.
|
| 380 |
"ci95": [
|
| 381 |
-
|
| 382 |
-
0.
|
| 383 |
],
|
| 384 |
"n": 200,
|
| 385 |
-
"verdict": "
|
| 386 |
}
|
| 387 |
}
|
| 388 |
}
|
|
|
|
| 3 |
"split": "dev",
|
| 4 |
"n_answers": 200,
|
| 5 |
"n_criteria": 604,
|
| 6 |
+
"built_at": "2026-11-04T00:14:40",
|
| 7 |
"conditions": "first turn (feedback_mode=hint_1, reveal_requested=false, history=[]), temperature 0, max_tokens 1024, enable_thinking=false, no JSON schema enforcement. RuBio: SYSTEM_PROMPT only; base: SYSTEM_PROMPT+POLICY. Invalid JSON counts as wrong.",
|
| 8 |
"finetuned": {
|
| 9 |
+
"key": "9b-v6-s3408",
|
| 10 |
"label": "RuBio-9B",
|
| 11 |
+
"hf": "mulkzz/RuBio-9B@f61ac9cb",
|
| 12 |
+
"in_dist_train_dev_criterion_match": 0.9743260590500642,
|
| 13 |
+
"name": "RuBio-9B RuIso-v6 (시드 3408)",
|
| 14 |
"answers": 200,
|
| 15 |
"missing_output": 0,
|
| 16 |
+
"json_invalid": 1,
|
| 17 |
"criterion_acc": {
|
| 18 |
+
"mean": 0.84875,
|
| 19 |
"ci95": [
|
| 20 |
+
0.8161433560231208,
|
| 21 |
+
0.8813566439768792
|
| 22 |
],
|
| 23 |
"n_answers": 200
|
| 24 |
},
|
| 25 |
"criterion_acc_excl_r2_2of3": {
|
| 26 |
+
"mean": 0.8770833333333333,
|
| 27 |
"ci95": [
|
| 28 |
+
0.8464292781603502,
|
| 29 |
+
0.9077373885063165
|
| 30 |
],
|
| 31 |
"n_answers": 200
|
| 32 |
},
|
| 33 |
"criterion_micro": {
|
| 34 |
+
"hit": 512,
|
| 35 |
"n": 604
|
| 36 |
},
|
| 37 |
"misconception": {
|
| 38 |
+
"k": 111,
|
| 39 |
"n": 133,
|
| 40 |
+
"p": 0.8345864661654135,
|
| 41 |
"ci95": [
|
| 42 |
+
0.7622364035210294,
|
| 43 |
+
0.888151358532034
|
| 44 |
]
|
| 45 |
},
|
| 46 |
"misconception_with_id": {
|
| 47 |
+
"k": 65,
|
| 48 |
"n": 79,
|
| 49 |
+
"p": 0.8227848101265823,
|
| 50 |
"ci95": [
|
| 51 |
+
0.7242382665030478,
|
| 52 |
+
0.8913956243248582
|
| 53 |
]
|
| 54 |
},
|
| 55 |
"json_valid": {
|
| 56 |
+
"k": 199,
|
| 57 |
"n": 200,
|
| 58 |
+
"p": 0.995,
|
| 59 |
"ci95": [
|
| 60 |
+
0.9722263725443985,
|
| 61 |
+
0.9991168287700527
|
| 62 |
]
|
| 63 |
},
|
| 64 |
"unit_criterion_micro": {
|
| 65 |
"I": {
|
| 66 |
+
"k": 170,
|
| 67 |
"n": 206,
|
| 68 |
+
"p": 0.8252427184466019,
|
| 69 |
"ci95": [
|
| 70 |
+
0.7675629836155808,
|
| 71 |
+
0.8710144010779794
|
| 72 |
]
|
| 73 |
},
|
| 74 |
"II": {
|
| 75 |
+
"k": 171,
|
| 76 |
"n": 200,
|
| 77 |
+
"p": 0.855,
|
| 78 |
"ci95": [
|
| 79 |
+
0.7995134505428394,
|
| 80 |
+
0.8971064211069184
|
| 81 |
]
|
| 82 |
},
|
| 83 |
"III": {
|
| 84 |
+
"k": 171,
|
| 85 |
"n": 198,
|
| 86 |
+
"p": 0.8636363636363636,
|
| 87 |
"ci95": [
|
| 88 |
+
0.8088693626231386,
|
| 89 |
+
0.9045619214376489
|
| 90 |
]
|
| 91 |
}
|
| 92 |
},
|
| 93 |
"other_issues": {
|
| 94 |
+
"k": 153,
|
| 95 |
"n": 200,
|
| 96 |
+
"p": 0.765,
|
| 97 |
"ci95": [
|
| 98 |
+
0.7015864484736013,
|
| 99 |
+
0.8184255683917109
|
| 100 |
]
|
| 101 |
},
|
| 102 |
"answer_status": {
|
| 103 |
+
"k": 144,
|
| 104 |
"n": 200,
|
| 105 |
+
"p": 0.72,
|
| 106 |
"ci95": [
|
| 107 |
+
0.6540770944519325,
|
| 108 |
+
0.7776309950211568
|
| 109 |
]
|
| 110 |
},
|
| 111 |
"gold_tier_criterion_acc": {
|
| 112 |
"r1-3/3": {
|
| 113 |
+
"k": 467,
|
| 114 |
"n": 521,
|
| 115 |
+
"p": 0.8963531669865643,
|
| 116 |
"ci95": [
|
| 117 |
+
0.8672147027925656,
|
| 118 |
+
0.9196896191692199
|
| 119 |
]
|
| 120 |
},
|
| 121 |
"r2-3/3": {
|
| 122 |
+
"k": 13,
|
| 123 |
"n": 23,
|
| 124 |
+
"p": 0.5652173913043478,
|
| 125 |
"ci95": [
|
| 126 |
+
0.36811411127539473,
|
| 127 |
+
0.7436533513557193
|
| 128 |
]
|
| 129 |
},
|
| 130 |
"r2-2/3": {
|
| 131 |
+
"k": 26,
|
| 132 |
"n": 51,
|
| 133 |
+
"p": 0.5098039215686274,
|
| 134 |
"ci95": [
|
| 135 |
+
0.37680949168569783,
|
| 136 |
+
0.6414248935095266
|
| 137 |
]
|
| 138 |
},
|
| 139 |
"human": {
|
|
|
|
| 148 |
},
|
| 149 |
"other_issues_split": {
|
| 150 |
"has_error_recall": {
|
| 151 |
+
"k": 42,
|
| 152 |
"n": 53,
|
| 153 |
+
"p": 0.7924528301886793,
|
| 154 |
"ci95": [
|
| 155 |
+
0.6654228381471919,
|
| 156 |
+
0.8799538844327056
|
| 157 |
]
|
| 158 |
},
|
| 159 |
"no_error_specificity": {
|
| 160 |
+
"k": 111,
|
| 161 |
"n": 147,
|
| 162 |
+
"p": 0.7551020408163265,
|
| 163 |
"ci95": [
|
| 164 |
+
0.6796736041927686,
|
| 165 |
+
0.8175372312920278
|
| 166 |
]
|
| 167 |
}
|
| 168 |
},
|
| 169 |
"confusion": {
|
| 170 |
+
"contradicted->(없음)": 1,
|
| 171 |
+
"contradicted->contradicted": 124,
|
| 172 |
+
"contradicted->missing": 5,
|
| 173 |
"contradicted->supported": 2,
|
| 174 |
+
"contradicted->unclear": 1,
|
| 175 |
+
"missing->contradicted": 7,
|
| 176 |
+
"missing->missing": 102,
|
| 177 |
+
"missing->supported": 22,
|
| 178 |
+
"missing->unclear": 11,
|
| 179 |
+
"supported->contradicted": 6,
|
| 180 |
+
"supported->missing": 9,
|
| 181 |
+
"supported->supported": 263,
|
| 182 |
+
"supported->unclear": 5,
|
| 183 |
+
"unclear->(없음)": 2,
|
| 184 |
+
"unclear->contradicted": 6,
|
| 185 |
+
"unclear->missing": 3,
|
| 186 |
+
"unclear->supported": 12,
|
| 187 |
+
"unclear->unclear": 23
|
| 188 |
}
|
| 189 |
},
|
| 190 |
"base": {
|
|
|
|
| 369 |
},
|
| 370 |
"paired_ft_minus_base": {
|
| 371 |
"all": {
|
| 372 |
+
"diff": 0.0825,
|
| 373 |
"ci95": [
|
| 374 |
+
0.04932451957550776,
|
| 375 |
+
0.11567548042449224
|
| 376 |
],
|
| 377 |
"n": 200,
|
| 378 |
"verdict": "개선"
|
| 379 |
},
|
| 380 |
"no_majority": {
|
| 381 |
+
"diff": 0.06333333333333334,
|
| 382 |
"ci95": [
|
| 383 |
+
0.02673315576101877,
|
| 384 |
+
0.09993351090564791
|
| 385 |
],
|
| 386 |
"n": 200,
|
| 387 |
+
"verdict": "개선"
|
| 388 |
}
|
| 389 |
}
|
| 390 |
}
|
figures/data/rubench_v1_dev200_all_9b.json
CHANGED
|
@@ -3,7 +3,7 @@
|
|
| 3 |
"split": "dev",
|
| 4 |
"n_answers": 200,
|
| 5 |
"n_criteria": 604,
|
| 6 |
-
"built_at": "2026-11-
|
| 7 |
"conditions": "first turn (feedback_mode=hint_1, reveal_requested=false, history=[]), no JSON schema enforcement, invalid JSON counts as wrong. RuBio: SYSTEM_PROMPT only; base and general models: SYSTEM_PROMPT+POLICY. RuBio and Qwen base: temperature 0, max_tokens 1024, thinking off. General models: vendor CLI at the listed reasoning effort, one fixed run.",
|
| 8 |
"panel": {
|
| 9 |
"round1": [
|
|
@@ -22,9 +22,10 @@
|
|
| 22 |
"GPT and Opus were round-1 gold panelists: the r1-3/3 tier (521 of 604 dev criteria) contains their own blind judgments, so their scores include self-agreement and are upper bounds.",
|
| 23 |
"Round-2 majority (r2-2/3) items include GPT, Opus and Gemini votes; see criterion_acc_excl_r2_2of3.",
|
| 24 |
"Gemini wrote the synthetic student answers with intended labels; reported as reference only (DESIGN 4-1).",
|
| 25 |
-
"Grok 4.7 (round-1 panelist) was not evaluated as a model."
|
|
|
|
| 26 |
],
|
| 27 |
-
"selection_note": "RuBio-9B
|
| 28 |
"models": {
|
| 29 |
"4b": {
|
| 30 |
"key": "4b",
|
|
@@ -392,134 +393,135 @@
|
|
| 392 |
},
|
| 393 |
"9b-wt": {
|
| 394 |
"key": "9b-wt",
|
|
|
|
| 395 |
"label": "RuBio-9B",
|
| 396 |
-
"hf": "mulkzz/RuBio-9B@
|
| 397 |
"role": "finetuned",
|
| 398 |
"size": "9B",
|
| 399 |
-
"name": "RuBio-9B
|
| 400 |
"answers": 200,
|
| 401 |
"missing_output": 0,
|
| 402 |
-
"json_invalid":
|
| 403 |
"criterion_acc": {
|
| 404 |
-
"mean": 0.
|
| 405 |
"ci95": [
|
| 406 |
-
0.
|
| 407 |
-
0.
|
| 408 |
],
|
| 409 |
"n_answers": 200
|
| 410 |
},
|
| 411 |
"criterion_acc_excl_r2_2of3": {
|
| 412 |
-
"mean": 0.
|
| 413 |
"ci95": [
|
| 414 |
-
0.
|
| 415 |
-
0.
|
| 416 |
],
|
| 417 |
"n_answers": 200
|
| 418 |
},
|
| 419 |
"criterion_micro": {
|
| 420 |
-
"hit":
|
| 421 |
"n": 604
|
| 422 |
},
|
| 423 |
"misconception": {
|
| 424 |
-
"k":
|
| 425 |
"n": 133,
|
| 426 |
-
"p": 0.
|
| 427 |
"ci95": [
|
| 428 |
-
0.
|
| 429 |
-
0.
|
| 430 |
]
|
| 431 |
},
|
| 432 |
"misconception_with_id": {
|
| 433 |
-
"k":
|
| 434 |
"n": 79,
|
| 435 |
-
"p": 0.
|
| 436 |
"ci95": [
|
| 437 |
-
0.
|
| 438 |
-
0.
|
| 439 |
]
|
| 440 |
},
|
| 441 |
"json_valid": {
|
| 442 |
-
"k":
|
| 443 |
"n": 200,
|
| 444 |
-
"p":
|
| 445 |
"ci95": [
|
| 446 |
-
0.
|
| 447 |
-
|
| 448 |
]
|
| 449 |
},
|
| 450 |
"unit_criterion_micro": {
|
| 451 |
"I": {
|
| 452 |
-
"k":
|
| 453 |
"n": 206,
|
| 454 |
-
"p": 0.
|
| 455 |
"ci95": [
|
| 456 |
-
0.
|
| 457 |
-
0.
|
| 458 |
]
|
| 459 |
},
|
| 460 |
"II": {
|
| 461 |
-
"k":
|
| 462 |
"n": 200,
|
| 463 |
-
"p": 0.
|
| 464 |
"ci95": [
|
| 465 |
-
0.
|
| 466 |
-
0.
|
| 467 |
]
|
| 468 |
},
|
| 469 |
"III": {
|
| 470 |
-
"k":
|
| 471 |
"n": 198,
|
| 472 |
-
"p": 0.
|
| 473 |
"ci95": [
|
| 474 |
-
0.
|
| 475 |
-
0.
|
| 476 |
]
|
| 477 |
}
|
| 478 |
},
|
| 479 |
"other_issues": {
|
| 480 |
-
"k":
|
| 481 |
"n": 200,
|
| 482 |
-
"p": 0.
|
| 483 |
"ci95": [
|
| 484 |
-
0.
|
| 485 |
-
0.
|
| 486 |
]
|
| 487 |
},
|
| 488 |
"answer_status": {
|
| 489 |
-
"k":
|
| 490 |
"n": 200,
|
| 491 |
-
"p": 0.
|
| 492 |
"ci95": [
|
| 493 |
-
0.
|
| 494 |
-
0.
|
| 495 |
]
|
| 496 |
},
|
| 497 |
"gold_tier_criterion_acc": {
|
| 498 |
"r1-3/3": {
|
| 499 |
-
"k":
|
| 500 |
"n": 521,
|
| 501 |
-
"p": 0.
|
| 502 |
"ci95": [
|
| 503 |
-
0.
|
| 504 |
-
0.
|
| 505 |
]
|
| 506 |
},
|
| 507 |
"r2-3/3": {
|
| 508 |
-
"k":
|
| 509 |
"n": 23,
|
| 510 |
-
"p": 0.
|
| 511 |
"ci95": [
|
| 512 |
-
0.
|
| 513 |
-
0.
|
| 514 |
]
|
| 515 |
},
|
| 516 |
"r2-2/3": {
|
| 517 |
-
"k":
|
| 518 |
"n": 51,
|
| 519 |
-
"p": 0.
|
| 520 |
"ci95": [
|
| 521 |
-
0.
|
| 522 |
-
0.
|
| 523 |
]
|
| 524 |
},
|
| 525 |
"human": {
|
|
@@ -534,41 +536,43 @@
|
|
| 534 |
},
|
| 535 |
"other_issues_split": {
|
| 536 |
"has_error_recall": {
|
| 537 |
-
"k":
|
| 538 |
"n": 53,
|
| 539 |
-
"p": 0.
|
| 540 |
"ci95": [
|
| 541 |
-
0.
|
| 542 |
-
0.
|
| 543 |
]
|
| 544 |
},
|
| 545 |
"no_error_specificity": {
|
| 546 |
-
"k":
|
| 547 |
"n": 147,
|
| 548 |
-
"p": 0.
|
| 549 |
"ci95": [
|
| 550 |
-
0.
|
| 551 |
-
0.
|
| 552 |
]
|
| 553 |
}
|
| 554 |
},
|
| 555 |
"confusion": {
|
| 556 |
-
"contradicted->
|
| 557 |
-
"contradicted->
|
|
|
|
| 558 |
"contradicted->supported": 2,
|
| 559 |
-
"contradicted->unclear":
|
| 560 |
-
"missing->contradicted":
|
| 561 |
-
"missing->missing":
|
| 562 |
-
"missing->supported":
|
| 563 |
-
"missing->unclear":
|
| 564 |
-
"supported->contradicted":
|
| 565 |
-
"supported->missing":
|
| 566 |
-
"supported->supported":
|
| 567 |
-
"supported->unclear":
|
| 568 |
-
"unclear->
|
| 569 |
-
"unclear->
|
| 570 |
-
"unclear->
|
| 571 |
-
"unclear->
|
|
|
|
| 572 |
},
|
| 573 |
"cli_seconds_p50": null
|
| 574 |
},
|
|
@@ -756,41 +760,42 @@
|
|
| 756 |
},
|
| 757 |
"27b": {
|
| 758 |
"key": "27b",
|
|
|
|
| 759 |
"label": "RuBio-27B",
|
| 760 |
-
"hf": "mulkzz/RuBio-27B@
|
| 761 |
"role": "finetuned",
|
| 762 |
"size": "27B",
|
| 763 |
-
"name": "RuBio-27B",
|
| 764 |
"answers": 200,
|
| 765 |
"missing_output": 0,
|
| 766 |
"json_invalid": 0,
|
| 767 |
"criterion_acc": {
|
| 768 |
-
"mean": 0.
|
| 769 |
"ci95": [
|
| 770 |
-
0.
|
| 771 |
-
0.
|
| 772 |
],
|
| 773 |
"n_answers": 200
|
| 774 |
},
|
| 775 |
"criterion_acc_excl_r2_2of3": {
|
| 776 |
-
"mean": 0.
|
| 777 |
"ci95": [
|
| 778 |
-
0.
|
| 779 |
-
0.
|
| 780 |
],
|
| 781 |
"n_answers": 200
|
| 782 |
},
|
| 783 |
"criterion_micro": {
|
| 784 |
-
"hit":
|
| 785 |
"n": 604
|
| 786 |
},
|
| 787 |
"misconception": {
|
| 788 |
-
"k":
|
| 789 |
"n": 133,
|
| 790 |
-
"p": 0.
|
| 791 |
"ci95": [
|
| 792 |
-
0.
|
| 793 |
-
0.
|
| 794 |
]
|
| 795 |
},
|
| 796 |
"misconception_with_id": {
|
|
@@ -813,126 +818,126 @@
|
|
| 813 |
},
|
| 814 |
"unit_criterion_micro": {
|
| 815 |
"I": {
|
| 816 |
-
"k":
|
| 817 |
"n": 206,
|
| 818 |
-
"p": 0.
|
| 819 |
"ci95": [
|
| 820 |
-
0.
|
| 821 |
-
0.
|
| 822 |
]
|
| 823 |
},
|
| 824 |
"II": {
|
| 825 |
-
"k":
|
| 826 |
"n": 200,
|
| 827 |
-
"p": 0.
|
| 828 |
"ci95": [
|
| 829 |
-
0.
|
| 830 |
-
0.
|
| 831 |
]
|
| 832 |
},
|
| 833 |
"III": {
|
| 834 |
-
"k":
|
| 835 |
"n": 198,
|
| 836 |
-
"p": 0.
|
| 837 |
"ci95": [
|
| 838 |
-
0.
|
| 839 |
-
0.
|
| 840 |
]
|
| 841 |
}
|
| 842 |
},
|
| 843 |
"other_issues": {
|
| 844 |
-
"k":
|
| 845 |
"n": 200,
|
| 846 |
-
"p": 0.
|
| 847 |
"ci95": [
|
| 848 |
-
0.
|
| 849 |
-
0.
|
| 850 |
]
|
| 851 |
},
|
| 852 |
"answer_status": {
|
| 853 |
-
"k":
|
| 854 |
"n": 200,
|
| 855 |
-
"p": 0.
|
| 856 |
"ci95": [
|
| 857 |
-
0.
|
| 858 |
-
0.
|
| 859 |
]
|
| 860 |
},
|
| 861 |
"gold_tier_criterion_acc": {
|
| 862 |
"r1-3/3": {
|
| 863 |
-
"k":
|
| 864 |
"n": 521,
|
| 865 |
-
"p": 0.
|
| 866 |
"ci95": [
|
| 867 |
-
0.
|
| 868 |
-
0.
|
| 869 |
]
|
| 870 |
},
|
| 871 |
"r2-3/3": {
|
| 872 |
-
"k":
|
| 873 |
"n": 23,
|
| 874 |
-
"p": 0.
|
| 875 |
"ci95": [
|
| 876 |
-
0.
|
| 877 |
-
0.
|
| 878 |
]
|
| 879 |
},
|
| 880 |
"r2-2/3": {
|
| 881 |
-
"k":
|
| 882 |
"n": 51,
|
| 883 |
-
"p": 0.
|
| 884 |
"ci95": [
|
| 885 |
-
0.
|
| 886 |
-
0.
|
| 887 |
]
|
| 888 |
},
|
| 889 |
"human": {
|
| 890 |
-
"k":
|
| 891 |
"n": 9,
|
| 892 |
-
"p": 0.
|
| 893 |
"ci95": [
|
| 894 |
-
0.
|
| 895 |
-
0.
|
| 896 |
]
|
| 897 |
}
|
| 898 |
},
|
| 899 |
"other_issues_split": {
|
| 900 |
"has_error_recall": {
|
| 901 |
-
"k":
|
| 902 |
"n": 53,
|
| 903 |
-
"p": 0.
|
| 904 |
"ci95": [
|
| 905 |
-
0.
|
| 906 |
-
0.
|
| 907 |
]
|
| 908 |
},
|
| 909 |
"no_error_specificity": {
|
| 910 |
-
"k":
|
| 911 |
"n": 147,
|
| 912 |
-
"p": 0.
|
| 913 |
"ci95": [
|
| 914 |
-
0.
|
| 915 |
-
0.
|
| 916 |
]
|
| 917 |
}
|
| 918 |
},
|
| 919 |
"confusion": {
|
| 920 |
-
"contradicted->contradicted":
|
| 921 |
-
"contradicted->missing":
|
| 922 |
"contradicted->supported": 1,
|
| 923 |
-
"contradicted->unclear":
|
| 924 |
-
"missing->contradicted":
|
| 925 |
-
"missing->missing":
|
| 926 |
-
"missing->supported":
|
| 927 |
-
"missing->unclear":
|
| 928 |
-
"supported->contradicted":
|
| 929 |
-
"supported->missing":
|
| 930 |
-
"supported->supported":
|
| 931 |
-
"supported->unclear":
|
| 932 |
-
"unclear->contradicted":
|
| 933 |
-
"unclear->missing":
|
| 934 |
-
"unclear->supported":
|
| 935 |
-
"unclear->unclear":
|
| 936 |
},
|
| 937 |
"cli_seconds_p50": null
|
| 938 |
},
|
|
@@ -1688,19 +1693,19 @@
|
|
| 1688 |
},
|
| 1689 |
"9b-wt-minus-4b": {
|
| 1690 |
"all": {
|
| 1691 |
-
"diff": 0.
|
| 1692 |
"ci95": [
|
| 1693 |
-
0.
|
| 1694 |
-
0.
|
| 1695 |
],
|
| 1696 |
"n": 200,
|
| 1697 |
"verdict": "개선"
|
| 1698 |
},
|
| 1699 |
"no_majority": {
|
| 1700 |
-
"diff": 0.
|
| 1701 |
"ci95": [
|
| 1702 |
-
0.
|
| 1703 |
-
0.
|
| 1704 |
],
|
| 1705 |
"n": 200,
|
| 1706 |
"verdict": "개선"
|
|
@@ -1728,19 +1733,19 @@
|
|
| 1728 |
},
|
| 1729 |
"27b-minus-4b": {
|
| 1730 |
"all": {
|
| 1731 |
-
"diff": 0.
|
| 1732 |
"ci95": [
|
| 1733 |
-
0.
|
| 1734 |
-
0.
|
| 1735 |
],
|
| 1736 |
"n": 200,
|
| 1737 |
"verdict": "개선"
|
| 1738 |
},
|
| 1739 |
"no_majority": {
|
| 1740 |
-
"diff": 0.
|
| 1741 |
"ci95": [
|
| 1742 |
-
0.
|
| 1743 |
-
0.
|
| 1744 |
],
|
| 1745 |
"n": 200,
|
| 1746 |
"verdict": "개선"
|
|
@@ -1848,19 +1853,19 @@
|
|
| 1848 |
},
|
| 1849 |
"9b-wt-minus-qwen4b": {
|
| 1850 |
"all": {
|
| 1851 |
-
"diff": 0.
|
| 1852 |
"ci95": [
|
| 1853 |
-
0.
|
| 1854 |
-
0.
|
| 1855 |
],
|
| 1856 |
"n": 200,
|
| 1857 |
"verdict": "개선"
|
| 1858 |
},
|
| 1859 |
"no_majority": {
|
| 1860 |
-
"diff": 0.
|
| 1861 |
"ci95": [
|
| 1862 |
-
0.
|
| 1863 |
-
0.
|
| 1864 |
],
|
| 1865 |
"n": 200,
|
| 1866 |
"verdict": "개선"
|
|
@@ -1888,19 +1893,19 @@
|
|
| 1888 |
},
|
| 1889 |
"27b-minus-qwen4b": {
|
| 1890 |
"all": {
|
| 1891 |
-
"diff": 0.
|
| 1892 |
"ci95": [
|
| 1893 |
-
0.
|
| 1894 |
-
0.
|
| 1895 |
],
|
| 1896 |
"n": 200,
|
| 1897 |
"verdict": "개선"
|
| 1898 |
},
|
| 1899 |
"no_majority": {
|
| 1900 |
-
"diff": 0.
|
| 1901 |
"ci95": [
|
| 1902 |
-
0.
|
| 1903 |
-
0.
|
| 1904 |
],
|
| 1905 |
"n": 200,
|
| 1906 |
"verdict": "개선"
|
|
@@ -1988,19 +1993,19 @@
|
|
| 1988 |
},
|
| 1989 |
"4b-minus-9b-wt": {
|
| 1990 |
"all": {
|
| 1991 |
-
"diff": -0.
|
| 1992 |
"ci95": [
|
| 1993 |
-
-0.
|
| 1994 |
-
-0.
|
| 1995 |
],
|
| 1996 |
"n": 200,
|
| 1997 |
"verdict": "악화"
|
| 1998 |
},
|
| 1999 |
"no_majority": {
|
| 2000 |
-
"diff": -0.
|
| 2001 |
"ci95": [
|
| 2002 |
-
-0.
|
| 2003 |
-
-0.
|
| 2004 |
],
|
| 2005 |
"n": 200,
|
| 2006 |
"verdict": "악화"
|
|
@@ -2008,19 +2013,19 @@
|
|
| 2008 |
},
|
| 2009 |
"qwen4b-minus-9b-wt": {
|
| 2010 |
"all": {
|
| 2011 |
-
"diff": -0.
|
| 2012 |
"ci95": [
|
| 2013 |
-
-0.
|
| 2014 |
-
-0.
|
| 2015 |
],
|
| 2016 |
"n": 200,
|
| 2017 |
"verdict": "악화"
|
| 2018 |
},
|
| 2019 |
"no_majority": {
|
| 2020 |
-
"diff": -0.
|
| 2021 |
"ci95": [
|
| 2022 |
-
-0.
|
| 2023 |
-
-0.
|
| 2024 |
],
|
| 2025 |
"n": 200,
|
| 2026 |
"verdict": "악화"
|
|
@@ -2028,59 +2033,59 @@
|
|
| 2028 |
},
|
| 2029 |
"qwen9b-minus-9b-wt": {
|
| 2030 |
"all": {
|
| 2031 |
-
"diff": -0.
|
| 2032 |
"ci95": [
|
| 2033 |
-
-0.
|
| 2034 |
-
-0.
|
| 2035 |
],
|
| 2036 |
"n": 200,
|
| 2037 |
"verdict": "악화"
|
| 2038 |
},
|
| 2039 |
"no_majority": {
|
| 2040 |
-
"diff": -0.
|
| 2041 |
"ci95": [
|
| 2042 |
-
-0.
|
| 2043 |
-
0.
|
| 2044 |
],
|
| 2045 |
"n": 200,
|
| 2046 |
-
"verdict": "
|
| 2047 |
}
|
| 2048 |
},
|
| 2049 |
"27b-minus-9b-wt": {
|
| 2050 |
"all": {
|
| 2051 |
-
"diff": 0.
|
| 2052 |
"ci95": [
|
| 2053 |
-
0.
|
| 2054 |
-
0.
|
| 2055 |
],
|
| 2056 |
"n": 200,
|
| 2057 |
-
"verdict": "
|
| 2058 |
},
|
| 2059 |
"no_majority": {
|
| 2060 |
-
"diff": 0.
|
| 2061 |
"ci95": [
|
| 2062 |
-
0.
|
| 2063 |
-
0.
|
| 2064 |
],
|
| 2065 |
"n": 200,
|
| 2066 |
-
"verdict": "
|
| 2067 |
}
|
| 2068 |
},
|
| 2069 |
"qwen27b-minus-9b-wt": {
|
| 2070 |
"all": {
|
| 2071 |
-
"diff": 0.
|
| 2072 |
"ci95": [
|
| 2073 |
-
-0.
|
| 2074 |
-
0.
|
| 2075 |
],
|
| 2076 |
"n": 200,
|
| 2077 |
"verdict": "차이 없음"
|
| 2078 |
},
|
| 2079 |
"no_majority": {
|
| 2080 |
-
"diff": 0.
|
| 2081 |
"ci95": [
|
| 2082 |
-
-0.
|
| 2083 |
-
0.
|
| 2084 |
],
|
| 2085 |
"n": 200,
|
| 2086 |
"verdict": "차이 없음"
|
|
@@ -2088,19 +2093,19 @@
|
|
| 2088 |
},
|
| 2089 |
"gpt-minus-9b-wt": {
|
| 2090 |
"all": {
|
| 2091 |
-
"diff": 0.
|
| 2092 |
"ci95": [
|
| 2093 |
-
0.
|
| 2094 |
-
0.
|
| 2095 |
],
|
| 2096 |
"n": 200,
|
| 2097 |
"verdict": "개선"
|
| 2098 |
},
|
| 2099 |
"no_majority": {
|
| 2100 |
-
"diff": 0.
|
| 2101 |
"ci95": [
|
| 2102 |
-
0.
|
| 2103 |
-
0.
|
| 2104 |
],
|
| 2105 |
"n": 200,
|
| 2106 |
"verdict": "개선"
|
|
@@ -2108,19 +2113,19 @@
|
|
| 2108 |
},
|
| 2109 |
"opus-minus-9b-wt": {
|
| 2110 |
"all": {
|
| 2111 |
-
"diff": 0.
|
| 2112 |
"ci95": [
|
| 2113 |
-
0.
|
| 2114 |
-
0.
|
| 2115 |
],
|
| 2116 |
"n": 200,
|
| 2117 |
"verdict": "개선"
|
| 2118 |
},
|
| 2119 |
"no_majority": {
|
| 2120 |
-
"diff": 0.
|
| 2121 |
"ci95": [
|
| 2122 |
-
0.
|
| 2123 |
-
0.
|
| 2124 |
],
|
| 2125 |
"n": 200,
|
| 2126 |
"verdict": "개선"
|
|
@@ -2128,19 +2133,19 @@
|
|
| 2128 |
},
|
| 2129 |
"gemini-minus-9b-wt": {
|
| 2130 |
"all": {
|
| 2131 |
-
"diff": 0.
|
| 2132 |
"ci95": [
|
| 2133 |
-
0.
|
| 2134 |
-
0.
|
| 2135 |
],
|
| 2136 |
"n": 200,
|
| 2137 |
"verdict": "개선"
|
| 2138 |
},
|
| 2139 |
"no_majority": {
|
| 2140 |
-
"diff": 0.
|
| 2141 |
"ci95": [
|
| 2142 |
-
0.
|
| 2143 |
-
0.
|
| 2144 |
],
|
| 2145 |
"n": 200,
|
| 2146 |
"verdict": "개선"
|
|
@@ -2188,39 +2193,39 @@
|
|
| 2188 |
},
|
| 2189 |
"9b-wt-minus-qwen9b": {
|
| 2190 |
"all": {
|
| 2191 |
-
"diff": 0.
|
| 2192 |
"ci95": [
|
| 2193 |
-
0.
|
| 2194 |
-
0.
|
| 2195 |
],
|
| 2196 |
"n": 200,
|
| 2197 |
"verdict": "개선"
|
| 2198 |
},
|
| 2199 |
"no_majority": {
|
| 2200 |
-
"diff": 0.
|
| 2201 |
"ci95": [
|
| 2202 |
-
|
| 2203 |
-
0.
|
| 2204 |
],
|
| 2205 |
"n": 200,
|
| 2206 |
-
"verdict": "
|
| 2207 |
}
|
| 2208 |
},
|
| 2209 |
"27b-minus-qwen9b": {
|
| 2210 |
"all": {
|
| 2211 |
-
"diff": 0.
|
| 2212 |
"ci95": [
|
| 2213 |
-
0.
|
| 2214 |
-
0.
|
| 2215 |
],
|
| 2216 |
"n": 200,
|
| 2217 |
"verdict": "개선"
|
| 2218 |
},
|
| 2219 |
"no_majority": {
|
| 2220 |
-
"diff": 0.
|
| 2221 |
"ci95": [
|
| 2222 |
-
0.
|
| 2223 |
-
0.
|
| 2224 |
],
|
| 2225 |
"n": 200,
|
| 2226 |
"verdict": "개선"
|
|
@@ -2308,19 +2313,19 @@
|
|
| 2308 |
},
|
| 2309 |
"4b-minus-27b": {
|
| 2310 |
"all": {
|
| 2311 |
-
"diff": -0.
|
| 2312 |
"ci95": [
|
| 2313 |
-
-0.
|
| 2314 |
-
-0.
|
| 2315 |
],
|
| 2316 |
"n": 200,
|
| 2317 |
"verdict": "악화"
|
| 2318 |
},
|
| 2319 |
"no_majority": {
|
| 2320 |
-
"diff": -0.
|
| 2321 |
"ci95": [
|
| 2322 |
-
-0.
|
| 2323 |
-
-0.
|
| 2324 |
],
|
| 2325 |
"n": 200,
|
| 2326 |
"verdict": "악화"
|
|
@@ -2328,19 +2333,19 @@
|
|
| 2328 |
},
|
| 2329 |
"qwen4b-minus-27b": {
|
| 2330 |
"all": {
|
| 2331 |
-
"diff": -0.
|
| 2332 |
"ci95": [
|
| 2333 |
-
-0.
|
| 2334 |
-
-0.
|
| 2335 |
],
|
| 2336 |
"n": 200,
|
| 2337 |
"verdict": "악화"
|
| 2338 |
},
|
| 2339 |
"no_majority": {
|
| 2340 |
-
"diff": -0.
|
| 2341 |
"ci95": [
|
| 2342 |
-
-0.
|
| 2343 |
-
-0.
|
| 2344 |
],
|
| 2345 |
"n": 200,
|
| 2346 |
"verdict": "악화"
|
|
@@ -2348,39 +2353,39 @@
|
|
| 2348 |
},
|
| 2349 |
"9b-wt-minus-27b": {
|
| 2350 |
"all": {
|
| 2351 |
-
"diff": -0.
|
| 2352 |
"ci95": [
|
| 2353 |
-
-0.
|
| 2354 |
-
|
| 2355 |
],
|
| 2356 |
"n": 200,
|
| 2357 |
-
"verdict": "
|
| 2358 |
},
|
| 2359 |
"no_majority": {
|
| 2360 |
-
"diff": -0.
|
| 2361 |
"ci95": [
|
| 2362 |
-
-0.
|
| 2363 |
-
|
| 2364 |
],
|
| 2365 |
"n": 200,
|
| 2366 |
-
"verdict": "
|
| 2367 |
}
|
| 2368 |
},
|
| 2369 |
"qwen9b-minus-27b": {
|
| 2370 |
"all": {
|
| 2371 |
-
"diff": -0.
|
| 2372 |
"ci95": [
|
| 2373 |
-
-0.
|
| 2374 |
-
-0.
|
| 2375 |
],
|
| 2376 |
"n": 200,
|
| 2377 |
"verdict": "악화"
|
| 2378 |
},
|
| 2379 |
"no_majority": {
|
| 2380 |
-
"diff": -0.
|
| 2381 |
"ci95": [
|
| 2382 |
-
-0.
|
| 2383 |
-
-0.
|
| 2384 |
],
|
| 2385 |
"n": 200,
|
| 2386 |
"verdict": "악화"
|
|
@@ -2388,39 +2393,39 @@
|
|
| 2388 |
},
|
| 2389 |
"qwen27b-minus-27b": {
|
| 2390 |
"all": {
|
| 2391 |
-
"diff": -0.
|
| 2392 |
"ci95": [
|
| 2393 |
-
-0.
|
| 2394 |
-
0.
|
| 2395 |
],
|
| 2396 |
"n": 200,
|
| 2397 |
-
"verdict": "
|
| 2398 |
},
|
| 2399 |
"no_majority": {
|
| 2400 |
-
"diff": -0.
|
| 2401 |
"ci95": [
|
| 2402 |
-
-0.
|
| 2403 |
-
0.
|
| 2404 |
],
|
| 2405 |
"n": 200,
|
| 2406 |
-
"verdict": "
|
| 2407 |
}
|
| 2408 |
},
|
| 2409 |
"gpt-minus-27b": {
|
| 2410 |
"all": {
|
| 2411 |
-
"diff": 0.
|
| 2412 |
"ci95": [
|
| 2413 |
-
0.
|
| 2414 |
-
0.
|
| 2415 |
],
|
| 2416 |
"n": 200,
|
| 2417 |
"verdict": "개선"
|
| 2418 |
},
|
| 2419 |
"no_majority": {
|
| 2420 |
-
"diff": 0.
|
| 2421 |
"ci95": [
|
| 2422 |
-
0.
|
| 2423 |
-
0.
|
| 2424 |
],
|
| 2425 |
"n": 200,
|
| 2426 |
"verdict": "개선"
|
|
@@ -2428,19 +2433,19 @@
|
|
| 2428 |
},
|
| 2429 |
"opus-minus-27b": {
|
| 2430 |
"all": {
|
| 2431 |
-
"diff": 0.
|
| 2432 |
"ci95": [
|
| 2433 |
-
0.
|
| 2434 |
-
0.
|
| 2435 |
],
|
| 2436 |
"n": 200,
|
| 2437 |
"verdict": "개선"
|
| 2438 |
},
|
| 2439 |
"no_majority": {
|
| 2440 |
-
"diff": 0.
|
| 2441 |
"ci95": [
|
| 2442 |
-
0.
|
| 2443 |
-
0.
|
| 2444 |
],
|
| 2445 |
"n": 200,
|
| 2446 |
"verdict": "개선"
|
|
@@ -2448,19 +2453,19 @@
|
|
| 2448 |
},
|
| 2449 |
"gemini-minus-27b": {
|
| 2450 |
"all": {
|
| 2451 |
-
"diff": 0.
|
| 2452 |
"ci95": [
|
| 2453 |
-
0.
|
| 2454 |
-
0.
|
| 2455 |
],
|
| 2456 |
"n": 200,
|
| 2457 |
"verdict": "개선"
|
| 2458 |
},
|
| 2459 |
"no_majority": {
|
| 2460 |
-
"diff": 0.
|
| 2461 |
"ci95": [
|
| 2462 |
-
0.
|
| 2463 |
-
0.
|
| 2464 |
],
|
| 2465 |
"n": 200,
|
| 2466 |
"verdict": "개선"
|
|
@@ -2508,19 +2513,19 @@
|
|
| 2508 |
},
|
| 2509 |
"9b-wt-minus-qwen27b": {
|
| 2510 |
"all": {
|
| 2511 |
-
"diff":
|
| 2512 |
"ci95": [
|
| 2513 |
-
-0.
|
| 2514 |
-
0.
|
| 2515 |
],
|
| 2516 |
"n": 200,
|
| 2517 |
"verdict": "차이 없음"
|
| 2518 |
},
|
| 2519 |
"no_majority": {
|
| 2520 |
-
"diff":
|
| 2521 |
"ci95": [
|
| 2522 |
-
-0.
|
| 2523 |
-
0.
|
| 2524 |
],
|
| 2525 |
"n": 200,
|
| 2526 |
"verdict": "차이 없음"
|
|
@@ -2548,22 +2553,22 @@
|
|
| 2548 |
},
|
| 2549 |
"27b-minus-qwen27b": {
|
| 2550 |
"all": {
|
| 2551 |
-
"diff": 0.
|
| 2552 |
"ci95": [
|
| 2553 |
-
|
| 2554 |
-
0.
|
| 2555 |
],
|
| 2556 |
"n": 200,
|
| 2557 |
-
"verdict": "
|
| 2558 |
},
|
| 2559 |
"no_majority": {
|
| 2560 |
-
"diff": 0.
|
| 2561 |
"ci95": [
|
| 2562 |
-
|
| 2563 |
-
0.
|
| 2564 |
],
|
| 2565 |
"n": 200,
|
| 2566 |
-
"verdict": "
|
| 2567 |
}
|
| 2568 |
},
|
| 2569 |
"gpt-minus-qwen27b": {
|
|
@@ -2668,19 +2673,19 @@
|
|
| 2668 |
},
|
| 2669 |
"9b-wt-minus-gpt": {
|
| 2670 |
"all": {
|
| 2671 |
-
"diff": -0.
|
| 2672 |
"ci95": [
|
| 2673 |
-
-0.
|
| 2674 |
-
-0.
|
| 2675 |
],
|
| 2676 |
"n": 200,
|
| 2677 |
"verdict": "악화"
|
| 2678 |
},
|
| 2679 |
"no_majority": {
|
| 2680 |
-
"diff": -0.
|
| 2681 |
"ci95": [
|
| 2682 |
-
-0.
|
| 2683 |
-
-0.
|
| 2684 |
],
|
| 2685 |
"n": 200,
|
| 2686 |
"verdict": "악화"
|
|
@@ -2708,19 +2713,19 @@
|
|
| 2708 |
},
|
| 2709 |
"27b-minus-gpt": {
|
| 2710 |
"all": {
|
| 2711 |
-
"diff": -0.
|
| 2712 |
"ci95": [
|
| 2713 |
-
-0.
|
| 2714 |
-
-0.
|
| 2715 |
],
|
| 2716 |
"n": 200,
|
| 2717 |
"verdict": "악화"
|
| 2718 |
},
|
| 2719 |
"no_majority": {
|
| 2720 |
-
"diff": -0.
|
| 2721 |
"ci95": [
|
| 2722 |
-
-0.
|
| 2723 |
-
-0.
|
| 2724 |
],
|
| 2725 |
"n": 200,
|
| 2726 |
"verdict": "악화"
|
|
@@ -2828,19 +2833,19 @@
|
|
| 2828 |
},
|
| 2829 |
"9b-wt-minus-opus": {
|
| 2830 |
"all": {
|
| 2831 |
-
"diff": -0.
|
| 2832 |
"ci95": [
|
| 2833 |
-
-0.
|
| 2834 |
-
-0.
|
| 2835 |
],
|
| 2836 |
"n": 200,
|
| 2837 |
"verdict": "악화"
|
| 2838 |
},
|
| 2839 |
"no_majority": {
|
| 2840 |
-
"diff": -0.
|
| 2841 |
"ci95": [
|
| 2842 |
-
-0.
|
| 2843 |
-
-0.
|
| 2844 |
],
|
| 2845 |
"n": 200,
|
| 2846 |
"verdict": "악화"
|
|
@@ -2868,19 +2873,19 @@
|
|
| 2868 |
},
|
| 2869 |
"27b-minus-opus": {
|
| 2870 |
"all": {
|
| 2871 |
-
"diff": -0.
|
| 2872 |
"ci95": [
|
| 2873 |
-
-0.
|
| 2874 |
-
-0.
|
| 2875 |
],
|
| 2876 |
"n": 200,
|
| 2877 |
"verdict": "악화"
|
| 2878 |
},
|
| 2879 |
"no_majority": {
|
| 2880 |
-
"diff": -0.
|
| 2881 |
"ci95": [
|
| 2882 |
-
-0.
|
| 2883 |
-
-0.
|
| 2884 |
],
|
| 2885 |
"n": 200,
|
| 2886 |
"verdict": "악화"
|
|
@@ -2988,19 +2993,19 @@
|
|
| 2988 |
},
|
| 2989 |
"9b-wt-minus-gemini": {
|
| 2990 |
"all": {
|
| 2991 |
-
"diff": -0.
|
| 2992 |
"ci95": [
|
| 2993 |
-
-0.
|
| 2994 |
-
-0.
|
| 2995 |
],
|
| 2996 |
"n": 200,
|
| 2997 |
"verdict": "악화"
|
| 2998 |
},
|
| 2999 |
"no_majority": {
|
| 3000 |
-
"diff": -0.
|
| 3001 |
"ci95": [
|
| 3002 |
-
-0.
|
| 3003 |
-
-0.
|
| 3004 |
],
|
| 3005 |
"n": 200,
|
| 3006 |
"verdict": "악화"
|
|
@@ -3028,19 +3033,19 @@
|
|
| 3028 |
},
|
| 3029 |
"27b-minus-gemini": {
|
| 3030 |
"all": {
|
| 3031 |
-
"diff": -0.
|
| 3032 |
"ci95": [
|
| 3033 |
-
-0.
|
| 3034 |
-
-0.
|
| 3035 |
],
|
| 3036 |
"n": 200,
|
| 3037 |
"verdict": "악화"
|
| 3038 |
},
|
| 3039 |
"no_majority": {
|
| 3040 |
-
"diff": -0.
|
| 3041 |
"ci95": [
|
| 3042 |
-
-0.
|
| 3043 |
-
-0.
|
| 3044 |
],
|
| 3045 |
"n": 200,
|
| 3046 |
"verdict": "악화"
|
|
@@ -3134,19 +3139,19 @@
|
|
| 3134 |
"json_invalid": 0,
|
| 3135 |
"paired_9b_minus_this": {
|
| 3136 |
"all": {
|
| 3137 |
-
"diff": 0.
|
| 3138 |
"ci95": [
|
| 3139 |
-
0.
|
| 3140 |
-
0.
|
| 3141 |
],
|
| 3142 |
"n": 200,
|
| 3143 |
"verdict": "개선"
|
| 3144 |
},
|
| 3145 |
"no_majority": {
|
| 3146 |
-
"diff": 0.
|
| 3147 |
"ci95": [
|
| 3148 |
-
0.
|
| 3149 |
-
0.
|
| 3150 |
],
|
| 3151 |
"n": 200,
|
| 3152 |
"verdict": "개선"
|
|
@@ -3177,22 +3182,22 @@
|
|
| 3177 |
"json_invalid": 0,
|
| 3178 |
"paired_9b_minus_this": {
|
| 3179 |
"all": {
|
| 3180 |
-
"diff": 0.
|
| 3181 |
"ci95": [
|
| 3182 |
-
0.
|
| 3183 |
-
0.
|
| 3184 |
],
|
| 3185 |
"n": 200,
|
| 3186 |
"verdict": "개선"
|
| 3187 |
},
|
| 3188 |
"no_majority": {
|
| 3189 |
-
"diff": 0.
|
| 3190 |
"ci95": [
|
| 3191 |
-
|
| 3192 |
-
0.
|
| 3193 |
],
|
| 3194 |
"n": 200,
|
| 3195 |
-
"verdict": "
|
| 3196 |
}
|
| 3197 |
}
|
| 3198 |
}
|
|
|
|
| 3 |
"split": "dev",
|
| 4 |
"n_answers": 200,
|
| 5 |
"n_criteria": 604,
|
| 6 |
+
"built_at": "2026-11-04T00:15:59",
|
| 7 |
"conditions": "first turn (feedback_mode=hint_1, reveal_requested=false, history=[]), no JSON schema enforcement, invalid JSON counts as wrong. RuBio: SYSTEM_PROMPT only; base and general models: SYSTEM_PROMPT+POLICY. RuBio and Qwen base: temperature 0, max_tokens 1024, thinking off. General models: vendor CLI at the listed reasoning effort, one fixed run.",
|
| 8 |
"panel": {
|
| 9 |
"round1": [
|
|
|
|
| 22 |
"GPT and Opus were round-1 gold panelists: the r1-3/3 tier (521 of 604 dev criteria) contains their own blind judgments, so their scores include self-agreement and are upper bounds.",
|
| 23 |
"Round-2 majority (r2-2/3) items include GPT, Opus and Gemini votes; see criterion_acc_excl_r2_2of3.",
|
| 24 |
"Gemini wrote the synthetic student answers with intended labels; reported as reference only (DESIGN 4-1).",
|
| 25 |
+
"Grok 4.7 (round-1 panelist) was not evaluated as a model.",
|
| 26 |
+
"RuBio-27B is the RuIso-v6 retrain (main since 2026-11-04), run on a RunPod A100 Pod with vLLM 0.32.0 under the same generation settings; RuBio-4B v2 was run on RunPod Serverless (worker-v1-vllm, vLLM 0.32.0). The decision to replace the 27B main weights used this same dev split."
|
| 27 |
],
|
| 28 |
+
"selection_note": "RuBio-9B (RuIso-v6 retrain) seed was chosen by lowest best_eval_loss before RuBench scoring; the decision to replace the main weights used this same dev split.",
|
| 29 |
"models": {
|
| 30 |
"4b": {
|
| 31 |
"key": "4b",
|
|
|
|
| 393 |
},
|
| 394 |
"9b-wt": {
|
| 395 |
"key": "9b-wt",
|
| 396 |
+
"src": "9b-v6-s3408",
|
| 397 |
"label": "RuBio-9B",
|
| 398 |
+
"hf": "mulkzz/RuBio-9B@f61ac9cb",
|
| 399 |
"role": "finetuned",
|
| 400 |
"size": "9B",
|
| 401 |
+
"name": "RuBio-9B RuIso-v6 (시드 3408)",
|
| 402 |
"answers": 200,
|
| 403 |
"missing_output": 0,
|
| 404 |
+
"json_invalid": 1,
|
| 405 |
"criterion_acc": {
|
| 406 |
+
"mean": 0.84875,
|
| 407 |
"ci95": [
|
| 408 |
+
0.8161433560231208,
|
| 409 |
+
0.8813566439768792
|
| 410 |
],
|
| 411 |
"n_answers": 200
|
| 412 |
},
|
| 413 |
"criterion_acc_excl_r2_2of3": {
|
| 414 |
+
"mean": 0.8770833333333333,
|
| 415 |
"ci95": [
|
| 416 |
+
0.8464292781603502,
|
| 417 |
+
0.9077373885063165
|
| 418 |
],
|
| 419 |
"n_answers": 200
|
| 420 |
},
|
| 421 |
"criterion_micro": {
|
| 422 |
+
"hit": 512,
|
| 423 |
"n": 604
|
| 424 |
},
|
| 425 |
"misconception": {
|
| 426 |
+
"k": 111,
|
| 427 |
"n": 133,
|
| 428 |
+
"p": 0.8345864661654135,
|
| 429 |
"ci95": [
|
| 430 |
+
0.7622364035210294,
|
| 431 |
+
0.888151358532034
|
| 432 |
]
|
| 433 |
},
|
| 434 |
"misconception_with_id": {
|
| 435 |
+
"k": 65,
|
| 436 |
"n": 79,
|
| 437 |
+
"p": 0.8227848101265823,
|
| 438 |
"ci95": [
|
| 439 |
+
0.7242382665030478,
|
| 440 |
+
0.8913956243248582
|
| 441 |
]
|
| 442 |
},
|
| 443 |
"json_valid": {
|
| 444 |
+
"k": 199,
|
| 445 |
"n": 200,
|
| 446 |
+
"p": 0.995,
|
| 447 |
"ci95": [
|
| 448 |
+
0.9722263725443985,
|
| 449 |
+
0.9991168287700527
|
| 450 |
]
|
| 451 |
},
|
| 452 |
"unit_criterion_micro": {
|
| 453 |
"I": {
|
| 454 |
+
"k": 170,
|
| 455 |
"n": 206,
|
| 456 |
+
"p": 0.8252427184466019,
|
| 457 |
"ci95": [
|
| 458 |
+
0.7675629836155808,
|
| 459 |
+
0.8710144010779794
|
| 460 |
]
|
| 461 |
},
|
| 462 |
"II": {
|
| 463 |
+
"k": 171,
|
| 464 |
"n": 200,
|
| 465 |
+
"p": 0.855,
|
| 466 |
"ci95": [
|
| 467 |
+
0.7995134505428394,
|
| 468 |
+
0.8971064211069184
|
| 469 |
]
|
| 470 |
},
|
| 471 |
"III": {
|
| 472 |
+
"k": 171,
|
| 473 |
"n": 198,
|
| 474 |
+
"p": 0.8636363636363636,
|
| 475 |
"ci95": [
|
| 476 |
+
0.8088693626231386,
|
| 477 |
+
0.9045619214376489
|
| 478 |
]
|
| 479 |
}
|
| 480 |
},
|
| 481 |
"other_issues": {
|
| 482 |
+
"k": 153,
|
| 483 |
"n": 200,
|
| 484 |
+
"p": 0.765,
|
| 485 |
"ci95": [
|
| 486 |
+
0.7015864484736013,
|
| 487 |
+
0.8184255683917109
|
| 488 |
]
|
| 489 |
},
|
| 490 |
"answer_status": {
|
| 491 |
+
"k": 144,
|
| 492 |
"n": 200,
|
| 493 |
+
"p": 0.72,
|
| 494 |
"ci95": [
|
| 495 |
+
0.6540770944519325,
|
| 496 |
+
0.7776309950211568
|
| 497 |
]
|
| 498 |
},
|
| 499 |
"gold_tier_criterion_acc": {
|
| 500 |
"r1-3/3": {
|
| 501 |
+
"k": 467,
|
| 502 |
"n": 521,
|
| 503 |
+
"p": 0.8963531669865643,
|
| 504 |
"ci95": [
|
| 505 |
+
0.8672147027925656,
|
| 506 |
+
0.9196896191692199
|
| 507 |
]
|
| 508 |
},
|
| 509 |
"r2-3/3": {
|
| 510 |
+
"k": 13,
|
| 511 |
"n": 23,
|
| 512 |
+
"p": 0.5652173913043478,
|
| 513 |
"ci95": [
|
| 514 |
+
0.36811411127539473,
|
| 515 |
+
0.7436533513557193
|
| 516 |
]
|
| 517 |
},
|
| 518 |
"r2-2/3": {
|
| 519 |
+
"k": 26,
|
| 520 |
"n": 51,
|
| 521 |
+
"p": 0.5098039215686274,
|
| 522 |
"ci95": [
|
| 523 |
+
0.37680949168569783,
|
| 524 |
+
0.6414248935095266
|
| 525 |
]
|
| 526 |
},
|
| 527 |
"human": {
|
|
|
|
| 536 |
},
|
| 537 |
"other_issues_split": {
|
| 538 |
"has_error_recall": {
|
| 539 |
+
"k": 42,
|
| 540 |
"n": 53,
|
| 541 |
+
"p": 0.7924528301886793,
|
| 542 |
"ci95": [
|
| 543 |
+
0.6654228381471919,
|
| 544 |
+
0.8799538844327056
|
| 545 |
]
|
| 546 |
},
|
| 547 |
"no_error_specificity": {
|
| 548 |
+
"k": 111,
|
| 549 |
"n": 147,
|
| 550 |
+
"p": 0.7551020408163265,
|
| 551 |
"ci95": [
|
| 552 |
+
0.6796736041927686,
|
| 553 |
+
0.8175372312920278
|
| 554 |
]
|
| 555 |
}
|
| 556 |
},
|
| 557 |
"confusion": {
|
| 558 |
+
"contradicted->(없음)": 1,
|
| 559 |
+
"contradicted->contradicted": 124,
|
| 560 |
+
"contradicted->missing": 5,
|
| 561 |
"contradicted->supported": 2,
|
| 562 |
+
"contradicted->unclear": 1,
|
| 563 |
+
"missing->contradicted": 7,
|
| 564 |
+
"missing->missing": 102,
|
| 565 |
+
"missing->supported": 22,
|
| 566 |
+
"missing->unclear": 11,
|
| 567 |
+
"supported->contradicted": 6,
|
| 568 |
+
"supported->missing": 9,
|
| 569 |
+
"supported->supported": 263,
|
| 570 |
+
"supported->unclear": 5,
|
| 571 |
+
"unclear->(없음)": 2,
|
| 572 |
+
"unclear->contradicted": 6,
|
| 573 |
+
"unclear->missing": 3,
|
| 574 |
+
"unclear->supported": 12,
|
| 575 |
+
"unclear->unclear": 23
|
| 576 |
},
|
| 577 |
"cli_seconds_p50": null
|
| 578 |
},
|
|
|
|
| 760 |
},
|
| 761 |
"27b": {
|
| 762 |
"key": "27b",
|
| 763 |
+
"src": "27b-v6-s3407",
|
| 764 |
"label": "RuBio-27B",
|
| 765 |
+
"hf": "mulkzz/RuBio-27B@c1cfefdb",
|
| 766 |
"role": "finetuned",
|
| 767 |
"size": "27B",
|
| 768 |
+
"name": "RuBio-27B RuIso-v6 (시드 3407)",
|
| 769 |
"answers": 200,
|
| 770 |
"missing_output": 0,
|
| 771 |
"json_invalid": 0,
|
| 772 |
"criterion_acc": {
|
| 773 |
+
"mean": 0.86875,
|
| 774 |
"ci95": [
|
| 775 |
+
0.8381287620959553,
|
| 776 |
+
0.8993712379040447
|
| 777 |
],
|
| 778 |
"n_answers": 200
|
| 779 |
},
|
| 780 |
"criterion_acc_excl_r2_2of3": {
|
| 781 |
+
"mean": 0.9,
|
| 782 |
"ci95": [
|
| 783 |
+
0.8714503967862466,
|
| 784 |
+
0.9285496032137535
|
| 785 |
],
|
| 786 |
"n_answers": 200
|
| 787 |
},
|
| 788 |
"criterion_micro": {
|
| 789 |
+
"hit": 524,
|
| 790 |
"n": 604
|
| 791 |
},
|
| 792 |
"misconception": {
|
| 793 |
+
"k": 118,
|
| 794 |
"n": 133,
|
| 795 |
+
"p": 0.8872180451127819,
|
| 796 |
"ci95": [
|
| 797 |
+
0.8222450524506153,
|
| 798 |
+
0.9304508968916938
|
| 799 |
]
|
| 800 |
},
|
| 801 |
"misconception_with_id": {
|
|
|
|
| 818 |
},
|
| 819 |
"unit_criterion_micro": {
|
| 820 |
"I": {
|
| 821 |
+
"k": 185,
|
| 822 |
"n": 206,
|
| 823 |
+
"p": 0.8980582524271845,
|
| 824 |
"ci95": [
|
| 825 |
+
0.8491894424118119,
|
| 826 |
+
0.9323530284071725
|
| 827 |
]
|
| 828 |
},
|
| 829 |
"II": {
|
| 830 |
+
"k": 166,
|
| 831 |
"n": 200,
|
| 832 |
+
"p": 0.83,
|
| 833 |
"ci95": [
|
| 834 |
+
0.7718412936135373,
|
| 835 |
+
0.8757208405960968
|
| 836 |
]
|
| 837 |
},
|
| 838 |
"III": {
|
| 839 |
+
"k": 173,
|
| 840 |
"n": 198,
|
| 841 |
+
"p": 0.8737373737373737,
|
| 842 |
"ci95": [
|
| 843 |
+
0.8202540537968194,
|
| 844 |
+
0.9129947659323233
|
| 845 |
]
|
| 846 |
}
|
| 847 |
},
|
| 848 |
"other_issues": {
|
| 849 |
+
"k": 173,
|
| 850 |
"n": 200,
|
| 851 |
+
"p": 0.865,
|
| 852 |
"ci95": [
|
| 853 |
+
0.810708740295804,
|
| 854 |
+
0.9055342263300035
|
| 855 |
]
|
| 856 |
},
|
| 857 |
"answer_status": {
|
| 858 |
+
"k": 154,
|
| 859 |
"n": 200,
|
| 860 |
+
"p": 0.77,
|
| 861 |
"ci95": [
|
| 862 |
+
0.7069170932155301,
|
| 863 |
+
0.822906471137807
|
| 864 |
]
|
| 865 |
},
|
| 866 |
"gold_tier_criterion_acc": {
|
| 867 |
"r1-3/3": {
|
| 868 |
+
"k": 483,
|
| 869 |
"n": 521,
|
| 870 |
+
"p": 0.927063339731286,
|
| 871 |
"ci95": [
|
| 872 |
+
0.9014725811198147,
|
| 873 |
+
0.9464025357639493
|
| 874 |
]
|
| 875 |
},
|
| 876 |
"r2-3/3": {
|
| 877 |
+
"k": 11,
|
| 878 |
"n": 23,
|
| 879 |
+
"p": 0.4782608695652174,
|
| 880 |
"ci95": [
|
| 881 |
+
0.29237189265791597,
|
| 882 |
+
0.670372286465046
|
| 883 |
]
|
| 884 |
},
|
| 885 |
"r2-2/3": {
|
| 886 |
+
"k": 23,
|
| 887 |
"n": 51,
|
| 888 |
+
"p": 0.45098039215686275,
|
| 889 |
"ci95": [
|
| 890 |
+
0.32267542174059904,
|
| 891 |
+
0.5861526522832785
|
| 892 |
]
|
| 893 |
},
|
| 894 |
"human": {
|
| 895 |
+
"k": 7,
|
| 896 |
"n": 9,
|
| 897 |
+
"p": 0.7777777777777778,
|
| 898 |
"ci95": [
|
| 899 |
+
0.45258959147532657,
|
| 900 |
+
0.9367747440984988
|
| 901 |
]
|
| 902 |
}
|
| 903 |
},
|
| 904 |
"other_issues_split": {
|
| 905 |
"has_error_recall": {
|
| 906 |
+
"k": 46,
|
| 907 |
"n": 53,
|
| 908 |
+
"p": 0.8679245283018868,
|
| 909 |
"ci95": [
|
| 910 |
+
0.7515977273649117,
|
| 911 |
+
0.9345213752356045
|
| 912 |
]
|
| 913 |
},
|
| 914 |
"no_error_specificity": {
|
| 915 |
+
"k": 127,
|
| 916 |
"n": 147,
|
| 917 |
+
"p": 0.8639455782312925,
|
| 918 |
"ci95": [
|
| 919 |
+
0.7991850746700077,
|
| 920 |
+
0.9101690506216352
|
| 921 |
]
|
| 922 |
}
|
| 923 |
},
|
| 924 |
"confusion": {
|
| 925 |
+
"contradicted->contradicted": 127,
|
| 926 |
+
"contradicted->missing": 4,
|
| 927 |
"contradicted->supported": 1,
|
| 928 |
+
"contradicted->unclear": 1,
|
| 929 |
+
"missing->contradicted": 8,
|
| 930 |
+
"missing->missing": 125,
|
| 931 |
+
"missing->supported": 2,
|
| 932 |
+
"missing->unclear": 7,
|
| 933 |
+
"supported->contradicted": 4,
|
| 934 |
+
"supported->missing": 20,
|
| 935 |
+
"supported->supported": 253,
|
| 936 |
+
"supported->unclear": 6,
|
| 937 |
+
"unclear->contradicted": 4,
|
| 938 |
+
"unclear->missing": 15,
|
| 939 |
+
"unclear->supported": 8,
|
| 940 |
+
"unclear->unclear": 19
|
| 941 |
},
|
| 942 |
"cli_seconds_p50": null
|
| 943 |
},
|
|
|
|
| 1693 |
},
|
| 1694 |
"9b-wt-minus-4b": {
|
| 1695 |
"all": {
|
| 1696 |
+
"diff": 0.07708333333333334,
|
| 1697 |
"ci95": [
|
| 1698 |
+
0.04658272750965517,
|
| 1699 |
+
0.1075839391570115
|
| 1700 |
],
|
| 1701 |
"n": 200,
|
| 1702 |
"verdict": "개선"
|
| 1703 |
},
|
| 1704 |
"no_majority": {
|
| 1705 |
+
"diff": 0.07833333333333334,
|
| 1706 |
"ci95": [
|
| 1707 |
+
0.043278517520531544,
|
| 1708 |
+
0.11338814914613513
|
| 1709 |
],
|
| 1710 |
"n": 200,
|
| 1711 |
"verdict": "개선"
|
|
|
|
| 1733 |
},
|
| 1734 |
"27b-minus-4b": {
|
| 1735 |
"all": {
|
| 1736 |
+
"diff": 0.09708333333333334,
|
| 1737 |
"ci95": [
|
| 1738 |
+
0.06123184381518337,
|
| 1739 |
+
0.1329348228514833
|
| 1740 |
],
|
| 1741 |
"n": 200,
|
| 1742 |
"verdict": "개선"
|
| 1743 |
},
|
| 1744 |
"no_majority": {
|
| 1745 |
+
"diff": 0.10125,
|
| 1746 |
"ci95": [
|
| 1747 |
+
0.06521017094468781,
|
| 1748 |
+
0.1372898290553122
|
| 1749 |
],
|
| 1750 |
"n": 200,
|
| 1751 |
"verdict": "개선"
|
|
|
|
| 1853 |
},
|
| 1854 |
"9b-wt-minus-qwen4b": {
|
| 1855 |
"all": {
|
| 1856 |
+
"diff": 0.13958333333333334,
|
| 1857 |
"ci95": [
|
| 1858 |
+
0.10343097151649225,
|
| 1859 |
+
0.17573569515017443
|
| 1860 |
],
|
| 1861 |
"n": 200,
|
| 1862 |
"verdict": "개선"
|
| 1863 |
},
|
| 1864 |
"no_majority": {
|
| 1865 |
+
"diff": 0.13166666666666668,
|
| 1866 |
"ci95": [
|
| 1867 |
+
0.0910702383211025,
|
| 1868 |
+
0.17226309501223086
|
| 1869 |
],
|
| 1870 |
"n": 200,
|
| 1871 |
"verdict": "개선"
|
|
|
|
| 1893 |
},
|
| 1894 |
"27b-minus-qwen4b": {
|
| 1895 |
"all": {
|
| 1896 |
+
"diff": 0.15958333333333333,
|
| 1897 |
"ci95": [
|
| 1898 |
+
0.12437326122557398,
|
| 1899 |
+
0.1947934054410927
|
| 1900 |
],
|
| 1901 |
"n": 200,
|
| 1902 |
"verdict": "개선"
|
| 1903 |
},
|
| 1904 |
"no_majority": {
|
| 1905 |
+
"diff": 0.15458333333333335,
|
| 1906 |
"ci95": [
|
| 1907 |
+
0.11312043866157052,
|
| 1908 |
+
0.1960462280050962
|
| 1909 |
],
|
| 1910 |
"n": 200,
|
| 1911 |
"verdict": "개선"
|
|
|
|
| 1993 |
},
|
| 1994 |
"4b-minus-9b-wt": {
|
| 1995 |
"all": {
|
| 1996 |
+
"diff": -0.07708333333333334,
|
| 1997 |
"ci95": [
|
| 1998 |
+
-0.1075839391570115,
|
| 1999 |
+
-0.04658272750965517
|
| 2000 |
],
|
| 2001 |
"n": 200,
|
| 2002 |
"verdict": "악화"
|
| 2003 |
},
|
| 2004 |
"no_majority": {
|
| 2005 |
+
"diff": -0.07833333333333334,
|
| 2006 |
"ci95": [
|
| 2007 |
+
-0.11338814914613513,
|
| 2008 |
+
-0.043278517520531544
|
| 2009 |
],
|
| 2010 |
"n": 200,
|
| 2011 |
"verdict": "악화"
|
|
|
|
| 2013 |
},
|
| 2014 |
"qwen4b-minus-9b-wt": {
|
| 2015 |
"all": {
|
| 2016 |
+
"diff": -0.13958333333333334,
|
| 2017 |
"ci95": [
|
| 2018 |
+
-0.17573569515017443,
|
| 2019 |
+
-0.10343097151649225
|
| 2020 |
],
|
| 2021 |
"n": 200,
|
| 2022 |
"verdict": "악화"
|
| 2023 |
},
|
| 2024 |
"no_majority": {
|
| 2025 |
+
"diff": -0.13166666666666668,
|
| 2026 |
"ci95": [
|
| 2027 |
+
-0.17226309501223086,
|
| 2028 |
+
-0.0910702383211025
|
| 2029 |
],
|
| 2030 |
"n": 200,
|
| 2031 |
"verdict": "악화"
|
|
|
|
| 2033 |
},
|
| 2034 |
"qwen9b-minus-9b-wt": {
|
| 2035 |
"all": {
|
| 2036 |
+
"diff": -0.0825,
|
| 2037 |
"ci95": [
|
| 2038 |
+
-0.11567548042449224,
|
| 2039 |
+
-0.04932451957550776
|
| 2040 |
],
|
| 2041 |
"n": 200,
|
| 2042 |
"verdict": "악화"
|
| 2043 |
},
|
| 2044 |
"no_majority": {
|
| 2045 |
+
"diff": -0.06333333333333334,
|
| 2046 |
"ci95": [
|
| 2047 |
+
-0.09993351090564791,
|
| 2048 |
+
-0.02673315576101877
|
| 2049 |
],
|
| 2050 |
"n": 200,
|
| 2051 |
+
"verdict": "악화"
|
| 2052 |
}
|
| 2053 |
},
|
| 2054 |
"27b-minus-9b-wt": {
|
| 2055 |
"all": {
|
| 2056 |
+
"diff": 0.02,
|
| 2057 |
"ci95": [
|
| 2058 |
+
-0.014332229414202247,
|
| 2059 |
+
0.05433222941420225
|
| 2060 |
],
|
| 2061 |
"n": 200,
|
| 2062 |
+
"verdict": "차이 없음"
|
| 2063 |
},
|
| 2064 |
"no_majority": {
|
| 2065 |
+
"diff": 0.02291666666666667,
|
| 2066 |
"ci95": [
|
| 2067 |
+
-0.011448688434242939,
|
| 2068 |
+
0.057282021767576276
|
| 2069 |
],
|
| 2070 |
"n": 200,
|
| 2071 |
+
"verdict": "차이 없음"
|
| 2072 |
}
|
| 2073 |
},
|
| 2074 |
"qwen27b-minus-9b-wt": {
|
| 2075 |
"all": {
|
| 2076 |
+
"diff": -0.030833333333333334,
|
| 2077 |
"ci95": [
|
| 2078 |
+
-0.06614857086946481,
|
| 2079 |
+
0.004481904202798138
|
| 2080 |
],
|
| 2081 |
"n": 200,
|
| 2082 |
"verdict": "차이 없음"
|
| 2083 |
},
|
| 2084 |
"no_majority": {
|
| 2085 |
+
"diff": -0.0275,
|
| 2086 |
"ci95": [
|
| 2087 |
+
-0.06235288624359042,
|
| 2088 |
+
0.007352886243590425
|
| 2089 |
],
|
| 2090 |
"n": 200,
|
| 2091 |
"verdict": "차이 없음"
|
|
|
|
| 2093 |
},
|
| 2094 |
"gpt-minus-9b-wt": {
|
| 2095 |
"all": {
|
| 2096 |
+
"diff": 0.06291666666666668,
|
| 2097 |
"ci95": [
|
| 2098 |
+
0.03084018163541989,
|
| 2099 |
+
0.09499315169791346
|
| 2100 |
],
|
| 2101 |
"n": 200,
|
| 2102 |
"verdict": "개선"
|
| 2103 |
},
|
| 2104 |
"no_majority": {
|
| 2105 |
+
"diff": 0.07875000000000001,
|
| 2106 |
"ci95": [
|
| 2107 |
+
0.04891013146731345,
|
| 2108 |
+
0.10858986853268658
|
| 2109 |
],
|
| 2110 |
"n": 200,
|
| 2111 |
"verdict": "개선"
|
|
|
|
| 2113 |
},
|
| 2114 |
"opus-minus-9b-wt": {
|
| 2115 |
"all": {
|
| 2116 |
+
"diff": 0.09333333333333334,
|
| 2117 |
"ci95": [
|
| 2118 |
+
0.061867359403474584,
|
| 2119 |
+
0.12479930726319209
|
| 2120 |
],
|
| 2121 |
"n": 200,
|
| 2122 |
"verdict": "개선"
|
| 2123 |
},
|
| 2124 |
"no_majority": {
|
| 2125 |
+
"diff": 0.08375,
|
| 2126 |
"ci95": [
|
| 2127 |
+
0.05559051266998988,
|
| 2128 |
+
0.11190948733001013
|
| 2129 |
],
|
| 2130 |
"n": 200,
|
| 2131 |
"verdict": "개선"
|
|
|
|
| 2133 |
},
|
| 2134 |
"gemini-minus-9b-wt": {
|
| 2135 |
"all": {
|
| 2136 |
+
"diff": 0.06375,
|
| 2137 |
"ci95": [
|
| 2138 |
+
0.02575279935154528,
|
| 2139 |
+
0.10174720064845472
|
| 2140 |
],
|
| 2141 |
"n": 200,
|
| 2142 |
"verdict": "개선"
|
| 2143 |
},
|
| 2144 |
"no_majority": {
|
| 2145 |
+
"diff": 0.05875,
|
| 2146 |
"ci95": [
|
| 2147 |
+
0.024038033740232924,
|
| 2148 |
+
0.09346196625976708
|
| 2149 |
],
|
| 2150 |
"n": 200,
|
| 2151 |
"verdict": "개선"
|
|
|
|
| 2193 |
},
|
| 2194 |
"9b-wt-minus-qwen9b": {
|
| 2195 |
"all": {
|
| 2196 |
+
"diff": 0.0825,
|
| 2197 |
"ci95": [
|
| 2198 |
+
0.04932451957550776,
|
| 2199 |
+
0.11567548042449224
|
| 2200 |
],
|
| 2201 |
"n": 200,
|
| 2202 |
"verdict": "개선"
|
| 2203 |
},
|
| 2204 |
"no_majority": {
|
| 2205 |
+
"diff": 0.06333333333333334,
|
| 2206 |
"ci95": [
|
| 2207 |
+
0.02673315576101877,
|
| 2208 |
+
0.09993351090564791
|
| 2209 |
],
|
| 2210 |
"n": 200,
|
| 2211 |
+
"verdict": "개선"
|
| 2212 |
}
|
| 2213 |
},
|
| 2214 |
"27b-minus-qwen9b": {
|
| 2215 |
"all": {
|
| 2216 |
+
"diff": 0.1025,
|
| 2217 |
"ci95": [
|
| 2218 |
+
0.06413880766177524,
|
| 2219 |
+
0.14086119233822475
|
| 2220 |
],
|
| 2221 |
"n": 200,
|
| 2222 |
"verdict": "개선"
|
| 2223 |
},
|
| 2224 |
"no_majority": {
|
| 2225 |
+
"diff": 0.08625,
|
| 2226 |
"ci95": [
|
| 2227 |
+
0.04496564082615172,
|
| 2228 |
+
0.12753435917384826
|
| 2229 |
],
|
| 2230 |
"n": 200,
|
| 2231 |
"verdict": "개선"
|
|
|
|
| 2313 |
},
|
| 2314 |
"4b-minus-27b": {
|
| 2315 |
"all": {
|
| 2316 |
+
"diff": -0.09708333333333334,
|
| 2317 |
"ci95": [
|
| 2318 |
+
-0.1329348228514833,
|
| 2319 |
+
-0.06123184381518337
|
| 2320 |
],
|
| 2321 |
"n": 200,
|
| 2322 |
"verdict": "악화"
|
| 2323 |
},
|
| 2324 |
"no_majority": {
|
| 2325 |
+
"diff": -0.10125,
|
| 2326 |
"ci95": [
|
| 2327 |
+
-0.1372898290553122,
|
| 2328 |
+
-0.06521017094468781
|
| 2329 |
],
|
| 2330 |
"n": 200,
|
| 2331 |
"verdict": "악화"
|
|
|
|
| 2333 |
},
|
| 2334 |
"qwen4b-minus-27b": {
|
| 2335 |
"all": {
|
| 2336 |
+
"diff": -0.15958333333333333,
|
| 2337 |
"ci95": [
|
| 2338 |
+
-0.1947934054410927,
|
| 2339 |
+
-0.12437326122557398
|
| 2340 |
],
|
| 2341 |
"n": 200,
|
| 2342 |
"verdict": "악화"
|
| 2343 |
},
|
| 2344 |
"no_majority": {
|
| 2345 |
+
"diff": -0.15458333333333335,
|
| 2346 |
"ci95": [
|
| 2347 |
+
-0.1960462280050962,
|
| 2348 |
+
-0.11312043866157052
|
| 2349 |
],
|
| 2350 |
"n": 200,
|
| 2351 |
"verdict": "악화"
|
|
|
|
| 2353 |
},
|
| 2354 |
"9b-wt-minus-27b": {
|
| 2355 |
"all": {
|
| 2356 |
+
"diff": -0.02,
|
| 2357 |
"ci95": [
|
| 2358 |
+
-0.05433222941420225,
|
| 2359 |
+
0.014332229414202247
|
| 2360 |
],
|
| 2361 |
"n": 200,
|
| 2362 |
+
"verdict": "차이 없음"
|
| 2363 |
},
|
| 2364 |
"no_majority": {
|
| 2365 |
+
"diff": -0.02291666666666667,
|
| 2366 |
"ci95": [
|
| 2367 |
+
-0.057282021767576276,
|
| 2368 |
+
0.011448688434242939
|
| 2369 |
],
|
| 2370 |
"n": 200,
|
| 2371 |
+
"verdict": "차이 없음"
|
| 2372 |
}
|
| 2373 |
},
|
| 2374 |
"qwen9b-minus-27b": {
|
| 2375 |
"all": {
|
| 2376 |
+
"diff": -0.1025,
|
| 2377 |
"ci95": [
|
| 2378 |
+
-0.14086119233822475,
|
| 2379 |
+
-0.06413880766177524
|
| 2380 |
],
|
| 2381 |
"n": 200,
|
| 2382 |
"verdict": "악화"
|
| 2383 |
},
|
| 2384 |
"no_majority": {
|
| 2385 |
+
"diff": -0.08625,
|
| 2386 |
"ci95": [
|
| 2387 |
+
-0.12753435917384826,
|
| 2388 |
+
-0.04496564082615172
|
| 2389 |
],
|
| 2390 |
"n": 200,
|
| 2391 |
"verdict": "악화"
|
|
|
|
| 2393 |
},
|
| 2394 |
"qwen27b-minus-27b": {
|
| 2395 |
"all": {
|
| 2396 |
+
"diff": -0.05083333333333334,
|
| 2397 |
"ci95": [
|
| 2398 |
+
-0.09092183669301501,
|
| 2399 |
+
-0.010744829973651678
|
| 2400 |
],
|
| 2401 |
"n": 200,
|
| 2402 |
+
"verdict": "악화"
|
| 2403 |
},
|
| 2404 |
"no_majority": {
|
| 2405 |
+
"diff": -0.05041666666666667,
|
| 2406 |
"ci95": [
|
| 2407 |
+
-0.0903058270121205,
|
| 2408 |
+
-0.01052750632121284
|
| 2409 |
],
|
| 2410 |
"n": 200,
|
| 2411 |
+
"verdict": "악화"
|
| 2412 |
}
|
| 2413 |
},
|
| 2414 |
"gpt-minus-27b": {
|
| 2415 |
"all": {
|
| 2416 |
+
"diff": 0.04291666666666667,
|
| 2417 |
"ci95": [
|
| 2418 |
+
0.01078839936599886,
|
| 2419 |
+
0.07504493396733448
|
| 2420 |
],
|
| 2421 |
"n": 200,
|
| 2422 |
"verdict": "개선"
|
| 2423 |
},
|
| 2424 |
"no_majority": {
|
| 2425 |
+
"diff": 0.05583333333333334,
|
| 2426 |
"ci95": [
|
| 2427 |
+
0.02437694687489072,
|
| 2428 |
+
0.08728971979177597
|
| 2429 |
],
|
| 2430 |
"n": 200,
|
| 2431 |
"verdict": "개선"
|
|
|
|
| 2433 |
},
|
| 2434 |
"opus-minus-27b": {
|
| 2435 |
"all": {
|
| 2436 |
+
"diff": 0.07333333333333333,
|
| 2437 |
"ci95": [
|
| 2438 |
+
0.04152834968415308,
|
| 2439 |
+
0.10513831698251358
|
| 2440 |
],
|
| 2441 |
"n": 200,
|
| 2442 |
"verdict": "개선"
|
| 2443 |
},
|
| 2444 |
"no_majority": {
|
| 2445 |
+
"diff": 0.060833333333333336,
|
| 2446 |
"ci95": [
|
| 2447 |
+
0.030981954682252844,
|
| 2448 |
+
0.09068471198441383
|
| 2449 |
],
|
| 2450 |
"n": 200,
|
| 2451 |
"verdict": "개선"
|
|
|
|
| 2453 |
},
|
| 2454 |
"gemini-minus-27b": {
|
| 2455 |
"all": {
|
| 2456 |
+
"diff": 0.04375,
|
| 2457 |
"ci95": [
|
| 2458 |
+
0.009717845060980758,
|
| 2459 |
+
0.07778215493901924
|
| 2460 |
],
|
| 2461 |
"n": 200,
|
| 2462 |
"verdict": "개선"
|
| 2463 |
},
|
| 2464 |
"no_majority": {
|
| 2465 |
+
"diff": 0.035833333333333335,
|
| 2466 |
"ci95": [
|
| 2467 |
+
0.001730887172231424,
|
| 2468 |
+
0.06993577949443525
|
| 2469 |
],
|
| 2470 |
"n": 200,
|
| 2471 |
"verdict": "개선"
|
|
|
|
| 2513 |
},
|
| 2514 |
"9b-wt-minus-qwen27b": {
|
| 2515 |
"all": {
|
| 2516 |
+
"diff": 0.030833333333333334,
|
| 2517 |
"ci95": [
|
| 2518 |
+
-0.004481904202798138,
|
| 2519 |
+
0.06614857086946481
|
| 2520 |
],
|
| 2521 |
"n": 200,
|
| 2522 |
"verdict": "차이 없음"
|
| 2523 |
},
|
| 2524 |
"no_majority": {
|
| 2525 |
+
"diff": 0.0275,
|
| 2526 |
"ci95": [
|
| 2527 |
+
-0.007352886243590425,
|
| 2528 |
+
0.06235288624359042
|
| 2529 |
],
|
| 2530 |
"n": 200,
|
| 2531 |
"verdict": "차이 없음"
|
|
|
|
| 2553 |
},
|
| 2554 |
"27b-minus-qwen27b": {
|
| 2555 |
"all": {
|
| 2556 |
+
"diff": 0.05083333333333334,
|
| 2557 |
"ci95": [
|
| 2558 |
+
0.010744829973651678,
|
| 2559 |
+
0.09092183669301501
|
| 2560 |
],
|
| 2561 |
"n": 200,
|
| 2562 |
+
"verdict": "개선"
|
| 2563 |
},
|
| 2564 |
"no_majority": {
|
| 2565 |
+
"diff": 0.05041666666666667,
|
| 2566 |
"ci95": [
|
| 2567 |
+
0.01052750632121284,
|
| 2568 |
+
0.0903058270121205
|
| 2569 |
],
|
| 2570 |
"n": 200,
|
| 2571 |
+
"verdict": "개선"
|
| 2572 |
}
|
| 2573 |
},
|
| 2574 |
"gpt-minus-qwen27b": {
|
|
|
|
| 2673 |
},
|
| 2674 |
"9b-wt-minus-gpt": {
|
| 2675 |
"all": {
|
| 2676 |
+
"diff": -0.06291666666666668,
|
| 2677 |
"ci95": [
|
| 2678 |
+
-0.09499315169791346,
|
| 2679 |
+
-0.03084018163541989
|
| 2680 |
],
|
| 2681 |
"n": 200,
|
| 2682 |
"verdict": "악화"
|
| 2683 |
},
|
| 2684 |
"no_majority": {
|
| 2685 |
+
"diff": -0.07875000000000001,
|
| 2686 |
"ci95": [
|
| 2687 |
+
-0.10858986853268658,
|
| 2688 |
+
-0.04891013146731345
|
| 2689 |
],
|
| 2690 |
"n": 200,
|
| 2691 |
"verdict": "악화"
|
|
|
|
| 2713 |
},
|
| 2714 |
"27b-minus-gpt": {
|
| 2715 |
"all": {
|
| 2716 |
+
"diff": -0.04291666666666667,
|
| 2717 |
"ci95": [
|
| 2718 |
+
-0.07504493396733448,
|
| 2719 |
+
-0.01078839936599886
|
| 2720 |
],
|
| 2721 |
"n": 200,
|
| 2722 |
"verdict": "악화"
|
| 2723 |
},
|
| 2724 |
"no_majority": {
|
| 2725 |
+
"diff": -0.05583333333333334,
|
| 2726 |
"ci95": [
|
| 2727 |
+
-0.08728971979177597,
|
| 2728 |
+
-0.02437694687489072
|
| 2729 |
],
|
| 2730 |
"n": 200,
|
| 2731 |
"verdict": "악화"
|
|
|
|
| 2833 |
},
|
| 2834 |
"9b-wt-minus-opus": {
|
| 2835 |
"all": {
|
| 2836 |
+
"diff": -0.09333333333333334,
|
| 2837 |
"ci95": [
|
| 2838 |
+
-0.12479930726319209,
|
| 2839 |
+
-0.061867359403474584
|
| 2840 |
],
|
| 2841 |
"n": 200,
|
| 2842 |
"verdict": "악화"
|
| 2843 |
},
|
| 2844 |
"no_majority": {
|
| 2845 |
+
"diff": -0.08375,
|
| 2846 |
"ci95": [
|
| 2847 |
+
-0.11190948733001013,
|
| 2848 |
+
-0.05559051266998988
|
| 2849 |
],
|
| 2850 |
"n": 200,
|
| 2851 |
"verdict": "악화"
|
|
|
|
| 2873 |
},
|
| 2874 |
"27b-minus-opus": {
|
| 2875 |
"all": {
|
| 2876 |
+
"diff": -0.07333333333333333,
|
| 2877 |
"ci95": [
|
| 2878 |
+
-0.10513831698251358,
|
| 2879 |
+
-0.04152834968415308
|
| 2880 |
],
|
| 2881 |
"n": 200,
|
| 2882 |
"verdict": "악화"
|
| 2883 |
},
|
| 2884 |
"no_majority": {
|
| 2885 |
+
"diff": -0.060833333333333336,
|
| 2886 |
"ci95": [
|
| 2887 |
+
-0.09068471198441383,
|
| 2888 |
+
-0.030981954682252844
|
| 2889 |
],
|
| 2890 |
"n": 200,
|
| 2891 |
"verdict": "악화"
|
|
|
|
| 2993 |
},
|
| 2994 |
"9b-wt-minus-gemini": {
|
| 2995 |
"all": {
|
| 2996 |
+
"diff": -0.06375,
|
| 2997 |
"ci95": [
|
| 2998 |
+
-0.10174720064845472,
|
| 2999 |
+
-0.02575279935154528
|
| 3000 |
],
|
| 3001 |
"n": 200,
|
| 3002 |
"verdict": "악화"
|
| 3003 |
},
|
| 3004 |
"no_majority": {
|
| 3005 |
+
"diff": -0.05875,
|
| 3006 |
"ci95": [
|
| 3007 |
+
-0.09346196625976708,
|
| 3008 |
+
-0.024038033740232924
|
| 3009 |
],
|
| 3010 |
"n": 200,
|
| 3011 |
"verdict": "악화"
|
|
|
|
| 3033 |
},
|
| 3034 |
"27b-minus-gemini": {
|
| 3035 |
"all": {
|
| 3036 |
+
"diff": -0.04375,
|
| 3037 |
"ci95": [
|
| 3038 |
+
-0.07778215493901924,
|
| 3039 |
+
-0.009717845060980758
|
| 3040 |
],
|
| 3041 |
"n": 200,
|
| 3042 |
"verdict": "악화"
|
| 3043 |
},
|
| 3044 |
"no_majority": {
|
| 3045 |
+
"diff": -0.035833333333333335,
|
| 3046 |
"ci95": [
|
| 3047 |
+
-0.06993577949443525,
|
| 3048 |
+
-0.001730887172231424
|
| 3049 |
],
|
| 3050 |
"n": 200,
|
| 3051 |
"verdict": "악화"
|
|
|
|
| 3139 |
"json_invalid": 0,
|
| 3140 |
"paired_9b_minus_this": {
|
| 3141 |
"all": {
|
| 3142 |
+
"diff": 0.81125,
|
| 3143 |
"ci95": [
|
| 3144 |
+
0.7708834392338799,
|
| 3145 |
+
0.8516165607661201
|
| 3146 |
],
|
| 3147 |
"n": 200,
|
| 3148 |
"verdict": "개선"
|
| 3149 |
},
|
| 3150 |
"no_majority": {
|
| 3151 |
+
"diff": 0.8354166666666667,
|
| 3152 |
"ci95": [
|
| 3153 |
+
0.7953006259012946,
|
| 3154 |
+
0.8755327074320388
|
| 3155 |
],
|
| 3156 |
"n": 200,
|
| 3157 |
"verdict": "개선"
|
|
|
|
| 3182 |
"json_invalid": 0,
|
| 3183 |
"paired_9b_minus_this": {
|
| 3184 |
"all": {
|
| 3185 |
+
"diff": 0.08875,
|
| 3186 |
"ci95": [
|
| 3187 |
+
0.051052105431661846,
|
| 3188 |
+
0.12644789456833816
|
| 3189 |
],
|
| 3190 |
"n": 200,
|
| 3191 |
"verdict": "개선"
|
| 3192 |
},
|
| 3193 |
"no_majority": {
|
| 3194 |
+
"diff": 0.07625000000000001,
|
| 3195 |
"ci95": [
|
| 3196 |
+
0.03594492540328466,
|
| 3197 |
+
0.11655507459671537
|
| 3198 |
],
|
| 3199 |
"n": 200,
|
| 3200 |
+
"verdict": "개선"
|
| 3201 |
}
|
| 3202 |
}
|
| 3203 |
}
|
figures/rubench_v1_dev200_9b.png
CHANGED
|
Git LFS Details
|
|
Git LFS Details
|
figures/rubench_v1_dev200_9b.svg
CHANGED
|
|
figures/rubench_v1_dev200_all_9b.png
CHANGED
|
Git LFS Details
|
|
Git LFS Details
|
figures/rubench_v1_dev200_all_9b.svg
CHANGED
|
|
figures/rubio_9b_header.svg
CHANGED
|
|
figures/rubio_9b_nature_fig1.png
CHANGED
|
Git LFS Details
|
|
Git LFS Details
|
figures/rubio_9b_nature_fig1.svg
CHANGED
|
|
model.safetensors-00001-of-00004.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
size 5276436216
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:5ba4673111cf684d3c4ffd44389ce19549698567c53d57f9ba9fd2570539c02a
|
| 3 |
size 5276436216
|
model.safetensors-00002-of-00004.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
size 5335161512
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:9424033399827cc66208141651af53d4bb831b063edd4274ad85d06cb5f07915
|
| 3 |
size 5335161512
|
model.safetensors-00003-of-00004.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
size 5368717440
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:042daf492a35ee17ae0bd029ab2572513bfaeb930653969dceea3c9139bd30c0
|
| 3 |
size 5368717440
|
model.safetensors-00004-of-00004.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
size 3325995712
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:e61f9f974c3c510632ec24e1e41b9c05c3af774616b2baa1726ddf52d48dd191
|
| 3 |
size 3325995712
|