benchmarks: GPQA Diamond 198Q = 88.38% (POE v6 — surpasses naive Union upper)
Browse files
README.md
CHANGED
|
@@ -105,16 +105,21 @@ curl http://localhost:8000/v1/models
|
|
| 105 |
### Option B — Manual launch (2 × B200 / H100, FP8)
|
| 106 |
|
| 107 |
```bash
|
| 108 |
-
#
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 109 |
CUDA_VISIBLE_DEVICES=0 VLLM_DP_MASTER_PORT=45011 \
|
| 110 |
-
vllm serve
|
| 111 |
--port 8021 --served-model-name darwin-28r \
|
| 112 |
--quantization fp8 --enforce-eager \
|
| 113 |
--limit-mm-per-prompt '{"image":0,"video":0}' &
|
| 114 |
|
| 115 |
-
# 2. AWAXIS-Think-31B on GPU 1
|
| 116 |
CUDA_VISIBLE_DEVICES=1 VLLM_DP_MASTER_PORT=45012 \
|
| 117 |
-
vllm serve
|
| 118 |
--port 8022 --served-model-name awaxis-31b \
|
| 119 |
--quantization fp8 --enforce-eager \
|
| 120 |
--limit-mm-per-prompt '{"image":0,"video":0}' &
|
|
@@ -126,6 +131,8 @@ python gateway/server.py --port 8000 \
|
|
| 126 |
--awaxis-url http://127.0.0.1:8022/v1
|
| 127 |
```
|
| 128 |
|
|
|
|
|
|
|
| 129 |
> **Single-GPU collocation.** With FP8 quantization the combined footprint is ~30 GB. Set both backends to `CUDA_VISIBLE_DEVICES=0` and `--gpu-memory-utilization 0.45` to colocate on a single 80 GB B200 / H100.
|
| 130 |
|
| 131 |
### OpenAI-compatible call
|
|
@@ -274,9 +281,22 @@ For multiple-choice and short-answer queries, both backends produce **N = 8 samp
|
|
| 274 |
```
|
| 275 |
Darwin-60B-DUO/
|
| 276 |
├── README.md ← this model card
|
|
|
|
| 277 |
├── config.json ← DUO configuration & orchestration metadata
|
| 278 |
├── tokenizer_info.json ← constituent tokenizer references
|
| 279 |
├── LICENSE ← Gemma + Apache-2.0 dual notice
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 280 |
├── gateway/
|
| 281 |
│ ├── server.py ← FastAPI OpenAI-compatible orchestrator
|
| 282 |
│ ├── router.py ← language / domain / MCQ classifier
|
|
@@ -286,9 +306,14 @@ Darwin-60B-DUO/
|
|
| 286 |
├── docker/
|
| 287 |
│ └── docker-compose.yml ← vLLM ×2 + gateway integrated launcher
|
| 288 |
└── benchmarks/
|
| 289 |
-
|
|
|
|
|
|
|
|
|
|
| 290 |
```
|
| 291 |
|
|
|
|
|
|
|
| 292 |
---
|
| 293 |
|
| 294 |
## Evaluation
|
|
@@ -304,7 +329,7 @@ Darwin-60B-DUO/
|
|
| 304 |
|
| 305 |
| Benchmark | Status |
|
| 306 |
|-----------|--------|
|
| 307 |
-
| GPQA Diamond (full 198 questions) | **
|
| 308 |
| KMMLU | TBA |
|
| 309 |
| CLIcK (Korean cultural reasoning) | TBA |
|
| 310 |
| Helmet / Ruler (long context) | TBA |
|
|
|
|
| 105 |
### Option B — Manual launch (2 × B200 / H100, FP8)
|
| 106 |
|
| 107 |
```bash
|
| 108 |
+
# Clone the repo (includes both constituent weights, ~120GB LFS)
|
| 109 |
+
git lfs install
|
| 110 |
+
git clone https://huggingface.co/FINAL-Bench/Darwin-60B-DUO
|
| 111 |
+
cd Darwin-60B-DUO
|
| 112 |
+
|
| 113 |
+
# 1. Darwin-28B-REASON on GPU 0 (from local subfolder)
|
| 114 |
CUDA_VISIBLE_DEVICES=0 VLLM_DP_MASTER_PORT=45011 \
|
| 115 |
+
vllm serve ./darwin-28r \
|
| 116 |
--port 8021 --served-model-name darwin-28r \
|
| 117 |
--quantization fp8 --enforce-eager \
|
| 118 |
--limit-mm-per-prompt '{"image":0,"video":0}' &
|
| 119 |
|
| 120 |
+
# 2. AWAXIS-Think-31B on GPU 1 (from local subfolder)
|
| 121 |
CUDA_VISIBLE_DEVICES=1 VLLM_DP_MASTER_PORT=45012 \
|
| 122 |
+
vllm serve ./awaxis-31b \
|
| 123 |
--port 8022 --served-model-name awaxis-31b \
|
| 124 |
--quantization fp8 --enforce-eager \
|
| 125 |
--limit-mm-per-prompt '{"image":0,"video":0}' &
|
|
|
|
| 131 |
--awaxis-url http://127.0.0.1:8022/v1
|
| 132 |
```
|
| 133 |
|
| 134 |
+
> **No external HF download.** Both weights are inside this repo (`./darwin-28r`, `./awaxis-31b`), so vLLM serves directly from local LFS without re-downloading from upstream repos.
|
| 135 |
+
|
| 136 |
> **Single-GPU collocation.** With FP8 quantization the combined footprint is ~30 GB. Set both backends to `CUDA_VISIBLE_DEVICES=0` and `--gpu-memory-utilization 0.45` to colocate on a single 80 GB B200 / H100.
|
| 137 |
|
| 138 |
### OpenAI-compatible call
|
|
|
|
| 281 |
```
|
| 282 |
Darwin-60B-DUO/
|
| 283 |
├── README.md ← this model card
|
| 284 |
+
├── image.png ← cover image
|
| 285 |
├── config.json ← DUO configuration & orchestration metadata
|
| 286 |
├── tokenizer_info.json ← constituent tokenizer references
|
| 287 |
├── LICENSE ← Gemma + Apache-2.0 dual notice
|
| 288 |
+
├── darwin-28r/ ← Darwin-28B-REASON full weights + tokenizer + config (~54GB)
|
| 289 |
+
│ ├── config.json
|
| 290 |
+
│ ├── tokenizer.json
|
| 291 |
+
│ ├── chat_template.jinja
|
| 292 |
+
│ ├── model.safetensors.index.json
|
| 293 |
+
│ └── model-00001-of-00012.safetensors ... model-00012-of-00012.safetensors
|
| 294 |
+
├── awaxis-31b/ ← AWAXIS-Think-31B full weights + tokenizer + config (~62.5GB)
|
| 295 |
+
│ ├── config.json
|
| 296 |
+
│ ├── tokenizer.json
|
| 297 |
+
│ ├── chat_template.jinja
|
| 298 |
+
│ ├── model.safetensors.index.json
|
| 299 |
+
│ └── model-00001-of-00002.safetensors ... model-00002-of-00002.safetensors
|
| 300 |
├── gateway/
|
| 301 |
│ ├── server.py ← FastAPI OpenAI-compatible orchestrator
|
| 302 |
│ ├── router.py ← language / domain / MCQ classifier
|
|
|
|
| 306 |
├── docker/
|
| 307 |
│ └── docker-compose.yml ← vLLM ×2 + gateway integrated launcher
|
| 308 |
└── benchmarks/
|
| 309 |
+
├── README.md ← full evaluation results
|
| 310 |
+
├── cascade_v4_32k.json ← cascade v4 raw data
|
| 311 |
+
├── recovery_v5.json ← recovery v5 raw data
|
| 312 |
+
└── poe_v6.json ← POE v6 raw data (latest)
|
| 313 |
```
|
| 314 |
|
| 315 |
+
> **Self-contained:** all model weights (both constituents) are included in this repository — `git clone` once and you have everything. Total LFS payload: **~120GB**.
|
| 316 |
+
|
| 317 |
---
|
| 318 |
|
| 319 |
## Evaluation
|
|
|
|
| 329 |
|
| 330 |
| Benchmark | Status |
|
| 331 |
|-----------|--------|
|
| 332 |
+
| GPQA Diamond (full 198 questions) | **88.38%** (175/198) — see `benchmarks/` |
|
| 333 |
| KMMLU | TBA |
|
| 334 |
| CLIcK (Korean cultural reasoning) | TBA |
|
| 335 |
| Helmet / Ruler (long context) | TBA |
|