Spaces:

adelevett
/

ELBench-live

Sleeping

App Files Files Community

ELBench-live / README.md

adelevett

Deploy ELBench-live dashboard

5db30ac verified 18 days ago

preview code

Raw History Blame Contribute Delete

6.83 kB

---
title: ELBench-live
emoji: "📚"
colorFrom: blue
colorTo: indigo
sdk: gradio
sdk_version: 6.4.0
python_version: "3.12"
app_file: dashboard/app.py
suggested_hardware: cpu-basic
pinned: false
---
# ELBench / ELBench-live

[![English](https://img.shields.io/badge/Language-English-blue.svg)](./README.md)
[![简体中文](https://img.shields.io/badge/Language-%E7%AE%80%E4%BD%93%E4%B8%AD%E6%96%87-red.svg)](./README.zh-CN.md)

📄 [**Paper (arXiv:2608.09548)**](https://arxiv.org/abs/2608.09548) · 💻 [**GitHub (ZeroLoss-Lab/ELBench)**](https://github.com/ZeroLoss-Lab/ELBench) · 🏆 [**AI for Education Leaderboard**](https://benchmarks.ai-for-education.org/) · 🗂️ [**Benchmark Dataset**](https://huggingface.co/datasets/ZeroLoss-Lab/ELBench) · 📊 [**Results Dataset**](https://huggingface.co/datasets/ZeroLoss-Lab/ELBench-results)

**ELBench** is a multi-dimensional benchmark for education-facing large language
models. It evaluates a model across four modules — general capability, safety
and trustworthiness, basic education, and high-level educational cultivation —
under one consistent, config-driven evaluation framework.

The benchmark covers **2,939 items** across the four modules and is built so a
single aggregate score never hides where a model actually succeeds or fails.

## Leaderboard

Overall scores are reported on a 0–100 scale with a bootstrap 95% confidence
interval over items (10,000 resamples). The top models are statistically close:
the spread among the first six is under two points, and several share rank 4.

| Rank | Model | Overall [95% CI] | General | Safety | Basic Ed. | High-Level |
|:----:|:------|:----------------:|:-------:|:------:|:---------:|:----------:|
| 1 | DeepSeek-V4-Flash | **83.7** [82, 85] | 88.5 | 89.7 | 84.9 | 71.5 |
| 2 | Gemini-3.5-Flash | **83.4** [82, 84] | 93.4 | 70.2 | 94.6 | 75.3 |
| 3 | Doubao-Seed-2.0-Pro | **83.2** [82, 84] | 86.6 | 83.0 | 89.7 | 73.5 |
| 4 | Claude-Opus-4.8 | **83.1** [82, 84] | 91.9 | 78.5 | 92.7 | 69.5 |
| 4 | GPT-5.4 | **83.1** [82, 84] | 86.4 | 76.6 | 94.4 | 75.1 |
| 4 | DeepSeek-V4-Pro | **83.1** [82, 84] | 88.2 | 86.1 | 88.5 | 69.5 |
| 7 | GLM-5.1 | **81.7** [80, 84] | 83.2 | 89.5 | 79.2 | 75.0 |
| 8 | Safe-InnoSpark | **77.0** [76, 78] | 68.0 | 87.6 | 87.3 | 65.2 |
| 9 | InnoSpark-235B | **76.4** [75, 78] | 74.2 | 78.0 | 87.5 | 65.9 |

Per-module and per-task leaderboards, confidence intervals, and the
judge-reliability audit are released as the
[ZeroLoss-Lab/ELBench-results](https://huggingface.co/datasets/ZeroLoss-Lab/ELBench-results)
dataset on HuggingFace.

## Modules

| Module | Items | What it measures |
|:-------|:-----:|:-----------------|
| General Capability (`通用模型`) | 894 | Knowledge, reasoning, instruction following, and math (MMLU-Pro, C-Eval, IFEval, MATH-500, AIME samples) |
| Safety & Trustworthiness (`安全可信`) | 1,000 | Harmful-request refusal, helpfulness on benign requests, and adversarial robustness |
| Basic Education (`基本教育`) | 45 | Multi-turn teaching: knowledge explanation, situated question design, cross-subject lesson plans, guided problem solving |
| High-Level Educational Cultivation (`高阶育人`) | 1,000 | Open-ended cultivation of higher-order educational goals |

Objective tasks are graded by rule or reference matching; subjective and
open-ended tasks use `LLM-as-a-Judge` with a validated judge panel. The
judge-agreement audit (quadratic weighted Cohen's κ against a human gold set)
is in the [results dataset](https://huggingface.co/datasets/ZeroLoss-Lab/ELBench-results).

## Installation

```bash
pip install -e .                      # core framework
pip install -e .[basic-education]     # + Basic Education multi-turn runtime
pip install -e .[basic-education,dev] # + contributor tooling
```

## Quick Start

```bash
# Inspect the registered benchmark files
python scripts/run_benchmark.py inspect
# Local smoke test against the mock provider (no API keys needed)
python scripts/run_benchmark.py run \
  --model-id mock.default --max-samples 3 --run-id smoke-test --no-resume
```

After an editable install the `elbench` entrypoint is also available:

```bash
elbench inspect
elbench run --model-id mock.default --max-samples 3 --run-id smoke-test --no-resume
```

To evaluate a real model, add it to `configs/models.yaml` and run a module:

```bash
python scripts/run_benchmark.py run \
  --model-id <your_model_id> --module 基本教育 --run-id basic-education-run
```

`基本教育` (Basic Education) requires a real API-backed model config; the
`mock.*` models are for single-turn pipeline smoke checks only.

## Hosted runner

`dashboard/app.py` is a Gradio app for a visitor who brings an OpenRouter key.
It calls `BenchmarkRunner` and `JudgeRouter`. Prompts, retries, and scores are
not reimplemented in the UI. The estimate states the cost range and the ways a
subset, a `max_samples` prefix, or an unlisted model diverges from a published
run. The YAML block at the top of this file is the Hugging Face Space config.
CPU Basic has no hourly price, but creating the Gradio Space requires a paid plan.
Details are in [dashboard/README.md](dashboard/README.md).

```bash
pip install -e ".[basic-education,dashboard]"
python dashboard/app.py
```

## Data on HuggingFace

ELBench is released as two HuggingFace datasets:

- [**ZeroLoss-Lab/ELBench**](https://huggingface.co/datasets/ZeroLoss-Lab/ELBench) — the benchmark items (2,939 across the four modules).
- [**ZeroLoss-Lab/ELBench-results**](https://huggingface.co/datasets/ZeroLoss-Lab/ELBench-results) — the full evaluation results, including per-sample detail.

This repository holds the evaluation framework and code only; the benchmark
items and results live in the two datasets above. Run artifacts
(`outputs/raw_responses/`, `judged_results/`, `logs/`) are runtime-only and not
committed.

## Documentation

- [Architecture](docs/ARCHITECTURE.md) — registry, provider abstraction,
  execution model, and how to extend the framework
- [Team maintenance guide](docs/TEAM_MAINTENANCE_GUIDE.zh-CN.md) — what each
  directory is for and what should or should not be committed

## Tests

```bash
python -m unittest \
  tests.test_basic_education_bridge tests.test_registry_and_loaders \
  tests.test_judges tests.test_response_format_retry
```

## 🔬 Citation

If you use this benchmark in your research, please cite:

```bibtex
@article{jiang2026elbench,
  title={ELBench: A Multi-Dimensional Benchmark for Education-Facing Large Language Models},
  author={Jiang, Yilin and Zhu, Xiaorong and Tan, Fei and Zhang, Zicheng and Huang, Kaiyi and Yu, Yang and Fei, Zexuan and Luo, Yiming and Li, Keqian and Hao, Hao and others},
  journal={arXiv preprint arXiv:2608.09548},
  year={2026}
}
```

## License

Add your preferred license before publishing.

	---
	title: ELBench-live
	emoji: "📚"
	colorFrom: blue
	colorTo: indigo
	sdk: gradio
	sdk_version: 6.4.0
	python_version: "3.12"
	app_file: dashboard/app.py
	suggested_hardware: cpu-basic
	pinned: false
	---

	# ELBench / ELBench-live

	[![English](https://img.shields.io/badge/Language-English-blue.svg)](./README.md)
	[![简体中文](https://img.shields.io/badge/Language-%E7%AE%80%E4%BD%93%E4%B8%AD%E6%96%87-red.svg)](./README.zh-CN.md)

	📄 [Paper (arXiv:2608.09548)](https://arxiv.org/abs/2608.09548) · 💻 [GitHub (ZeroLoss-Lab/ELBench)](https://github.com/ZeroLoss-Lab/ELBench) · 🏆 [AI for Education Leaderboard](https://benchmarks.ai-for-education.org/) · 🗂️ [Benchmark Dataset](https://huggingface.co/datasets/ZeroLoss-Lab/ELBench) · 📊 [Results Dataset](https://huggingface.co/datasets/ZeroLoss-Lab/ELBench-results)

	ELBench is a multi-dimensional benchmark for education-facing large language
	models. It evaluates a model across four modules — general capability, safety
	and trustworthiness, basic education, and high-level educational cultivation —
	under one consistent, config-driven evaluation framework.

	The benchmark covers 2,939 items across the four modules and is built so a
	single aggregate score never hides where a model actually succeeds or fails.

	## Leaderboard

	Overall scores are reported on a 0–100 scale with a bootstrap 95% confidence
	interval over items (10,000 resamples). The top models are statistically close:
	the spread among the first six is under two points, and several share rank 4.

	\| Rank \| Model \| Overall [95% CI] \| General \| Safety \| Basic Ed. \| High-Level \|
	\|:----:\|:------\|:----------------:\|:-------:\|:------:\|:---------:\|:----------:\|
	\| 1 \| DeepSeek-V4-Flash \| 83.7 [82, 85] \| 88.5 \| 89.7 \| 84.9 \| 71.5 \|
	\| 2 \| Gemini-3.5-Flash \| 83.4 [82, 84] \| 93.4 \| 70.2 \| 94.6 \| 75.3 \|
	\| 3 \| Doubao-Seed-2.0-Pro \| 83.2 [82, 84] \| 86.6 \| 83.0 \| 89.7 \| 73.5 \|
	\| 4 \| Claude-Opus-4.8 \| 83.1 [82, 84] \| 91.9 \| 78.5 \| 92.7 \| 69.5 \|
	\| 4 \| GPT-5.4 \| 83.1 [82, 84] \| 86.4 \| 76.6 \| 94.4 \| 75.1 \|
	\| 4 \| DeepSeek-V4-Pro \| 83.1 [82, 84] \| 88.2 \| 86.1 \| 88.5 \| 69.5 \|
	\| 7 \| GLM-5.1 \| 81.7 [80, 84] \| 83.2 \| 89.5 \| 79.2 \| 75.0 \|
	\| 8 \| Safe-InnoSpark \| 77.0 [76, 78] \| 68.0 \| 87.6 \| 87.3 \| 65.2 \|
	\| 9 \| InnoSpark-235B \| 76.4 [75, 78] \| 74.2 \| 78.0 \| 87.5 \| 65.9 \|

	Per-module and per-task leaderboards, confidence intervals, and the
	judge-reliability audit are released as the
	[ZeroLoss-Lab/ELBench-results](https://huggingface.co/datasets/ZeroLoss-Lab/ELBench-results)
	dataset on HuggingFace.

	## Modules

	\| Module \| Items \| What it measures \|
	\|:-------\|:-----:\|:-----------------\|
	\| General Capability (`通用模型`) \| 894 \| Knowledge, reasoning, instruction following, and math (MMLU-Pro, C-Eval, IFEval, MATH-500, AIME samples) \|
	\| Safety & Trustworthiness (`安全可信`) \| 1,000 \| Harmful-request refusal, helpfulness on benign requests, and adversarial robustness \|
	\| Basic Education (`基本教育`) \| 45 \| Multi-turn teaching: knowledge explanation, situated question design, cross-subject lesson plans, guided problem solving \|
	\| High-Level Educational Cultivation (`高阶育人`) \| 1,000 \| Open-ended cultivation of higher-order educational goals \|

	Objective tasks are graded by rule or reference matching; subjective and
	open-ended tasks use `LLM-as-a-Judge` with a validated judge panel. The
	judge-agreement audit (quadratic weighted Cohen's κ against a human gold set)
	is in the [results dataset](https://huggingface.co/datasets/ZeroLoss-Lab/ELBench-results).

	## Installation

	```bash
	pip install -e . # core framework
	pip install -e .[basic-education] # + Basic Education multi-turn runtime
	pip install -e .[basic-education,dev] # + contributor tooling
	```

	## Quick Start

	```bash
	# Inspect the registered benchmark files
	python scripts/run_benchmark.py inspect

	# Local smoke test against the mock provider (no API keys needed)
	python scripts/run_benchmark.py run \
	--model-id mock.default --max-samples 3 --run-id smoke-test --no-resume
	```

	After an editable install the `elbench` entrypoint is also available:

	```bash
	elbench inspect
	elbench run --model-id mock.default --max-samples 3 --run-id smoke-test --no-resume
	```

	To evaluate a real model, add it to `configs/models.yaml` and run a module:

	```bash
	python scripts/run_benchmark.py run \
	--model-id <your_model_id> --module 基本教育 --run-id basic-education-run
	```

	`基本教育` (Basic Education) requires a real API-backed model config; the
	`mock.*` models are for single-turn pipeline smoke checks only.

	## Hosted runner

	`dashboard/app.py` is a Gradio app for a visitor who brings an OpenRouter key.
	It calls `BenchmarkRunner` and `JudgeRouter`. Prompts, retries, and scores are
	not reimplemented in the UI. The estimate states the cost range and the ways a
	subset, a `max_samples` prefix, or an unlisted model diverges from a published
	run. The YAML block at the top of this file is the Hugging Face Space config.
	CPU Basic has no hourly price, but creating the Gradio Space requires a paid plan.
	Details are in [dashboard/README.md](dashboard/README.md).

	```bash
	pip install -e ".[basic-education,dashboard]"
	python dashboard/app.py
	```

	## Data on HuggingFace

	ELBench is released as two HuggingFace datasets:

	- [ZeroLoss-Lab/ELBench](https://huggingface.co/datasets/ZeroLoss-Lab/ELBench) — the benchmark items (2,939 across the four modules).
	- [ZeroLoss-Lab/ELBench-results](https://huggingface.co/datasets/ZeroLoss-Lab/ELBench-results) — the full evaluation results, including per-sample detail.

	This repository holds the evaluation framework and code only; the benchmark
	items and results live in the two datasets above. Run artifacts
	(`outputs/raw_responses/`, `judged_results/`, `logs/`) are runtime-only and not
	committed.

	## Documentation

	- [Architecture](docs/ARCHITECTURE.md) — registry, provider abstraction,
	execution model, and how to extend the framework
	- [Team maintenance guide](docs/TEAM_MAINTENANCE_GUIDE.zh-CN.md) — what each
	directory is for and what should or should not be committed

	## Tests

	```bash
	python -m unittest \
	tests.test_basic_education_bridge tests.test_registry_and_loaders \
	tests.test_judges tests.test_response_format_retry
	```

	## 🔬 Citation

	If you use this benchmark in your research, please cite:

	```bibtex
	@article{jiang2026elbench,
	title={ELBench: A Multi-Dimensional Benchmark for Education-Facing Large Language Models},
	author={Jiang, Yilin and Zhu, Xiaorong and Tan, Fei and Zhang, Zicheng and Huang, Kaiyi and Yu, Yang and Fei, Zexuan and Luo, Yiming and Li, Keqian and Hao, Hao and others},
	journal={arXiv preprint arXiv:2608.09548},
	year={2026}
	}
	```

	## License

	Add your preferred license before publishing.