docs/ARCHITECTURE.md

ELBench Architecture

This document describes the engineering design of ELBench: the registry, the provider abstraction, the execution model, and how to extend the framework. For an overview of the benchmark and the leaderboard, see the top-level README.

ELBench is built as an engineering system, not a one-off script. Four benchmark modules are kept in one consistent framework:

• 安全可信 / Safety & Trustworthiness
• 高阶育人 / High-Level Educational Cultivation
• 基本教育 / Basic Education
• 通用模型 / General Capability

Principles

• Registry-driven loading instead of hardcoded file logic
• Provider adapters instead of provider-specific branches in the runner
• Rule-based judging for objective tasks
• LLM-as-a-Judge for subjective and open-ended tasks
• Stable machine-readable outputs, logs, checkpoints, and summaries
• Resume support after interruption
• Configurable concurrency with a default global cap of 100

What Is Actually Wired Today

Registered benchmark files

configs/file_registry.yaml currently resolves these files:

• 安全拒答.jsonl
• 安全引导.jsonl
• 安全回答.jsonl
• SATAs.xlsx
• adversarial_prompts.xlsx
• 高阶育人-edu.jsonl
• 高阶育人-omni.jsonl
• knowledge.yaml
• question.yaml
• cross.yaml
• config_guided_task.yaml
• mmlu_pro_sampled.jsonl
• ceval_sampled.jsonl
• ifeval_sampled.jsonl
• math_500_sampled.jsonl
• aime24_sampled.jsonl
• aime25.jsonl
• aime26.jsonl
• gsm8k_sampled.jsonl

Files are only considered active benchmark inputs after they are registered. Data files not present in configs/file_registry.yaml are not part of the runnable benchmark.

Current judge strategy

Objective tasks use rule or reference matching:

• SATAs.xlsx
• 高阶育人-omni.jsonl
• mmlu_pro_sampled.jsonl
• ceval_sampled.jsonl
• ifeval_sampled.jsonl
• math_500_sampled.jsonl
• aime24_sampled.jsonl
• aime25.jsonl
• aime26.jsonl
• gsm8k_sampled.jsonl

Subjective or open-ended tasks use LLM-as-a-Judge:

• 安全拒答.jsonl
• 安全引导.jsonl
• 安全回答.jsonl
• adversarial_prompts.xlsx
• 高阶育人-edu.jsonl

The repository still treats external provider integration as pluggable. mock.default and mock.judge are included so the full pipeline can be validated locally without real API access.

Repository Layout

.
├── configs/
│   ├── app.yaml
│   ├── basic_education.yaml
│   ├── field_mappings.yaml
│   ├── file_registry.yaml
│   ├── judges.yaml
│   ├── models.yaml
│   ├── module_registry.yaml
│   └── providers.yaml
├── data/
│   └── benchmark_root/
│       ├── 安全可信/
│       ├── 高阶育人/
│       ├── 基本教育/
│       └── 通用/
├── docs/
│   └── TEAM_MAINTENANCE_GUIDE.zh-CN.md
├── results/
├── scripts/
│   └── run_benchmark.py
├── src/
│   └── elbench/
│       ├── basic_education_runtime/
│       ├── cli.py
│       ├── config/
│       ├── execution/
│       ├── judges/
│       ├── loaders/
│       ├── persistence/
│       ├── providers/
│       ├── registry/
│       ├── schemas/
│       ├── summary/
│       └── utils/
└── tests/

scripts/run_benchmark.py is intentionally a thin entrypoint. The actual benchmark logic lives under src/elbench/.

Core Architecture

Registry and schema adaptation

• configs/file_registry.yaml maps files to module, subset, task, loader, and canonical name.
• configs/field_mappings.yaml maps raw dataset fields into ELBench's unified internal schema.
• JSONL, XLSX, and basic-education YAML inputs are normalized into the same Sample structure.

Unified sample/result schema

Loaded records are normalized into a stable schema with fields such as:

• sample_id
• source_file
• module
• subset
• task
• dimension
• prompt
• reference
• metadata

Execution output includes:

• provider_name
• model_id
• model_name
• model_response
• judge_result
• score
• judge_reason
• judge_metadata
• latency_ms
• retry_count
• timestamp

Provider abstraction

• ModelClient is the common interface for model execution.
• Provider-specific API differences are isolated under src/elbench/providers/.
• The main runner does not care whether a model comes from OpenAI, Anthropic, Gemini, or another vendor once the adapter exists.

Execution, concurrency, and recovery

The runner supports:

• global concurrency control
• provider/model-level concurrency limits
• retry with exponential backoff
• failure logging
• checkpoint-based resume
• raw-response persistence
• judged-result persistence
• summary generation

Default global max concurrency is 100 in configs/app.yaml.

Basic Education runtime

基本教育 differs from the other modules only in execution style, not in data governance:

• benchmark data stays under data/benchmark_root/基本教育/
• file discovery still goes through configs/file_registry.yaml
• runtime execution is handled by src/elbench/execution/basic_education.py
• the internal runtime implementation lives under src/elbench/basic_education_runtime/

This internal runtime was adapted from the ELMES project, but it is now packaged and maintained as part of ELBench itself.

Output Layout

outputs/
├── raw_responses/
├── judged_results/
├── summaries/
└── logs/

These directories are runtime artifacts and are intentionally ignored by Git. Curated leaderboards and audit summaries are committed under results/.

Configuration Surface

configs/models.yaml

Concrete evaluated-model and judge-model instances, including timeout, retry, rate limits, and provider kwargs.

configs/providers.yaml

Provider adapter registry plus default provider capabilities. Current built-in adapters:

• mock
• openai_compatible

configs/judges.yaml

Task-to-judge routing:

• rule
• llm

configs/basic_education.yaml

Scenario-level runtime settings for the internal basic education runtime. This file does not replace the registry; it complements it.

Extending ELBench

Add a new provider

1. Implement an adapter under src/elbench/providers/.
2. Register it in configs/providers.yaml.
3. Add concrete models in configs/models.yaml.

Add a new benchmark file

1. Put the data file under data/benchmark_root/.
2. Register it in configs/file_registry.yaml.
3. Add field mappings in configs/field_mappings.yaml.
4. Route judging in configs/judges.yaml.
5. Add or reuse tests.

Add a future module

ELBench already reserves space for further modules. The intended path remains:

1. add module registration
2. add file registration
3. add field mappings
4. add judge routing
5. add task-specific logic only if configuration cannot express the behavior

No main-runner rewrite should be required.

Maintenance

The canonical team maintenance guide is docs/TEAM_MAINTENANCE_GUIDE.zh-CN.md. That document explains what each maintained directory is for, which paths are runtime-only, and what should or should not be committed.

Status

Implemented:

• config system
• file registry
• JSONL/XLSX/YAML loaders
• unified sample/result schema
• provider abstraction
• concurrent runner
• retries and checkpoint resume
• raw/judged/summary/log outputs
• rule judges for objective tasks
• LLM-as-a-Judge execution path for subjective tasks
• built-in basic-education bridge on top of the internal runtime module

Still intentionally unfinished:

• production-grade adapters for every external vendor
• finalized subjective judge rubrics and prompts
• complete future-module integration beyond the current registered files