Buckets:

gijl
/

lv-storage

Files

xet

gijl/lv-storage / llama.cpp /examples /llama-eval /README.md

gijl

15 days ago

preview code

download

raw

13.7 kB

# llama-eval

Simple evaluation tool for llama.cpp with support for multiple datasets.

For a full description, usage examples, and sample results, see:

- [PR 21152](https://github.com/ggml-org/llama.cpp/pull/21152)

## Quick start

```bash
# Single server
python3 llama-eval.py \
  --server http://localhost:8033 \
  --model my-model \
  --dataset gsm8k --n_cases 100 \
  --grader-type regex --threads 32
# Multiple servers (comma-separated URLs and thread counts)
python3 llama-eval.py \
  --server http://server1:8033,http://server2:8033 \
  --server-name server1,server2 \
  --threads 16,16 \
  --dataset aime2025 --n_cases 240 \
  --grader-type regex
```

### Authentication

A server started with `--api-key` (or any OpenAI-compatible endpoint behind a
token) needs one here too. `--api-key` is sent as `Authorization: Bearer` on
every eval request; it falls back to `$LLAMA_API_KEY`, which keeps the secret
out of your shell history and off the process list.

```bash
# One key for every server
python3 llama-eval.py --server http://localhost:8033 --api-key sk-... ...
# From the environment
export LLAMA_API_KEY=sk-...
python3 llama-eval.py --server http://localhost:8033 ...
# One key per server, in --server order
python3 llama-eval.py \
  --server http://server1:8033,http://server2:8033 \
  --api-key sk-one,sk-two --threads 16,16 ...
```

The LLM grader authenticates too. It reuses the key of the server it runs on,
so `--grader-type llm` without `--grader-server` needs nothing extra; a grader
on a separate endpoint takes `--grader-api-key`.

Endpoints on an opencode host additionally get an `x-opencode-session` header,
generated fresh for every task. An agentic episode sends the same id on all of
its turns, so opencode sees one session per task instead of one per request;
other servers never see the header.

## Agentic suite

`--dataset agentic` is a different shape from the other code suites. It runs
**SACB** (Simple Agent Coding Benchmark), hosted at
[`ilintar/SACB`](https://huggingface.co/datasets/ilintar/SACB) -- 60 tasks over
three hand-written repositories, in Python and TypeScript. Rather than
answering one prompt, the model is given a small but real repository, a
deliberately narrow tool set, and a bug report, and drives its own multi-turn
conversation until it declares itself finished. It is then graded by running a
test suite it was never allowed to see.

```bash
python3 llama-eval.py \
  --server http://localhost:8033 --model my-model \
  --dataset agentic --threads 1 --temperature 0
```

### The tools

| tool | purpose |
|---|---|
| `list_files` | enumerate the tree |
| `read_file` | read, with 1-based line numbers and optional line-range narrowing |
| `search` | regular-expression search across the repository |
| `edit_lines` | replace an inclusive line range |
| `edit_replace` | replace an exact snippet, which must be unique |
| `write_file` | write a whole file |
| `lint` | run the language's linter and type checker |
| `finish` | declare the change complete |

Three constraints are deliberate:

**There is no shell and no way to run the tests.** The only feedback channel is
`lint`. A syntax or type error can be driven out mechanically; a logic error has
to be reasoned about. A harness that lets an agent iterate against the test
suite measures something closer to search than to understanding.

**There are two edit tools**, one line-addressed and one content-addressed.
Which one a model reaches for, and whether it keeps line numbers straight after
its own earlier edit, is itself a signal.

**The tests never touch the working tree.** They are held outside it and copied
into a throwaway copy only after the agent has stopped, so they can be neither
read nor edited. Every tool path is resolved and confinement-checked against the
repository root, so a symlink planted inside the tree cannot reach outside it.

### Scoring

A task is **resolved** when every test that was failing now passes *and* nothing
that was already passing broke. Both sets are derived by running the tests twice
when the corpus is built -- once against the defective tree and once against the
reference fix -- never hand-written, so a task whose defect no test exercises is
rejected rather than shipped.

Because many tasks contain several independent defects, the fraction of target
tests passed is reported alongside, which separates "fixed two of the three
faults" from "changed nothing". Process metrics are reported too: turns, tool
calls, rejected calls, edits, and the peak conversation size.

### Dependencies

Nothing here is installed unless you select this suite, and then only for the
language you actually run. `--agentic-lang python` provisions ruff and mypy;
`--agentic-lang typescript` provisions the TypeScript compiler and uses node's
own test runner. Neither touches the other, and no other suite touches either.

### Useful flags

| flag | meaning |
|---|---|
| `--agentic-lang` | restrict to one language, and provision only its toolchain |
| `--agentic-max-turns` | give up after this many assistant turns (default 50) |
| `--agentic-max-tokens` | cap generation per turn (default: no cap); bounds a model that falls into a repetition loop, but also cuts reasoning off mid-thought. Truncated turns are counted as `truncated_turns` |
| `--agentic-context-limit` | abandon an episode once its context exceeds this many tokens |
| `--agentic-transcripts DIR` | save each episode's full message history to `DIR/<task_id>.json`, plus a `DIR/<task_id>.html` showing every reasoning block, reply, tool call and tool result; the main report links to it |


## Code suites (HumanEval, ClassEval)

`--dataset humaneval` runs the 164 OpenAI HumanEval problems; `--dataset
classeval` runs the 100 ClassEval class-generation tasks. Unlike the other
suites these are not graded by matching text: the model's reply is parsed for a
```` ```python ```` block, that code is combined with the task's own unit tests,
and the result is **executed**.

ClassEval asks for a whole class rather than one function, so it is scored the
way upstream does - **class level** (every one of the task's tests passes, which
is what feeds pass@k) and **method level** (what fraction of all 2196 test
methods passed). The second number separates "generated nothing usable" from
"got most of the class right", and both are printed at the end of a run.

Give it more room to generate than HumanEval - a class with ten methods does not
fit in a short budget:

```bash
python3 llama-eval.py \
  --server http://localhost:8033 --model my-model \
  --dataset classeval --threads 8 --n_predict 4096 --temperature 0
```

```bash
# pass@1, greedy
python3 llama-eval.py \
  --server http://localhost:8033 --model my-model \
  --dataset humaneval --threads 8 --n_predict 1024 --temperature 0
# pass@10 -- each extra multiple of 164 is another sample per problem
python3 llama-eval.py \
  --server http://localhost:8033 --model my-model \
  --dataset humaneval --n_cases 1640 --temperature 0.8 --top-p 0.95
```

`--grader-type exec` is the default for code suites, so neither it nor
`--dataset-source` normally needs to be passed.

### Execution sandbox - no Docker required

Generated code is untrusted, but these suites need far less than a container to
run safely. **The package set is derived from the dataset's own imports at
startup**, not hardcoded and not taken from the upstream `requirements.txt` -
which for ClassEval lists `torch`, `openai` and `transformers` that its *tests*
never touch. So each suite installs exactly what it needs:

| suite | third-party packages |
|---|---|
| HumanEval | **none** - every problem imports only stdlib (`typing, math, random, copy, string, collections, hashlib, re`) |
| ClassEval | 13: `beautifulsoup4, gensim, lxml, netifaces, nltk, numpy, openpyxl, pandas, pillow, pypdf2, python-docx, reportlab, scipy` |

Two of ClassEval's are *declared* rather than derived, because no import
statement mentions them: `lxml` is selected by string
(`BeautifulSoup(html, 'lxml')`, ClassEval_44 - without it that task silently
drops from 23/23 to 7/23), and `scipy` is a hard gensim requirement. Note also
that `PyPDF2` must **not** be modernised to `pypdf`: ClassEval_69's test imports
`PdfFileReader`, a name pypdf removed.

The venv is cached under `~/.cache/llama-eval/` (override with
`LLAMA_EVAL_CACHE`) and rebuilt only when the package set, interpreter version
or provisioning steps change; force one with `--rebuild-sandbox`. `uv` is used
when available, which makes even the ClassEval venv build in seconds.

**Interpreter selection.** A suite can prefer a different Python than the one
running the tool, and declares it: ClassEval asks for 3.13, because gensim
publishes no wheel past cp313 and its generated C no longer compiles on 3.14
(it still uses `PyLongObject.ob_digit`). That interpreter is found automatically,
including via `uv python`; `--sandbox-python` overrides it.

3.14 is fully supported anyway. ClassEval uses exactly two gensim APIs
(`utils.decode_htmlentities`, `matutils.unitvec`), both pure Python upstream, so
on 3.14+ the suite drops gensim from the install set and provisions a small shim
instead. Both paths score **100/100**.

**Offline provisioning.** A suite that would otherwise fetch data at test time
declares a provisioning step that runs once at venv build time, while the
network is still up. ClassEval uses this to pre-seed exactly three nltk corpora
(`punkt_tab`, `averaged_perceptron_tagger_eng`, `wordnet`, ~33 MB), so the tests
themselves run with no network at all. Those are the *current* names - the
dataset asks for `punkt` and `averaged_perceptron_tagger`, which nltk has since
renamed, and seeding the legacy names too is not harmless: with a network
available nltk considers them satisfied and re-fetches the modern ones anyway.
The corpora are written *inside* the venv, not `~/nltk_data`, because `$HOME` is
a tmpfs under the strongest isolation tier.

Each candidate program runs in a throwaway directory as its own process group,
under the strongest isolation the host offers:

| tier | requires | isolates |
|---|---|---|
| `bwrap` | bubblewrap installed | read-only root, tmpfs `/tmp` and `$HOME`, **no network** |
| `unshare` | `unshare --map-current-user` | **no network** |
| `unshare-root` | any `unshare -r` | no network, but **runs as uid 0** - see below |
| `rlimit` | always available | cpu/memory/file-size/process caps only |

**Candidate code must not run as root**, which is why `unshare -r` is a
last-resort tier that prints a warning. Root bypasses file permission bits, so a
test asserting that writing to a `chmod 0444` file *fails* instead sees it
succeed - ClassEval_50 scores 14/16 under uid 0 and 16/16 under uid 1000, with
nothing in the output to suggest the sandbox caused it. Install bubblewrap.
All tiers additionally cap CPU time, address space (4 GiB), file size and core
dumps, and kill the whole process group on timeout (`--exec-timeout`, default
15 s - raise it for ClassEval, whose tests do real work). Writes to the
throwaway working directory are allowed, which is all the file-handling tasks in
ClassEval need.
Note that the weakest tier does not contain filesystem access. If you are
scoring output from a model you do not trust, check the tier printed at startup.
### Reproducibility
Several tasks draw their test inputs from the *global, unseeded* RNG, so a
verdict is not reproducible run to run - which adds noise exactly when you are
comparing two models or two quantisations. `--exec-seed` (default `0`) seeds the
RNG before each test; pass `--exec-seed -1` for the stock upstream behaviour.
This does not change what the model is asked to compute.
It matters more than it sounds. HumanEval/38, /50 and /53 are affected, and
ClassEval_58 (duplicate mine placement) fails about 30% of the time on its own
*reference* solution - measured here at 21/30 passes unseeded versus **30/30
seeded**.

### Dataset fidelity

The canonical dataset runs **unmodified on current Python** - all 164 reference
solutions pass on CPython 3.14 with no patches, so no forked or re-hosted copy
is needed. `--dataset-source` accepts a local `.jsonl` or a HuggingFace repo id
if you do want to point at a modified copy.

HumanEval does contain a handful of long-standing defects where the *prompt*
contradicts the graded test (notably HumanEval/47, /116 and /148). Those
penalise a model that reads the docstring carefully, but correcting them makes
the benchmark marginally easier and the resulting scores no longer comparable
with published HumanEval numbers, so nothing is patched by default.

ClassEval is different: the upstream data has defects that cap the achievable
score below 100% for reasons unrelated to the model, so `--dataset classeval`
defaults to a **patched fork**,
[`ilintar/ClassEval`](https://huggingface.co/datasets/ilintar/ClassEval).
Six edits over five tasks lift the reference solutions from 96/100 classes and
2182/2196 test methods to **100/100 and 2196/2196**. Three repair tests nothing
could pass:

- **ClassEval_17** was a time bomb - a hard-coded 2024 date checked against
  `datetime.now()`, unpassable since January 2024.
- **ClassEval_48** asserted `get_hostname('0.0.0.0') == 'LAPTOP-2CS86KUM'`, the
  dataset author's own machine name.
- **ClassEval_31** compared floats with `assertEqual`, failing on a 4-ULP numpy
  difference.
The other three fix only the *reference* solutions (NumPy 2.0 removals in
ClassEval_51, a 30%-flaky mine generator in ClassEval_58) and cannot affect a
model's score. No `skeleton` was touched, so what the model is asked to write is
byte-identical to upstream. Pass `--dataset-source FudanSELab/ClassEval` for the
unpatched original.

Xet Storage Details

Size:: 13.7 kB
Xet hash:: 164f2a3c898c04f4f256325b674834276e9f92cddddce19756e235bb93f271b0

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.

	# llama-eval

	Simple evaluation tool for llama.cpp with support for multiple datasets.

	For a full description, usage examples, and sample results, see:

	- [PR 21152](https://github.com/ggml-org/llama.cpp/pull/21152)

	## Quick start

	```bash
	# Single server
	python3 llama-eval.py \
	--server http://localhost:8033 \
	--model my-model \
	--dataset gsm8k --n_cases 100 \
	--grader-type regex --threads 32

	# Multiple servers (comma-separated URLs and thread counts)
	python3 llama-eval.py \
	--server http://server1:8033,http://server2:8033 \
	--server-name server1,server2 \
	--threads 16,16 \
	--dataset aime2025 --n_cases 240 \
	--grader-type regex
	```

	### Authentication

	A server started with `--api-key` (or any OpenAI-compatible endpoint behind a
	token) needs one here too. `--api-key` is sent as `Authorization: Bearer` on
	every eval request; it falls back to `$LLAMA_API_KEY`, which keeps the secret
	out of your shell history and off the process list.

	```bash
	# One key for every server
	python3 llama-eval.py --server http://localhost:8033 --api-key sk-... ...

	# From the environment
	export LLAMA_API_KEY=sk-...
	python3 llama-eval.py --server http://localhost:8033 ...

	# One key per server, in --server order
	python3 llama-eval.py \
	--server http://server1:8033,http://server2:8033 \
	--api-key sk-one,sk-two --threads 16,16 ...
	```

	The LLM grader authenticates too. It reuses the key of the server it runs on,
	so `--grader-type llm` without `--grader-server` needs nothing extra; a grader
	on a separate endpoint takes `--grader-api-key`.

	Endpoints on an opencode host additionally get an `x-opencode-session` header,
	generated fresh for every task. An agentic episode sends the same id on all of
	its turns, so opencode sees one session per task instead of one per request;
	other servers never see the header.

	## Agentic suite

	`--dataset agentic` is a different shape from the other code suites. It runs
	SACB (Simple Agent Coding Benchmark), hosted at
	[`ilintar/SACB`](https://huggingface.co/datasets/ilintar/SACB) -- 60 tasks over
	three hand-written repositories, in Python and TypeScript. Rather than
	answering one prompt, the model is given a small but real repository, a
	deliberately narrow tool set, and a bug report, and drives its own multi-turn
	conversation until it declares itself finished. It is then graded by running a
	test suite it was never allowed to see.

	```bash
	python3 llama-eval.py \
	--server http://localhost:8033 --model my-model \
	--dataset agentic --threads 1 --temperature 0
	```

	### The tools

	\| tool \| purpose \|
	\|---\|---\|
	\| `list_files` \| enumerate the tree \|
	\| `read_file` \| read, with 1-based line numbers and optional line-range narrowing \|
	\| `search` \| regular-expression search across the repository \|
	\| `edit_lines` \| replace an inclusive line range \|
	\| `edit_replace` \| replace an exact snippet, which must be unique \|
	\| `write_file` \| write a whole file \|
	\| `lint` \| run the language's linter and type checker \|
	\| `finish` \| declare the change complete \|

	Three constraints are deliberate:

	There is no shell and no way to run the tests. The only feedback channel is
	`lint`. A syntax or type error can be driven out mechanically; a logic error has
	to be reasoned about. A harness that lets an agent iterate against the test
	suite measures something closer to search than to understanding.

	There are two edit tools, one line-addressed and one content-addressed.
	Which one a model reaches for, and whether it keeps line numbers straight after
	its own earlier edit, is itself a signal.

	The tests never touch the working tree. They are held outside it and copied
	into a throwaway copy only after the agent has stopped, so they can be neither
	read nor edited. Every tool path is resolved and confinement-checked against the
	repository root, so a symlink planted inside the tree cannot reach outside it.

	### Scoring

	A task is resolved when every test that was failing now passes and nothing
	that was already passing broke. Both sets are derived by running the tests twice
	when the corpus is built -- once against the defective tree and once against the
	reference fix -- never hand-written, so a task whose defect no test exercises is
	rejected rather than shipped.

	Because many tasks contain several independent defects, the fraction of target
	tests passed is reported alongside, which separates "fixed two of the three
	faults" from "changed nothing". Process metrics are reported too: turns, tool
	calls, rejected calls, edits, and the peak conversation size.

	### Dependencies

	Nothing here is installed unless you select this suite, and then only for the
	language you actually run. `--agentic-lang python` provisions ruff and mypy;
	`--agentic-lang typescript` provisions the TypeScript compiler and uses node's
	own test runner. Neither touches the other, and no other suite touches either.

	### Useful flags

	\| flag \| meaning \|
	\|---\|---\|
	\| `--agentic-lang` \| restrict to one language, and provision only its toolchain \|
	\| `--agentic-max-turns` \| give up after this many assistant turns (default 50) \|
	\| `--agentic-max-tokens` \| cap generation per turn (default: no cap); bounds a model that falls into a repetition loop, but also cuts reasoning off mid-thought. Truncated turns are counted as `truncated_turns` \|
	\| `--agentic-context-limit` \| abandon an episode once its context exceeds this many tokens \|
	\| `--agentic-transcripts DIR` \| save each episode's full message history to `DIR/<task_id>.json`, plus a `DIR/<task_id>.html` showing every reasoning block, reply, tool call and tool result; the main report links to it \|


	## Code suites (HumanEval, ClassEval)

	`--dataset humaneval` runs the 164 OpenAI HumanEval problems; `--dataset
	classeval` runs the 100 ClassEval class-generation tasks. Unlike the other
	suites these are not graded by matching text: the model's reply is parsed for a
	```` ```python ```` block, that code is combined with the task's own unit tests,
	and the result is executed.

	ClassEval asks for a whole class rather than one function, so it is scored the
	way upstream does - class level (every one of the task's tests passes, which
	is what feeds pass@k) and method level (what fraction of all 2196 test
	methods passed). The second number separates "generated nothing usable" from
	"got most of the class right", and both are printed at the end of a run.

	Give it more room to generate than HumanEval - a class with ten methods does not
	fit in a short budget:

	```bash
	python3 llama-eval.py \
	--server http://localhost:8033 --model my-model \
	--dataset classeval --threads 8 --n_predict 4096 --temperature 0
	```

	```bash
	# pass@1, greedy
	python3 llama-eval.py \
	--server http://localhost:8033 --model my-model \
	--dataset humaneval --threads 8 --n_predict 1024 --temperature 0

	# pass@10 -- each extra multiple of 164 is another sample per problem
	python3 llama-eval.py \
	--server http://localhost:8033 --model my-model \
	--dataset humaneval --n_cases 1640 --temperature 0.8 --top-p 0.95
	```

	`--grader-type exec` is the default for code suites, so neither it nor
	`--dataset-source` normally needs to be passed.

	### Execution sandbox - no Docker required

	Generated code is untrusted, but these suites need far less than a container to
	run safely. **The package set is derived from the dataset's own imports at
	startup**, not hardcoded and not taken from the upstream `requirements.txt` -
	which for ClassEval lists `torch`, `openai` and `transformers` that its tests
	never touch. So each suite installs exactly what it needs:

	\| suite \| third-party packages \|
	\|---\|---\|
	\| HumanEval \| none - every problem imports only stdlib (`typing, math, random, copy, string, collections, hashlib, re`) \|
	\| ClassEval \| 13: `beautifulsoup4, gensim, lxml, netifaces, nltk, numpy, openpyxl, pandas, pillow, pypdf2, python-docx, reportlab, scipy` \|

	Two of ClassEval's are declared rather than derived, because no import
	statement mentions them: `lxml` is selected by string
	(`BeautifulSoup(html, 'lxml')`, ClassEval_44 - without it that task silently
	drops from 23/23 to 7/23), and `scipy` is a hard gensim requirement. Note also
	that `PyPDF2` must not be modernised to `pypdf`: ClassEval_69's test imports
	`PdfFileReader`, a name pypdf removed.

	The venv is cached under `~/.cache/llama-eval/` (override with
	`LLAMA_EVAL_CACHE`) and rebuilt only when the package set, interpreter version
	or provisioning steps change; force one with `--rebuild-sandbox`. `uv` is used
	when available, which makes even the ClassEval venv build in seconds.

	Interpreter selection. A suite can prefer a different Python than the one
	running the tool, and declares it: ClassEval asks for 3.13, because gensim
	publishes no wheel past cp313 and its generated C no longer compiles on 3.14
	(it still uses `PyLongObject.ob_digit`). That interpreter is found automatically,
	including via `uv python`; `--sandbox-python` overrides it.

	3.14 is fully supported anyway. ClassEval uses exactly two gensim APIs
	(`utils.decode_htmlentities`, `matutils.unitvec`), both pure Python upstream, so
	on 3.14+ the suite drops gensim from the install set and provisions a small shim
	instead. Both paths score 100/100.

	Offline provisioning. A suite that would otherwise fetch data at test time
	declares a provisioning step that runs once at venv build time, while the
	network is still up. ClassEval uses this to pre-seed exactly three nltk corpora
	(`punkt_tab`, `averaged_perceptron_tagger_eng`, `wordnet`, ~33 MB), so the tests
	themselves run with no network at all. Those are the current names - the
	dataset asks for `punkt` and `averaged_perceptron_tagger`, which nltk has since
	renamed, and seeding the legacy names too is not harmless: with a network
	available nltk considers them satisfied and re-fetches the modern ones anyway.
	The corpora are written inside the venv, not `~/nltk_data`, because `$HOME` is
	a tmpfs under the strongest isolation tier.

	Each candidate program runs in a throwaway directory as its own process group,
	under the strongest isolation the host offers:

	\| tier \| requires \| isolates \|
	\|---\|---\|---\|
	\| `bwrap` \| bubblewrap installed \| read-only root, tmpfs `/tmp` and `$HOME`, no network \|
	\| `unshare` \| `unshare --map-current-user` \| no network \|
	\| `unshare-root` \| any `unshare -r` \| no network, but runs as uid 0 - see below \|
	\| `rlimit` \| always available \| cpu/memory/file-size/process caps only \|

	Candidate code must not run as root, which is why `unshare -r` is a
	last-resort tier that prints a warning. Root bypasses file permission bits, so a
	test asserting that writing to a `chmod 0444` file fails instead sees it
	succeed - ClassEval_50 scores 14/16 under uid 0 and 16/16 under uid 1000, with
	nothing in the output to suggest the sandbox caused it. Install bubblewrap.

	All tiers additionally cap CPU time, address space (4 GiB), file size and core
	dumps, and kill the whole process group on timeout (`--exec-timeout`, default
	15 s - raise it for ClassEval, whose tests do real work). Writes to the
	throwaway working directory are allowed, which is all the file-handling tasks in
	ClassEval need.

	Note that the weakest tier does not contain filesystem access. If you are
	scoring output from a model you do not trust, check the tier printed at startup.

	### Reproducibility

	Several tasks draw their test inputs from the global, unseeded RNG, so a
	verdict is not reproducible run to run - which adds noise exactly when you are
	comparing two models or two quantisations. `--exec-seed` (default `0`) seeds the
	RNG before each test; pass `--exec-seed -1` for the stock upstream behaviour.
	This does not change what the model is asked to compute.

	It matters more than it sounds. HumanEval/38, /50 and /53 are affected, and
	ClassEval_58 (duplicate mine placement) fails about 30% of the time on its own
	reference solution - measured here at 21/30 passes unseeded versus **30/30
	seeded**.

	### Dataset fidelity

	The canonical dataset runs unmodified on current Python - all 164 reference
	solutions pass on CPython 3.14 with no patches, so no forked or re-hosted copy
	is needed. `--dataset-source` accepts a local `.jsonl` or a HuggingFace repo id
	if you do want to point at a modified copy.

	HumanEval does contain a handful of long-standing defects where the prompt
	contradicts the graded test (notably HumanEval/47, /116 and /148). Those
	penalise a model that reads the docstring carefully, but correcting them makes
	the benchmark marginally easier and the resulting scores no longer comparable
	with published HumanEval numbers, so nothing is patched by default.

	ClassEval is different: the upstream data has defects that cap the achievable
	score below 100% for reasons unrelated to the model, so `--dataset classeval`
	defaults to a patched fork,
	[`ilintar/ClassEval`](https://huggingface.co/datasets/ilintar/ClassEval).
	Six edits over five tasks lift the reference solutions from 96/100 classes and
	2182/2196 test methods to 100/100 and 2196/2196. Three repair tests nothing
	could pass:

	- ClassEval_17 was a time bomb - a hard-coded 2024 date checked against
	`datetime.now()`, unpassable since January 2024.
	- ClassEval_48 asserted `get_hostname('0.0.0.0') == 'LAPTOP-2CS86KUM'`, the
	dataset author's own machine name.
	- ClassEval_31 compared floats with `assertEqual`, failing on a 4-ULP numpy
	difference.

	The other three fix only the reference solutions (NumPy 2.0 removals in
	ClassEval_51, a 30%-flaky mine generator in ClassEval_58) and cannot affect a
	model's score. No `skeleton` was touched, so what the model is asked to write is
	byte-identical to upstream. Pass `--dataset-source FudanSELab/ClassEval` for the
	unpatched original.