---
title: README
emoji: 🏝️
colorFrom: blue
colorTo: green
sdk: static
pinned: true
---

# OpenTWBench

**An open evaluation suite for Taiwan-domain knowledge in large language models.**

🔗 **[opentwbench.ai](https://opentwbench.ai)** — leaderboard, subject map, and how the data is built

---

Most Chinese-language benchmarks are built from mainland sources, in Simplified
Chinese, about mainland institutions. A model can score well on them while knowing
nothing about the Labor Standards Act, the Household Registration Act, or how a
Taiwanese pharmacist is licensed. These datasets exist to make that gap measurable.

The questions come from Taiwan's **national examinations**, published as open data by
the **Ministry of Examination (考選部)** — professionally written, officially
answer-keyed, and covering essentially every regulated profession in the country.

## The datasets

One dataset per **學科 (academic subject)**, named `tw-<subject>-bench`, in the
**Twinkle Eval MCQ** format (`question`, `A`–`D`, `answer`) with provenance columns
alongside. Subjects are the unit, not the profession sitting the exam:
神經疾病物理治療學 and 骨科疾病物理治療學 are separate benchmarks, because a model can
know one and not the other.

```python
from datasets import load_dataset
ds = load_dataset("OpenTWBench/tw-medicine-bench", split="test")
```

## Reading the scores honestly

- These are **public past papers**, very likely in pretraining corpora. Use the suite
  for *relative* comparison between models, not to claim a model "passes" an exam.
- **Check the parse rate before believing a low score** — a model that knows the
  answer but writes it in an unexpected shape is a formatting failure.
- **Shuffle the options.** The source papers carry an answer-position skew.

Taiwan **law** is covered separately and in depth by
[`lianghsun/tw-legal-benchmark-v2`](https://huggingface.co/datasets/lianghsun/tw-legal-benchmark-v2);
questions already published there are excluded from this org, item by item.

## Contributing

A wrong answer key is worse than a missing question. Open a discussion on the dataset
it affects, quoting the `paper_id` and `q_no`, and it can be traced to the original PDF.

*Questions and answer keys are official publications of 考選部, released as open data.
Packaging and metadata: Apache-2.0.*