org About / profile README

README.md

@@ -0,0 +1,105 @@
+---
+title: CricketBench
+---
+
+# CricketBench
+
+A domain benchmark for **cricket reasoning in large language models**.
+
+Cricket is a game of dense rules, sharp situations, deep records, precise
+terminology, and layered strategy. It's rewarding to model but easy to fake:
+most existing cricket "benchmarks" test surface trivia rather than reasoning.
+CricketBench targets the reasoning gap directly.
+
+## What's here
+
+- **[Dataset](https://huggingface.co/datasets/cricketbench/cricketbench)** —
+  The canonical multiple-choice questions, versioned by release, organised
+  into five reasoning dimensions:
+    - `laws_and_rule_precision` — the 42 Laws of Cricket + ICC Playing Conditions
+    - `match_situation_reasoning` — run rates, DLS, follow-on, powerplay
+    - `player_and_statistical_recall` — records under reasoning, not trivia
+    - `commentary_generation_quality` — terminology, shot/delivery identification
+    - `tactical_and_strategic_reasoning` — captaincy, field settings, decisions
+- **[Leaderboard](https://huggingface.co/spaces/cricketbench/cricketbench-leaderboard)**
+  — Per-model accuracy and per-dimension scores across the current release.
+- **[Baselines](https://huggingface.co/datasets/cricketbench/cricketbench-baselines)**
+  — Per-model raw answers, kept alongside so anyone can reproduce
+  discrimination scores from the same runs.
+
+## How it's built
+
+Every question passes through a closed loop: hand-authored seeds →
+LLM-generated variants → red-team critique → answered by 8+ models across the
+capability gradient → per-question item-discrimination (point-biserial r
+against per-model ability) → human review. Only items that survive the human
+review land in the dataset.
+
+Design decisions in one line each:
+
+- **Per-model item discrimination, not frontier-vs-open buckets** — capability
+  is measured from the data, not assigned by us.
+- **Unparseable model outputs are excluded from scoring**, not counted wrong.
+- **Red-team `major` severity routes to human revise** regardless of the
+  statistical signal.
+- **Top-model-miss diagnostic** flags items where the strongest legitimate
+  model missed while ≥2 mid-tier models got it right — usually a sign the
+  key is wrong.
+- **Model rankings require n ≥ 30 per dimension** before we consider them
+  publishable.
+
+## Where it comes from
+
+CricketBench was founded in **Chennai** in 2026, in a city where cricket is
+lived. Chepauk, TNCA, the CSK yellow shirts, the smell of dusty afternoon
+cricket at Marina — the project takes its texture from the place. The code,
+the review discipline, and the tone all trace back there.
+
+Contributors from anywhere are warmly welcome. Every approved question carries
+provenance: who reviewed it, which models were baselined against it, and how
+sharply it discriminated. The intent is a benchmark that any researcher,
+anywhere, can trust and cite.
+
+## How to help
+
+- **Contribute questions** — open an issue with a candidate item, or submit a
+  PR against the dataset.
+- **Contribute baseline model results** — submit a JSONL of predictions to the
+  leaderboard Space.
+- **Contribute revisions** — flag ambiguous items or wrong keys via HF
+  Discussion.
+- **Contribute reviews** — most valuable of all. The pipeline generates faster
+  than a single person can review.
+
+## Maintainers
+
+- **[Hariprasad Sudharshan](https://huggingface.co/brettleehari)** — Founder
+  and project lead. Senior product manager, based in Chennai. Responsible for
+  taxonomy, scoring policies, and question review discipline.
+
+Interested in becoming a maintainer? Contribute a batch of reviewed questions
+or a set of high-quality baseline model results and open a Discussion —
+sustained contributors are invited to write access on the org.
+
+## Citation
+
+```bibtex
+@misc{cricketbench2026,
+  title        = {{CricketBench}: A domain benchmark for cricket reasoning
+                  in large language models},
+  author       = {Sudharshan, Hariprasad and {CricketBench contributors}},
+  year         = {2026},
+  howpublished = {Hugging Face Datasets},
+  url          = {https://huggingface.co/datasets/cricketbench/cricketbench},
+  version      = {v0.0}
+}
+```
+
+## Contact
+
+Discussions on the HF Community tab of any `cricketbench/*` repo.
+
+---
+
+Licensed CC-BY-SA 4.0. Every version is git-tagged; papers should cite a
+specific release.