README.md

# Chess position datasets

Two position databases of chess positions, plus their variance reports. Each is
published in two forms: a partitioned Parquet dataset for columnar scans, and
the VPD1 SQLite container it was built from for indexed single-position lookup.

Positions carry no evaluation. Each row is a legal six-field FEN with derived
board features. The `result` column records the source game's outcome as
metadata; it is not a position label.

## `lc0-selfplay-2m`

2,000,000 positions sampled from Leela Chess Zero self-play training data
published at `storage.lczero.org/files/training_data`, drawn from 252,686
distinct games across 56 archives and 9 runs spanning 2018-09 to 2026-08.

Partitioned by generation quality, since the networks producing the games get
weaker down the list:

| Partition | Positions | Runs |
|---|---:|---|
| `strong` | 1,200,000 | test80, test91 |
| `mid` | 440,000 | test79, test75, late test60 |
| `low` | 260,000 | test40, early test60, test71_5 |
| `early` | 100,000 | test30, run3 |

`test71` is excluded: it is the Chess960 run, measured at 30-36% Chess960
against at most 1.2% in every other run. Chess960 positions are excluded
throughout, so every FEN is legal standard chess.

Records were converted with the LC0 rescorer without tablebase rescoring or
deblundering, and without the position filtering the Stockfish NNUE path
applies. Up to 12 positions were taken per game.

## `lichess-elite-300k`

300,000 positions from 180,510 games in the
[Lichess Elite Database](https://database.nikonoel.fr/), which filters the
Lichess standard database to games where a 2400+ player faced a 2200+ player.
Twelve months are sampled evenly, spread across 2020-06 to 2025-10, at three
positions per game.

This is strong human **blitz**: roughly 88% of eligible games are 3+0 or 3+2,
5% rapid, under 1% classical, White Elo median 2550. Bullet and ultrabullet are
excluded. Draws are 12.3% of source games, against 44.7% in the self-play set —
human blitz is substantially more decisive than engine self-play, so this set
supplies sharper and less balanced positions. Per-month speed composition is
recorded in the SQLite `metadata` table.

## Layout

Parquet is partitioned Hive-style by `source_split` and `phase`, Zstandard
compressed, in 65,536-row groups. `_manifest.json` in each dataset carries the
source checksum plus per-file row counts, sizes, ID bounds and checksums.

| Column | Meaning |
|---|---|
| `id` | Row identifier |
| `random_key` | Deterministic 63-bit key for constant-time random sampling |
| `fen` | Normalised six-field FEN; fullmove number set to 1 |
| `source_member` | Source archive or month the position came from |
| `game_number` | Sequential source-game number |
| `ply` | Ply at which the position was sampled |
| `result` | Source game result; metadata, not a position label |
| `side_to_move` | 1 for White, 0 for Black |
| `piece_count` | Occupied squares |
| `non_pawn_material` | Combined N/B/R/Q material in pawn units |
| `material_balance` | White material minus Black material |
| `legal_moves` | Legal move count |
| `in_check` | Whether the side to move is in check |
| `castling_mask` | KQkq availability as four bits |
| `halfmove_clock` | Fifty-move-rule clock |

`source_split` and `phase` are encoded in the directory paths rather than
repeated per row. `phase` is `opening`, `middlegame` or `endgame`, held to a
15/60/25 ratio within every partition.

FENs are unique within each dataset. Across the two, 4,851 FENs appear in both,
almost entirely common opening positions.

## Verification

Every row was re-checked against python-chess: all FENs parse as legal standard
chess, and `side_to_move`, `piece_count`, `in_check` and `legal_moves` were
recomputed and matched on sampled subsets. Parquet and SQLite forms were
confirmed to hold identical FEN sets.

The `*-variance.json` reports record distribution statistics and threshold
checks for each database.