Buckets:
| Name | Size | Uploaded | Xet hash |
|---|---|---|---|
| README.md | 4.52 kB xet | 89bd1465 | |
| SHA256SUMS | 330 kB xet | a1d1cedd | |
| climb-2504.13161.pdf | 3.67 MB xet | 539a8930 |
ClimbMix — raw text, globally shuffled, sharded like Long-Data-Collections
NVIDIA Nemotron-ClimbMix,
detokenized from the released GPT-2 token ids back to raw text, globally
shuffled at document level (seed 42), and resharded into the same on-disk
layout as the Long-Data-Collections arrow dataset that flame reads.
Why the shuffle matters: the upstream release is sorted by cluster — the leading files are 100% cluster 1, the trailing ones 100% cluster 20. Read sequentially (e.g. streaming mode, where HF's ~1k-row buffer shuffle cannot fix runs of tens of millions of same-cluster rows), that ordering is badly non-i.i.d. Here every shard carries all 20 clusters at the global proportions (verified to within ±0.15 pp on sampled shards), so the data is safe under both streaming and map-style readers.
Paper: CLIMB (arXiv:2504.13161) — PDF in docs/.
Layout
climbmix-text-sharded/
data-00000-of-03584.arrow # pyarrow IPC *stream* format, ~460 MB each
...
data-03583-of-03584.arrow
dataset_info.json
state.json
shuffle_manifest.json # seed, per-shard row counts
Documents are i.i.d. across and within shards (uniform global permutation: seeded random bucket assignment, then an in-memory permutation per bucket). Shard row counts vary statistically: mean 154,385, spread 1.81%.
| Field | Type | Notes |
|---|---|---|
text |
string |
raw document text |
cluster_id |
int32 |
CLIMB semantic cluster label, carried through from the source |
The reference dataset has only text. cluster_id is kept because it is the
only thing that makes post-hoc mixture re-weighting possible; flame reads
sample['text'] and ignores the extra column.
Contents
| Documents | 553,315,056 |
| GPT-2 tokens in source | 356,864,528,972 (~356.9 B) |
| Text size | ~1.6 TB |
| Shards | 3584 (divides evenly by 8/16/32/64/128/256) |
| License | CC BY-NC 4.0 (non-commercial), inherited from the source |
Note on the token count. The upstream dataset card advertises 400 B tokens.
Summing len(tokens) over every document in all 100 released parts gives
~356.9 B. Per-part averages genuinely range from 494 to 698 tokens/row, so
extrapolating from any single part overestimates. Budget training runs against
357 B, not 400 B.
Use with flame
Tokenization happens online at train time (OnlineTokenizedIterableDataset),
exactly as with Long-Data-Collections — nothing here is pre-tokenized.
seq_len = 4096
dataset = "arrow"
dataset_name = "default"
dataset_split = "train"
data_files = "/path/to/climbmix-text-sharded/data-*.arrow"
tokenizer_path = "fla-hub/transformer-1.3B-100B"
seq_len = 4096 fits this corpus: after retokenizing to the 32k Llama vocab,
99.1% of documents land at or under 4096 tokens (p99 = 3956). The 32k setting
used for Long-Data-Collections would make ~87% of every sequence cross-document
padding, since ClimbMix is short web text (median 321 tokens), not long documents.
Provenance
Built by scripts/climb/ on branch climbmix-arrow-shards of
Zefan-Cai/llm:
download_parts.py → climbmix_to_arrow.py → shuffle_bucket.py →
finalize_dataset.py, checked by verify_multiset.py.
Verification performed:
- per-part row reconciliation — 98 parts × 5,533,150 rows, part_99 at 5,533,206, part_0 at 5,533,150, summing to 553,315,056 exactly
- independent re-download of
part_50— 5,533,150 source lines, and NVIDIA's owntoken_countfield agrees with the actual token array lengths - decode round-trip — re-encoding the decoded text with the GPT-2 tokenizer reproduced the source ids with 0 mismatches over 400 sampled documents
- shuffle content check — order-independent multiset digest (commutative sum of per-row blake2b over text+cluster_id) identical before and after the shuffle across all 553,315,056 rows
- shuffle quality check — 7 sampled shards spanning the range each contain all 20 clusters at the global proportions, within ±0.15 pp
Related: Nemotron-ClimbLab is the search space the CLIMB method operates on (~1.26 T tokens, 20 clusters as directories). ClimbMix, here, is the result — 400 B-advertised / 357 B-actual tokens with the discovered mixture already applied. ClimbLab is deliberately not included in this bucket.
- Total size
- 1.65 TB
- Files
- 3,592
- Last updated
- Oct 1
- Pre-warmed CDN
- US EU US EU