zefancainlp/CLIMB / docs /README.md
zefancainlp's picture
|
download
raw
4.52 kB

ClimbMix — raw text, globally shuffled, sharded like Long-Data-Collections

NVIDIA Nemotron-ClimbMix, detokenized from the released GPT-2 token ids back to raw text, globally shuffled at document level (seed 42), and resharded into the same on-disk layout as the Long-Data-Collections arrow dataset that flame reads.

Why the shuffle matters: the upstream release is sorted by cluster — the leading files are 100% cluster 1, the trailing ones 100% cluster 20. Read sequentially (e.g. streaming mode, where HF's ~1k-row buffer shuffle cannot fix runs of tens of millions of same-cluster rows), that ordering is badly non-i.i.d. Here every shard carries all 20 clusters at the global proportions (verified to within ±0.15 pp on sampled shards), so the data is safe under both streaming and map-style readers.

Paper: CLIMB (arXiv:2504.13161) — PDF in docs/.

Layout

climbmix-text-sharded/
  data-00000-of-03584.arrow   # pyarrow IPC *stream* format, ~460 MB each
  ...
  data-03583-of-03584.arrow
  dataset_info.json
  state.json
  shuffle_manifest.json       # seed, per-shard row counts

Documents are i.i.d. across and within shards (uniform global permutation: seeded random bucket assignment, then an in-memory permutation per bucket). Shard row counts vary statistically: mean 154,385, spread 1.81%.

Field Type Notes
text string raw document text
cluster_id int32 CLIMB semantic cluster label, carried through from the source

The reference dataset has only text. cluster_id is kept because it is the only thing that makes post-hoc mixture re-weighting possible; flame reads sample['text'] and ignores the extra column.

Contents

Documents 553,315,056
GPT-2 tokens in source 356,864,528,972 (~356.9 B)
Text size ~1.6 TB
Shards 3584 (divides evenly by 8/16/32/64/128/256)
License CC BY-NC 4.0 (non-commercial), inherited from the source

Note on the token count. The upstream dataset card advertises 400 B tokens. Summing len(tokens) over every document in all 100 released parts gives ~356.9 B. Per-part averages genuinely range from 494 to 698 tokens/row, so extrapolating from any single part overestimates. Budget training runs against 357 B, not 400 B.

Use with flame

Tokenization happens online at train time (OnlineTokenizedIterableDataset), exactly as with Long-Data-Collections — nothing here is pre-tokenized.

seq_len = 4096
dataset = "arrow"
dataset_name = "default"
dataset_split = "train"
data_files = "/path/to/climbmix-text-sharded/data-*.arrow"
tokenizer_path = "fla-hub/transformer-1.3B-100B"

seq_len = 4096 fits this corpus: after retokenizing to the 32k Llama vocab, 99.1% of documents land at or under 4096 tokens (p99 = 3956). The 32k setting used for Long-Data-Collections would make ~87% of every sequence cross-document padding, since ClimbMix is short web text (median 321 tokens), not long documents.

Provenance

Built by scripts/climb/ on branch climbmix-arrow-shards of Zefan-Cai/llm: download_parts.py → climbmix_to_arrow.py → shuffle_bucket.py → finalize_dataset.py, checked by verify_multiset.py.

Verification performed:

  • per-part row reconciliation — 98 parts × 5,533,150 rows, part_99 at 5,533,206, part_0 at 5,533,150, summing to 553,315,056 exactly
  • independent re-download of part_50 — 5,533,150 source lines, and NVIDIA's own token_count field agrees with the actual token array lengths
  • decode round-trip — re-encoding the decoded text with the GPT-2 tokenizer reproduced the source ids with 0 mismatches over 400 sampled documents
  • shuffle content check — order-independent multiset digest (commutative sum of per-row blake2b over text+cluster_id) identical before and after the shuffle across all 553,315,056 rows
  • shuffle quality check — 7 sampled shards spanning the range each contain all 20 clusters at the global proportions, within ±0.15 pp

Related: Nemotron-ClimbLab is the search space the CLIMB method operates on (~1.26 T tokens, 20 clusters as directories). ClimbMix, here, is the result — 400 B-advertised / 357 B-actual tokens with the discovered mixture already applied. ClimbLab is deliberately not included in this bucket.

Xet Storage Details

Size:
4.52 kB
·
Xet hash:
89bd1465a7fcdb6be7f9901ebf2bfda174e1a87a5eb06a177e5da20d84e48d82

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.