zefancainlp/CLIMB / docs /README.md
zefancainlp's picture
|
download
raw
4.52 kB
# ClimbMix — raw text, globally shuffled, sharded like Long-Data-Collections
NVIDIA [Nemotron-ClimbMix](https://huggingface.co/datasets/nvidia/Nemotron-ClimbMix),
detokenized from the released GPT-2 token ids back to raw text, **globally
shuffled at document level (seed 42)**, and resharded into the same on-disk
layout as the `Long-Data-Collections` arrow dataset that `flame` reads.
**Why the shuffle matters:** the upstream release is sorted by cluster — the
leading files are 100% cluster 1, the trailing ones 100% cluster 20. Read
sequentially (e.g. streaming mode, where HF's ~1k-row buffer shuffle cannot fix
runs of tens of millions of same-cluster rows), that ordering is badly
non-i.i.d. Here every shard carries all 20 clusters at the global proportions
(verified to within ±0.15 pp on sampled shards), so the data is safe under both
streaming and map-style readers.
Paper: [CLIMB (arXiv:2504.13161)](https://arxiv.org/abs/2504.13161) — PDF in `docs/`.
## Layout
```
climbmix-text-sharded/
data-00000-of-03584.arrow # pyarrow IPC *stream* format, ~460 MB each
...
data-03583-of-03584.arrow
dataset_info.json
state.json
shuffle_manifest.json # seed, per-shard row counts
```
Documents are i.i.d. across and within shards (uniform global permutation:
seeded random bucket assignment, then an in-memory permutation per bucket).
Shard row counts vary statistically: mean 154,385, spread 1.81%.
| Field | Type | Notes |
|---|---|---|
| `text` | `string` | raw document text |
| `cluster_id` | `int32` | CLIMB semantic cluster label, carried through from the source |
The reference dataset has only `text`. `cluster_id` is kept because it is the
only thing that makes post-hoc mixture re-weighting possible; `flame` reads
`sample['text']` and ignores the extra column.
## Contents
| | |
|---|---|
| Documents | 553,315,056 |
| GPT-2 tokens in source | 356,864,528,972 (~356.9 B) |
| Text size | ~1.6 TB |
| Shards | 3584 (divides evenly by 8/16/32/64/128/256) |
| License | CC BY-NC 4.0 (non-commercial), inherited from the source |
**Note on the token count.** The upstream dataset card advertises 400 B tokens.
Summing `len(tokens)` over every document in all 100 released parts gives
**~356.9 B**. Per-part averages genuinely range from 494 to 698 tokens/row, so
extrapolating from any single part overestimates. Budget training runs against
357 B, not 400 B.
## Use with flame
Tokenization happens **online at train time** (`OnlineTokenizedIterableDataset`),
exactly as with Long-Data-Collections — nothing here is pre-tokenized.
```toml
seq_len = 4096
dataset = "arrow"
dataset_name = "default"
dataset_split = "train"
data_files = "/path/to/climbmix-text-sharded/data-*.arrow"
tokenizer_path = "fla-hub/transformer-1.3B-100B"
```
`seq_len = 4096` fits this corpus: after retokenizing to the 32k Llama vocab,
99.1% of documents land at or under 4096 tokens (p99 = 3956). The 32k setting
used for Long-Data-Collections would make ~87% of every sequence cross-document
padding, since ClimbMix is short web text (median 321 tokens), not long documents.
## Provenance
Built by `scripts/climb/` on branch `climbmix-arrow-shards` of
[Zefan-Cai/llm](https://github.com/Zefan-Cai/llm):
`download_parts.py` → `climbmix_to_arrow.py` → `shuffle_bucket.py` →
`finalize_dataset.py`, checked by `verify_multiset.py`.
Verification performed:
- per-part row reconciliation — 98 parts × 5,533,150 rows, part_99 at 5,533,206,
part_0 at 5,533,150, summing to 553,315,056 exactly
- independent re-download of `part_50` — 5,533,150 source lines, and NVIDIA's own
`token_count` field agrees with the actual token array lengths
- decode round-trip — re-encoding the decoded text with the GPT-2 tokenizer
reproduced the source ids with 0 mismatches over 400 sampled documents
- shuffle content check — order-independent multiset digest (commutative sum of
per-row blake2b over text+cluster_id) identical before and after the shuffle
across all 553,315,056 rows
- shuffle quality check — 7 sampled shards spanning the range each contain all
20 clusters at the global proportions, within ±0.15 pp
Related: [Nemotron-ClimbLab](https://huggingface.co/datasets/nvidia/Nemotron-ClimbLab)
is the *search space* the CLIMB method operates on (~1.26 T tokens, 20 clusters as
directories). ClimbMix, here, is the *result* — 400 B-advertised / 357 B-actual
tokens with the discovered mixture already applied. ClimbLab is deliberately not
included in this bucket.

Xet Storage Details

Size:
4.52 kB
·
Xet hash:
89bd1465a7fcdb6be7f9901ebf2bfda174e1a87a5eb06a177e5da20d84e48d82

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.