Buckets:
| # ClimbMix — raw text, globally shuffled, sharded like Long-Data-Collections | |
| NVIDIA [Nemotron-ClimbMix](https://huggingface.co/datasets/nvidia/Nemotron-ClimbMix), | |
| detokenized from the released GPT-2 token ids back to raw text, **globally | |
| shuffled at document level (seed 42)**, and resharded into the same on-disk | |
| layout as the `Long-Data-Collections` arrow dataset that `flame` reads. | |
| **Why the shuffle matters:** the upstream release is sorted by cluster — the | |
| leading files are 100% cluster 1, the trailing ones 100% cluster 20. Read | |
| sequentially (e.g. streaming mode, where HF's ~1k-row buffer shuffle cannot fix | |
| runs of tens of millions of same-cluster rows), that ordering is badly | |
| non-i.i.d. Here every shard carries all 20 clusters at the global proportions | |
| (verified to within ±0.15 pp on sampled shards), so the data is safe under both | |
| streaming and map-style readers. | |
| Paper: [CLIMB (arXiv:2504.13161)](https://arxiv.org/abs/2504.13161) — PDF in `docs/`. | |
| ## Layout | |
| ``` | |
| climbmix-text-sharded/ | |
| data-00000-of-03584.arrow # pyarrow IPC *stream* format, ~460 MB each | |
| ... | |
| data-03583-of-03584.arrow | |
| dataset_info.json | |
| state.json | |
| shuffle_manifest.json # seed, per-shard row counts | |
| ``` | |
| Documents are i.i.d. across and within shards (uniform global permutation: | |
| seeded random bucket assignment, then an in-memory permutation per bucket). | |
| Shard row counts vary statistically: mean 154,385, spread 1.81%. | |
| | Field | Type | Notes | | |
| |---|---|---| | |
| | `text` | `string` | raw document text | | |
| | `cluster_id` | `int32` | CLIMB semantic cluster label, carried through from the source | | |
| The reference dataset has only `text`. `cluster_id` is kept because it is the | |
| only thing that makes post-hoc mixture re-weighting possible; `flame` reads | |
| `sample['text']` and ignores the extra column. | |
| ## Contents | |
| | | | | |
| |---|---| | |
| | Documents | 553,315,056 | | |
| | GPT-2 tokens in source | 356,864,528,972 (~356.9 B) | | |
| | Text size | ~1.6 TB | | |
| | Shards | 3584 (divides evenly by 8/16/32/64/128/256) | | |
| | License | CC BY-NC 4.0 (non-commercial), inherited from the source | | |
| **Note on the token count.** The upstream dataset card advertises 400 B tokens. | |
| Summing `len(tokens)` over every document in all 100 released parts gives | |
| **~356.9 B**. Per-part averages genuinely range from 494 to 698 tokens/row, so | |
| extrapolating from any single part overestimates. Budget training runs against | |
| 357 B, not 400 B. | |
| ## Use with flame | |
| Tokenization happens **online at train time** (`OnlineTokenizedIterableDataset`), | |
| exactly as with Long-Data-Collections — nothing here is pre-tokenized. | |
| ```toml | |
| seq_len = 4096 | |
| dataset = "arrow" | |
| dataset_name = "default" | |
| dataset_split = "train" | |
| data_files = "/path/to/climbmix-text-sharded/data-*.arrow" | |
| tokenizer_path = "fla-hub/transformer-1.3B-100B" | |
| ``` | |
| `seq_len = 4096` fits this corpus: after retokenizing to the 32k Llama vocab, | |
| 99.1% of documents land at or under 4096 tokens (p99 = 3956). The 32k setting | |
| used for Long-Data-Collections would make ~87% of every sequence cross-document | |
| padding, since ClimbMix is short web text (median 321 tokens), not long documents. | |
| ## Provenance | |
| Built by `scripts/climb/` on branch `climbmix-arrow-shards` of | |
| [Zefan-Cai/llm](https://github.com/Zefan-Cai/llm): | |
| `download_parts.py` → `climbmix_to_arrow.py` → `shuffle_bucket.py` → | |
| `finalize_dataset.py`, checked by `verify_multiset.py`. | |
| Verification performed: | |
| - per-part row reconciliation — 98 parts × 5,533,150 rows, part_99 at 5,533,206, | |
| part_0 at 5,533,150, summing to 553,315,056 exactly | |
| - independent re-download of `part_50` — 5,533,150 source lines, and NVIDIA's own | |
| `token_count` field agrees with the actual token array lengths | |
| - decode round-trip — re-encoding the decoded text with the GPT-2 tokenizer | |
| reproduced the source ids with 0 mismatches over 400 sampled documents | |
| - shuffle content check — order-independent multiset digest (commutative sum of | |
| per-row blake2b over text+cluster_id) identical before and after the shuffle | |
| across all 553,315,056 rows | |
| - shuffle quality check — 7 sampled shards spanning the range each contain all | |
| 20 clusters at the global proportions, within ±0.15 pp | |
| Related: [Nemotron-ClimbLab](https://huggingface.co/datasets/nvidia/Nemotron-ClimbLab) | |
| is the *search space* the CLIMB method operates on (~1.26 T tokens, 20 clusters as | |
| directories). ClimbMix, here, is the *result* — 400 B-advertised / 357 B-actual | |
| tokens with the discovered mixture already applied. ClimbLab is deliberately not | |
| included in this bucket. | |
Xet Storage Details
- Size:
- 4.52 kB
- Xet hash:
- 89bd1465a7fcdb6be7f9901ebf2bfda174e1a87a5eb06a177e5da20d84e48d82
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.