419 kB

Ctrl+K

2 contributors

History: 33 commits

ViuAI

Fix CUDA OOM on T4: cast model to amp_dtype directly, batch_size=2, safe RMSNorm

9d0f9ca verified about 1 hour ago

add_curated_hinglish.py
3.23 kB
docs/feat: sync translation & hinglish expansion for data/scripts/add_curated_hinglish.py 13 days ago
add_dictionary_stories_uncensored.py
3.7 kB
docs & scripts: add dictionary, expanded stories, and massive uncensored datasets milestone (117GB/83B tokens) 13 days ago
add_full_scale_5_domains.py
10.8 kB
docs & scripts: add full-scale 5-domain milestone (145.93GB/105B tokens) 10 days ago
add_massive_uncensored_alpaca.py
1.38 kB
docs & scripts: add dictionary, expanded stories, and massive uncensored datasets milestone (117GB/83B tokens) 13 days ago
add_open_source_toxicity.py
3.32 kB
feat/docs: sync open-source toxicity datasets & docs for data/scripts/add_open_source_toxicity.py 13 days ago
add_samanantar.py
2.28 kB
docs/feat: sync translation & hinglish expansion for data/scripts/add_samanantar.py 13 days ago
add_science_stem_datasets.py
4.7 kB
docs & scripts: add dedicated science stream milestone (128.75GB/92B tokens) 12 days ago
add_science_stream.py
6 kB
docs & scripts: add dedicated science stream milestone (128.75GB/92B tokens) 12 days ago
add_stories_and_uncensored.py
20 kB
feat/docs: sync stories & uncensored scripts & docs for data/scripts/add_stories_and_uncensored.py 13 days ago
add_wikipedia.py
2.92 kB
docs/feat: sync deep domains, wikipedia ingestion & docs for data/scripts/add_wikipedia.py 13 days ago
check_classic_novels.py
456 Bytes
feat: wipe-proof HF persistence (full ckpt + live log + newest-2 cleanup) 8 days ago
clean_dedup.py
3.26 kB
docs: add official comprehensive data audit report (145.93GB/105.1B tokens - 100% PASS) 10 days ago
collect_automobile_and_tyohar_boost.py
12.3 kB
sync local: enforce_mix sampler, tokenizer v2 (mask fix + 35/30/35), mix optional docs, repair scripts 8 days ago
collect_distilled_corpus.py
11.4 kB
sync local: enforce_mix sampler, tokenizer v2 (mask fix + 35/30/35), mix optional docs, repair scripts 8 days ago
collect_domains.py
30 kB
Add Pillar 26 Grammar & Language Mechanics (Hindi Vyakaran, English Grammar, Hinglish Syntax) 14 days ago
collect_p1_p2_heavy_boost.py
11.6 kB
sync local: enforce_mix sampler, tokenizer v2 (mask fix + 35/30/35), mix optional docs, repair scripts 8 days ago
collect_p1_p2_real_pillars.py
15.3 kB
sync local: enforce_mix sampler, tokenizer v2 (mask fix + 35/30/35), mix optional docs, repair scripts 8 days ago
collect_romance_and_chat_corpus.py
16.8 kB
feat: wipe-proof HF persistence (full ckpt + live log + newest-2 cleanup) 8 days ago
collect_spiritual_and_boost_corpus.py
22.5 kB
fix: exclude non-5col schema paths from HF stream (wikipedia/grammar/gsm8k CastError on shuffle) 8 days ago
filter_by_safety.py
1.73 kB
docs: add official comprehensive data audit report (145.93GB/105.1B tokens - 100% PASS) 10 days ago
generate_basic_arithmetic.py
9.74 kB
docs & scripts: add full-scale 5-domain milestone (145.93GB/105B tokens) 10 days ago
generate_deep_domains.py
68.8 kB
docs/feat: sync deep domains, wikipedia ingestion & docs for data/scripts/generate_deep_domains.py 13 days ago
generate_toxicity_lexicon.py
18.1 kB
docs: add official comprehensive data audit report (145.93GB/105.1B tokens - 100% PASS) 10 days ago
generate_vyakaran_master.py
32 kB
docs: add official comprehensive data audit report (145.93GB/105.1B tokens - 100% PASS) 10 days ago
ingest_erotica_adult.py
14.3 kB
Fix CUDA OOM on T4: cast model to amp_dtype directly, batch_size=2, safe RMSNorm about 1 hour ago
ingest_large_cinema_mythology.py
22.4 kB
Sync data/scripts/ingest_large_cinema_mythology.py with massive Mythology and Bollywood Cinema corpora expansion 2 days ago
inspect_romance_datasets.py
1.66 kB
feat: wipe-proof HF persistence (full ckpt + live log + newest-2 cleanup) 8 days ago
kaggle_collector.py
21.2 kB
feat: 100% overnight-resilient batch repair with 999 retries and persistent rate-limit handling 10 days ago
kaggle_repair_all.py
19.3 kB
sync local: enforce_mix sampler, tokenizer v2 (mask fix + 35/30/35), mix optional docs, repair scripts 8 days ago
push_dataset_to_hf.py
3.63 kB
init: 28L MoE 241M audit-fixed, smoke pass 16 days ago
read_live_samples.py
1.56 kB
docs & scripts: add dictionary, expanded stories, and massive uncensored datasets milestone (117GB/83B tokens) 13 days ago
repair_backfill_hindi.py
4.97 kB
docs: add official comprehensive data audit report (145.93GB/105.1B tokens - 100% PASS) 10 days ago
repair_reshard_large.py
4.91 kB
docs: add official comprehensive data audit report (145.93GB/105.1B tokens - 100% PASS) 10 days ago
standardize_to_unified.py
10.6 kB
docs: add official comprehensive data audit report (145.93GB/105.1B tokens - 100% PASS) 10 days ago
test_xlit.py
2.67 kB
docs & scripts: add dictionary, expanded stories, and massive uncensored datasets milestone (117GB/83B tokens) 13 days ago