Release TinyThinkModel-72M-V1 as an owner-authorized private snapshot

Final step 22889, 72M parameters, context 1024. Includes exact resume state, verified HellaSwag metrics and linked data commit 9d8dd335c0c946ae84ff0919b09c1042da5315f7. No public redistribution clearance is asserted.

LICENSE

@@ -0,0 +1,31 @@
+TinyThinkModel-72M-V1: private research snapshot and license scope
+
+This repository is an owner-authorized PRIVATE archival/research upload. It is
+not a public model or dataset release. Access to these files does not grant
+permission to redistribute all of their third-party contents or to use them
+commercially. Public release and the final model-weights license remain pending
+review of the training-data terms and naming obligations described below.
+
+Copyright (c) 2026 sraivante applies only to original project code, documentation,
+and original selection/arrangement contributions. Those original contributions
+are licensed under Apache License 2.0; see licenses/Apache-2.0.txt. This grant does
+not relicense third-party records or establish an Apache license for the trained
+weights. Third-party ownership and notices are retained in THIRD_PARTY.md.
+
+AM-DeepSeek-R1-Distilled-1.4M is marked CC-BY-NC-4.0 and its authors specify
+research-only usage. NVIDIA's dataset includes per-record CC-BY-4.0, ODC-BY and
+CC-BY-SA terms and states that models trained on relevant synthetic content may
+be subject to Llama 3.1/3.3 redistribution and use requirements. In particular,
+the Llama licenses contain a model-name prefix requirement. The local cleaned
+schema did not retain per-record license/generator metadata. Mixture-of-Thoughts
+also incorporates multiple components rather than declaring one blanket license.
+
+The requested exact name TinyThinkModel-72M-V1 is retained for this private
+snapshot. No claim that public distribution under this name has been cleared is
+made. No public-redistribution clearance is implied for HellaSwag: the pinned
+Hugging Face snapshot has no declared license in its metadata and the historical
+upstream license could not be verified at release preparation time.
+
+No blanket Apache-2.0, open-source or commercial-use license is granted for the
+combined training corpus, benchmark data or model weights. No warranties of
+accuracy, fitness, non-infringement or safety are provided by this project.

NOTICE

@@ -0,0 +1,2 @@
+Copyright (c) 2026 sraivante. Original contributions only; third-party rights retained.
+See LICENSE and THIRD_PARTY.md. This repository is private.

README.md

@@ -0,0 +1,103 @@
+---
+language:
+- en
+library_name: transformers
+pipeline_tag: text-generation
+license: other
+license_name: private-research-snapshot
+license_link: LICENSE
+tags:
+- qwen3
+- from-scratch
+- experimental
+- research
+- text-generation
+datasets:
+- sraivante/TinyThinkModel-72M-V1-Data
+model-index:
+- name: TinyThinkModel-72M-V1
+  results:
+  - task:
+      type: text-generation
+      name: Zero-shot continuation scoring
+    dataset:
+      type: Rowan/hellaswag
+      name: HellaSwag validation (10042 examples)
+    metrics:
+    - type: acc
+      name: Accuracy (%)
+      value: 27.1360286795
+    - type: acc_norm
+      name: Normalized accuracy (%)
+      value: 28.3608842860
+---
+# TinyThinkModel-72M-V1
+
+**Private experimental research snapshot by sraivante.** Exactly **72,000,000 parameters**, **1,024-token context**, and a **33,069-token custom byte-level BPE**. Model weights and tokenizer were trained from scratch. The Qwen3-0.6B architecture was a design reference; no pretrained Qwen weights or tokenizer were used. The internal training project was called Think72.
+
+This is the **final export at step 22,889**, after **1,500,053,504 processed token positions**, trained on **two Tesla T4 GPUs**. Training completed on 2026-10-18. The private [associated data repository](https://huggingface.co/datasets/sraivante/TinyThinkModel-72M-V1-Data) contains the exact prepared training, validation and test corpus, tokenizer audit, source hashes, and exact HellaSwag evaluation data and per-question scores. Its release tag is `v1`; the immutable dataset commit is `9d8dd335c0c946ae84ff0919b09c1042da5315f7`.
+
+## Purpose and observed limitations
+
+For studying small language-model training, tokenization, English text continuation and experimental math/science/code responses. This model is unreliable as a general assistant: it can make elementary arithmetic errors, repeat text, and produce incorrect facts or reasoning. The small parameter count, 1,024-token context and predominantly synthetic/reasoning corpus limit general-language performance. No multilingual ability, strong reasoning or production suitability is established. Optional `<think>` text is learned formatting, not a correctness guarantee.
+
+## Evaluation
+
+| Evaluation | Result |
+|---|---:|
+| HellaSwag zero-shot raw accuracy | **27.14%**, 2,725 / 10,042 |
+| HellaSwag zero-shot length-normalized accuracy | **28.36%**, 2,848 / 10,042 |
+| Uniform random-choice expectation | 25% |
+| Final validation loss / perplexity | 1.502758600 / 4.49406933 |
+| Held-out test loss / perplexity | 1.515999056 / 4.55396853 |
+
+HellaSwag used the official `lm-evaluation-harness==0.4.13` task and HFLM scorer on the full validation split, FP32, no quantization, no chat template, no BOS token and no few-shot demonstrations. Normalization uses character length of the official preprocessed ending. All 40,168 continuation requests fit the context (maximum 170 tokens, zero truncation). Both T4 GPUs were used. Raw accuracy's Wilson 95% interval is 26.28–28.01%; normalized accuracy's is 27.49–29.25%. The 3.36 percentage-point normalized improvement over random choice is modest.
+
+Every original benchmark row, label, prompt and choice was checked locally, and both metrics were independently recomputed from recorded scores. This check reuses the saved inference scores; it is not a second inference run. See [evaluation/hellaswag/summary.json](evaluation/hellaswag/summary.json) and [verification.json](evaluation/hellaswag/verification.json). Contamination with training data has not been exhaustively audited. Perplexities each use a deterministic sample of 262,144 target tokens; they are not task-accuracy percentages. The best validation checkpoint was step 22,750 (loss 1.502276920); this release intentionally contains the requested final step 22,889.
+
+## Loading and generation
+
+Install `torch`, `transformers==4.56.2`, `tokenizers==0.22.1` and `safetensors`. Authenticate with a Hugging Face account granted access to these private repositories, for example using `hf auth login`. Do not paste tokens into code or notebooks.
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+repo = "sraivante/TinyThinkModel-72M-V1"
+tokenizer = AutoTokenizer.from_pretrained(repo, revision="v1", token=True)
+model = AutoModelForCausalLM.from_pretrained(
+    repo, revision="v1", token=True, torch_dtype=torch.float32
+).eval()
+messages = [{"role": "user", "content": "What is 12 plus 7?"}]
+inputs = tokenizer.apply_chat_template(
+    messages, tokenize=True, add_generation_prompt=True,
+    return_tensors="pt", return_dict=True,
+    tokenizer_kwargs={"return_token_type_ids": False},
+)
+remaining = 1024 - inputs["input_ids"].shape[1]
+if remaining <= 0:
+    raise ValueError("Prompt must leave room in the trained 1024-token context")
+with torch.inference_mode():
+    output = model.generate(
+        **inputs, max_new_tokens=min(128, remaining), do_sample=False,
+        use_cache=True, pad_token_id=tokenizer.pad_token_id,
+        eos_token_id=tokenizer.eos_token_id,
+    )
+print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=False))
+```
+
+The example runs on CPU. Keep prompt plus generation within 1,024 tokens; there is no validated longer-context mode. The root `model.safetensors` is 288,017,256 bytes. `training/step-00022889.pt` is an optional 864,233,185-byte full resume checkpoint, not required for inference; it contains optimizer/scaler, scheduler, data cursor and both GPU ranks' RNG state. Load full PyTorch checkpoints only from a trusted source.
+
+## Architecture and training
+
+Qwen3ForCausalLM: 14 decoder layers; hidden width 512; SwiGLU intermediate width 1,536; 8 query heads, 4 KV heads, head dimension 128; tied token embeddings/output head; RMSNorm and Q/K RMSNorm; RoPE theta 1,000,000; full causal SDPA attention; no projection bias or dropout. There was one from-scratch next-token training stage on formatted conversations, with no separate SFT, adapter, preference-optimization or RL stage.
+
+Microbatch 8 per GPU, 4 gradient-accumulation steps, global batch 64 sequences, 65,536 token positions/update. AdamW: learning rate 0.0006, 150 warmup steps, cosine decay to 0.00006, betas (0.9, 0.95), epsilon 1e-8, matrix weight decay 0.1, gradient clipping 1.0. FP16 autocast with FP32 weights/optimizer and gradient scaling. Loss covers all non-pad next-token targets, including prompts. See [training/training_settings.json](training/training_settings.json) and `code/` for the original scripts. Source scripts retain their original internal project names for reproducibility.
+
+The prepared corpus contains 1,529,699,784 train tokens / 670,031 records, 7,572,390 validation tokens / 3,348 records, and 7,881,773 test tokens / 3,389 records. Training consumed 1.500B token positions rather than every prepared token. Matching normalized question hashes share a split; exact formatted duplicates were removed. Semantic near-duplicates may remain. Historical upstream revisions of the original local files are unknown; local source hashes and the exact corpus manifest identify the consumed snapshot. Current upstream observations are recorded separately.
+
+Model SHA-256: `0f7bdaaae88077174d371c650759b71b83fce6e917638703b72466d59a2b3c9e`. Corpus manifest SHA-256: `f32e5f3dab44177534cf7457176f83327a02a087c3db2367d771ef64adb2a380`. All release files are listed in `release_files.json`; `release_identity.json` links this checkpoint to the data revision. The final checkpoint has already passed full recovery verification, and local UI inference was exercised on the same exported weights.
+
+## Rights and access
+
+**Private upload only, at the owner's explicit request.** Read [LICENSE](LICENSE) and [THIRD_PARTY.md](THIRD_PARTY.md). Copyright (c) 2026 sraivante applies only to original project contributions and original selection/arrangement under Apache License 2.0. Third-party rights remain intact, including research/non-commercial and possible Llama-related requirements. Neither these weights nor the combined dataset have been cleared for unrestricted public redistribution or commercial use. The exact requested model name is retained for private storage; public naming and license review remain pending.

THIRD_PARTY.md

@@ -0,0 +1,32 @@
+# Third-party attribution and terms
+
+Copyright (c) 2026 sraivante covers only original material and selection/arrangement, under Apache License 2.0. Original dataset creators retain ownership of their questions, answers, reasoning traces and annotations. Formatting, selection and tokenization do not transfer that ownership.
+
+| Source and credited creator | Observed license metadata | Role |
+|---|---|---|
+| [AI-MO/NuminaMath-CoT](https://huggingface.co/datasets/AI-MO/NuminaMath-CoT) | apache-2.0 | training |
+| [GAIR/LIMO](https://huggingface.co/datasets/GAIR/LIMO) | apache-2.0 | training |
+| [NovaSky-AI/Sky-T1_data_17k](https://huggingface.co/datasets/NovaSky-AI/Sky-T1_data_17k) | apache-2.0 | training |
+| [a-m-team/AM-DeepSeek-R1-Distilled-1.4M](https://huggingface.co/datasets/a-m-team/AM-DeepSeek-R1-Distilled-1.4M) | cc-by-nc-4.0 | training |
+| [bespokelabs/Bespoke-Stratos-17k](https://huggingface.co/datasets/bespokelabs/Bespoke-Stratos-17k) | apache-2.0 | training |
+| [meta-math/MetaMathQA](https://huggingface.co/datasets/meta-math/MetaMathQA) | mit | training |
+| [microsoft/orca-math-word-problems-200k](https://huggingface.co/datasets/microsoft/orca-math-word-problems-200k) | mit | training |
+| [nvidia/Llama-Nemotron-Post-Training-Dataset](https://huggingface.co/datasets/nvidia/Llama-Nemotron-Post-Training-Dataset) | cc-by-4.0 | training |
+| [open-r1/Mixture-of-Thoughts](https://huggingface.co/datasets/open-r1/Mixture-of-Thoughts) | No blanket license declared | training |
+| [open-r1/OpenR1-Math-220k](https://huggingface.co/datasets/open-r1/OpenR1-Math-220k) | apache-2.0 | training |
+| [open-thoughts/OpenThoughts-114k](https://huggingface.co/datasets/open-thoughts/OpenThoughts-114k) | apache-2.0 | training |
+| [openai/gsm8k](https://huggingface.co/datasets/openai/gsm8k) | ['mit'] | training |
+| [simplescaling/s1K](https://huggingface.co/datasets/simplescaling/s1K) | apache-2.0 | training |
+| [simplescaling/s1K-1.1](https://huggingface.co/datasets/simplescaling/s1K-1.1) | mit | training |
+| [Rowan/hellaswag](https://huggingface.co/datasets/Rowan/hellaswag) | No blanket license declared | evaluation |
+| [open-r1/codeforces-cots](https://huggingface.co/datasets/open-r1/codeforces-cots) | cc-by-4.0 | component reference |
+
+The source links credit the publishing organizations/authors and provide their dataset cards and citations. The recorded upstream revisions were observed on 2026-10-18; they are not represented as historical revisions of local training files. The source manifest records exact local SHA-256 identities and the corpus manifest records the selection actually prepared for this run.
+
+The [AM dataset usage limits](https://huggingface.co/datasets/a-m-team/AM-DeepSeek-R1-Distilled-1.4M#limitation-and-usage-limits) specify research-only, non-commercial use. [NVIDIA's terms](https://huggingface.co/datasets/nvidia/Llama-Nemotron-Post-Training-Dataset#licenseterms-of-use) distinguish per-record CC-BY-4.0, ODC-BY and CC-BY-SA licenses and reference Llama requirements. Mixture-of-Thoughts includes OpenR1-Math, CodeForces CoTs and NVIDIA science data. The five-column local cleaned schema does not preserve original per-record license/generator metadata; those permissions must be resolved before public data redistribution.
+
+The [Llama 3.1](https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/LICENSE) and [Llama 3.3](https://github.com/meta-llama/llama-models/blob/main/models/llama3_3/LICENSE) licenses include requirements for models trained on applicable outputs, including a Llama name prefix. This is why the exact requested model name is currently retained only in a private repository. This architecture uses Qwen3 code with random weights, and contains no pretrained Qwen or Llama weight initialization.
+
+HellaSwag is credited to Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi and Yejin Choi, 2019, [HellaSwag: Can a Machine Really Finish Your Sentence?](https://arxiv.org/abs/1905.07830). The exact Rowan/HellaSwag validation snapshot is archived privately for reproducible evaluation; its license metadata is absent and public redistribution clearance has not been established. It was not intentionally added to the training corpus, but contamination has not been exhaustively audited.
+
+Qwen3's architecture reference is [Qwen/Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B); the implementation is Hugging Face Transformers 4.56.2. The official evaluation implementation is [EleutherAI/lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness), pinned to 0.4.13. Their code and respective licenses remain with their authors.

chat_template.jinja

@@ -0,0 +1,4 @@
+{% for message in messages %}{{ '<|im_start|>' + message['role'] + '
+' + message['content'] + '<|im_end|>
+' }}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant
+' }}{% endif %}

code/checkpoint_cloud.py

@@ -0,0 +1,95 @@
+"""Optional finite Kaggle continuation using a notebook-scoped Kaggle Secret.
+
+Never embeds, writes or logs the credential. Enabled only by the launcher after
+the user authorizes the Secret and its binding to this notebook.
+"""
+import json
+import os
+from pathlib import Path
+import shutil
+import time
+from common import dump, sha, now
+
+BACKUP_SLUG='faithdemo/think72-training-checkpoints'
+
+def api_client():
+    if not os.environ.get('KAGGLE_API_TOKEN'):
+        from kaggle_secrets import UserSecretsClient
+        os.environ['KAGGLE_API_TOKEN']=UserSecretsClient().get_secret('THINK72_KAGGLE_TOKEN')
+    from kaggle.api.kaggle_api_extended import KaggleApi
+    api=KaggleApi();api.authenticate()
+    assert api.config_values.get('username')=='faithdemo'
+    return api
+
+def backup_checkpoint(out, checkpoint, force=False):
+    if os.environ.get('THINK72_CLOUD_RESUME')!='1':return False
+    out=Path(out); ledger=out/'cloud_backup_status.json'
+    previous=json.loads(ledger.read_text()) if ledger.exists() else {}
+    if not force and time.time()-previous.get('last_success_epoch',0)<7200:return True
+    api=api_client()
+    stage=out.parent/'cloud_checkpoint';stage.mkdir(exist_ok=True)
+    target=stage/'resume.pt'
+    if target.exists():
+        assert target.resolve().parent==stage.resolve()
+        target.unlink()
+    os.link(checkpoint,target)
+    step=json.loads((out/'checkpoints/latest.json').read_text())['step']
+    state={'step':step,'checkpoint_sha256':sha(target),'bytes':target.stat().st_size,
+           'session_index':int(os.environ.get('THINK72_SESSION_INDEX','1')),'created_utc':now()}
+    dump(stage/'cloud_state.json',state)
+    dump(stage/'dataset-metadata.json',{
+        'id':BACKUP_SLUG,'title':'Think72 Training Checkpoints','licenses':[{'name':'other'}],
+        'description':'Private resumable 72M Qwen3-style model training state: model weights, optimizer, scaler, RNG and data cursor. Related token snapshot: https://www.kaggle.com/datasets/faithdemo/think72-english-tokens-v1 . Original project contributions Copyright (c) 2026 sraivante, Apache-2.0; training data retains third-party terms including CC-BY-NC-4.0. No blanket commercial license is asserted.'})
+    exists=True
+    try:api.dataset_status(BACKUP_SLUG)
+    except Exception as exc:
+        response=getattr(exc,'response',None)
+        if response is not None and response.status_code in (403,404):
+            # Kaggle returns 403 for absent private dataset slugs. Confirm the
+            # authenticated owner's matching datasets before attempting create.
+            owned=api.dataset_list(mine=True,search='Think72 Training Checkpoints') or []
+            if any(getattr(item,'ref',None)==BACKUP_SLUG for item in owned):raise
+            exists=False
+        else:raise
+    if exists:
+        result=api.dataset_create_version(str(stage),f'Checkpoint step {step}',quiet=True,
+                                         convert_to_csv=False,delete_old_versions=False)
+    else:result=api.dataset_create_new(str(stage),public=False,quiet=True,convert_to_csv=False)
+    error=getattr(result,'error',None)
+    assert not error,'Checkpoint dataset upload did not succeed'
+    for _ in range(120):
+        status=api.dataset_status(BACKUP_SLUG)
+        if status=='ready':break
+        if status in ('error','failed'):raise RuntimeError('Checkpoint dataset processing failed')
+        time.sleep(5)
+    else:raise TimeoutError('Checkpoint dataset not ready')
+    dump(ledger,{**state,'dataset':BACKUP_SLUG,'last_success_epoch':time.time(),'status':'READY'})
+    print('CLOUD_CHECKPOINT_READY',step,flush=True)
+    return True
+
+def continue_if_needed(root, source_script):
+    root=Path(root);out=root/'run'
+    status=json.loads((out/'run_status.json').read_text())
+    latest=json.loads((out/'checkpoints/latest.json').read_text())
+    backup_checkpoint(out,out/'checkpoints'/latest['path'],force=True)
+    if status['status']=='COMPLETE':return
+    assert status['status']=='CHECKPOINTED_NEEDS_RESUME'
+    index=int(os.environ.get('THINK72_SESSION_INDEX','1'))
+    assert index<4,'Automatic continuation limit reached; inspect progress before further sessions'
+    next_dir=root/'next_session';next_dir.mkdir(exist_ok=True)
+    shutil.copyfile(source_script,next_dir/'run.py')
+    dump(next_dir/'kernel-metadata.json',{
+        'id':'faithdemo/think72-qwen3-t4x2','title':'Think72 Qwen3 T4x2',
+        'code_file':'run.py','language':'python','kernel_type':'script','is_private':True,
+        'enable_gpu':True,'enable_internet':True,'machine_shape':'NvidiaTeslaT4',
+        'dataset_sources':['faithdemo/think72-english-tokens-v1',BACKUP_SLUG],
+        'kernel_sources':[],'competition_sources':[],'model_sources':[],
+        'docker_image_pinning_type':'latest'})
+    api=api_client()
+    result=api.kernels_push(str(next_dir),timeout='43200',acc='NvidiaTeslaT4')
+    assert not getattr(result,'error',None),'Continuation request failed'
+    report={'from_step':status['step'],'next_session_index':index+1,
+            'requested_accelerator':'NvidiaTeslaT4','url':getattr(result,'url',None),
+            'submitted_utc':now(),'target_tokens':1_500_000_000}
+    dump(root/'continuation_submitted.json',report)
+    print('NEXT_T4X2_SESSION_SUBMITTED',json.dumps(report),flush=True)

code/common.py

@@ -0,0 +1,66 @@
+"""Shared, deterministic model/data definitions. Copyright (c) 2026 sraivante."""
+from __future__ import annotations
+import hashlib
+import json
+import re
+import unicodedata
+from datetime import datetime, timezone
+from pathlib import Path
+
+SEED = 20261017
+CONTEXT = 1024
+VOCAB = 33069
+SPECIALS = ["<|pad|>", "<|endoftext|>", "<|im_end|>", "<|im_start|>", "<think>", "</think>"]
+MAX_SOURCE_BYTES = 20_000_000_000
+MAX_FILE_BYTES = 7_000_000_000
+
+def now():
+    return datetime.now(timezone.utc).isoformat()
+
+def sha(path):
+    h = hashlib.sha256()
+    with Path(path).open('rb') as f:
+        for chunk in iter(lambda: f.read(8 * 1024 * 1024), b''):
+            h.update(chunk)
+    return h.hexdigest()
+
+def dump(path, value):
+    path = Path(path)
+    path.parent.mkdir(parents=True, exist_ok=True)
+    tmp = path.with_name(path.name + '.partial')
+    tmp.write_text(json.dumps(value, indent=2, ensure_ascii=False) + '\n', encoding='utf-8')
+    tmp.replace(path)
+
+def prompt_key(prompt):
+    # All identical normalized questions share a split, including different solutions.
+    s = re.sub(r'\s+', ' ', unicodedata.normalize('NFKC', str(prompt))).strip().casefold()
+    return hashlib.sha256(s.encode('utf-8')).digest()
+
+def split_for(prompt):
+    bucket = int.from_bytes(prompt_key(prompt)[:8], 'big') % 1000
+    return 'validation' if bucket < 5 else ('test' if bucket < 10 else 'train')
+
+def render_row(row):
+    prompt = str(row.get('prompt') or '').strip()
+    answer = str(row.get('answer') or '').strip()
+    thought = str(row.get('think') or '').strip()
+    if not prompt or not (answer or thought):
+        return None
+    # Preserve source Unicode, math, code indentation and newlines. No ASCII filtering.
+    reasoning = '<think>\n' + thought + '\n</think>\n' if thought else ''
+    return ('<|im_start|>user\n' + prompt + '<|im_end|>\n'
+            + '<|im_start|>assistant\n' + reasoning + answer + '<|im_end|>\n<|endoftext|>')
+
+def model_config():
+    from transformers import Qwen3Config
+    return Qwen3Config(
+        vocab_size=VOCAB, hidden_size=512, intermediate_size=1536,
+        num_hidden_layers=14, num_attention_heads=8, num_key_value_heads=4,
+        head_dim=128, max_position_embeddings=CONTEXT,
+        hidden_act='silu', rms_norm_eps=1e-6, rope_theta=1_000_000.0,
+        attention_bias=False, attention_dropout=0.0, tie_word_embeddings=True,
+        use_sliding_window=False, sliding_window=None, max_window_layers=14,
+        pad_token_id=0, bos_token_id=1, eos_token_id=2, use_cache=False,
+        architectures=['Qwen3ForCausalLM'],
+    )
+

code/inventory.py

@@ -0,0 +1,40 @@
+"""Read-only source audit. Never modifies the supplied source directory."""
+import argparse
+from collections import Counter
+from pathlib import Path
+import pyarrow.parquet as pq
+from common import dump, sha, now, MAX_SOURCE_BYTES, MAX_FILE_BYTES
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument('--source', default='D:/AI_DATA/wiki_clean/thinking/english')
+    ap.add_argument('--out', default='artifacts/source_manifest.json')
+    args = ap.parse_args()
+    files, excluded, total = [], [], 0
+    for p in sorted(Path(args.source).glob('*.parquet')):
+        size = p.stat().st_size
+        if size >= MAX_FILE_BYTES or total + size >= MAX_SOURCE_BYTES:
+            excluded.append({'name': p.name, 'bytes': size, 'reason': 'strict byte limit'})
+            continue
+        pf = pq.ParquetFile(p)
+        assert {'source','kind','prompt','think','answer'} <= set(pf.schema_arrow.names)
+        row = next(pf.iter_batches(batch_size=1)).to_pylist()[0]
+        owner, repo = p.name.split('__')[:2]
+        files.append({'name': p.name, 'bytes': size, 'sha256': sha(p),
+                      'rows': pf.metadata.num_rows, 'row_groups': pf.num_row_groups,
+                      'source': row['source'], 'upstream_repo': owner + '/' + repo,
+                      'source_revision': None, 'local_mtime_ns': p.stat().st_mtime_ns})
+        total += size
+        print('AUDIT', len(files), p.name, size, flush=True)
+    manifest = {'created_utc': now(), 'source_directory': str(Path(args.source).resolve()),
+                'source_revision_note': 'Historical upstream revisions were not retained in these cleaned local files. Local SHA-256 values define this exact snapshot; current upstream revisions are not claimed as historical identities.',
+                'limits': {'total_bytes_strictly_less_than': MAX_SOURCE_BYTES,
+                           'single_file_bytes_strictly_less_than': MAX_FILE_BYTES},
+                'file_count': len(files), 'total_bytes': total,
+                'total_rows': sum(x['rows'] for x in files),
+                'files': files, 'excluded': excluded}
+    dump(args.out, manifest)
+    print('AUDIT_COMPLETE', len(files), total, manifest['total_rows'], flush=True)
+
+if __name__ == '__main__': main()
+

code/prepare.py

@@ -0,0 +1,220 @@
+"""Train a byte-level BPE and build audited, streaming 1024-token LM data."""
+from __future__ import annotations
+import argparse
+from collections import Counter
+import json
+import os
+os.environ.setdefault('RAYON_NUM_THREADS', '4')
+import random
+from pathlib import Path
+import time
+import numpy as np
+import pyarrow.parquet as pq
+from tokenizers import Tokenizer, models, trainers, pre_tokenizers, decoders
+from common import *
+
+CHAT_TEMPLATE = "{% for message in messages %}{{ '<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>\n' }}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}"
+
+def rows(path, seed=SEED):
+    pf = pq.ParquetFile(path)
+    groups = list(range(pf.num_row_groups))
+    random.Random(seed).shuffle(groups)
+    for group in groups:
+        row_offset = sum(pf.metadata.row_group(i).num_rows for i in range(group))
+        for batch in pf.iter_batches(batch_size=128, row_groups=[group],
+                                     columns=['source','kind','prompt','think','answer']):
+            for i, row in enumerate(batch.to_pylist()):
+                yield row_offset + i, row
+            row_offset += batch.num_rows
+
+def train_tokenizer(source, manifest, out, sample_bytes):
+    from transformers import PreTrainedTokenizerFast
+    counts = Counter()
+    byte_counts = Counter()
+    def texts():
+        # Each source file gets the same byte allowance, avoiding a tokenizer
+        # dominated by the two largest distillation files.
+        per_file = max(65536, sample_bytes // len(manifest['files']))
+        for item in manifest['files']:
+            for _, row in rows(source / item['name']):
+                if split_for(row['prompt']) != 'train': continue
+                text = render_row(row)
+                if text is None: continue
+                yield text
+                counts[item['name']] += 1
+                byte_counts[item['name']] += len(text.encode('utf-8'))
+                if byte_counts[item['name']] >= per_file: break
+            print('TOKENIZER_SAMPLE', item['name'], counts[item['name']], flush=True)
+    tokenizer = Tokenizer(models.BPE())
+    tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=False, use_regex=True)
+    tokenizer.decoder = decoders.ByteLevel()
+    trainer = trainers.BpeTrainer(vocab_size=VOCAB, min_frequency=2,
+                                 special_tokens=SPECIALS,
+                                 initial_alphabet=pre_tokenizers.ByteLevel.alphabet(),
+                                 show_progress=False)
+    tokenizer.train_from_iterator(texts(), trainer=trainer)
+    if tokenizer.get_vocab_size() < VOCAB:
+        tokenizer.add_tokens([f'<|reserved_{i}|>' for i in range(VOCAB-tokenizer.get_vocab_size())])
+    assert tokenizer.get_vocab_size() == VOCAB
+    fast = PreTrainedTokenizerFast(tokenizer_object=tokenizer,
+                                  pad_token=SPECIALS[0], bos_token=SPECIALS[1], eos_token=SPECIALS[2],
+                                  additional_special_tokens=SPECIALS[3:], model_max_length=CONTEXT,
+                                  clean_up_tokenization_spaces=False)
+    fast.chat_template = CHAT_TEMPLATE
+    out.mkdir(parents=True, exist_ok=True)
+    fast.save_pretrained(out)
+    report = tokenizer_checks(fast)
+    report.update({'training_split_only': True, 'sample_rows_by_file': dict(counts),
+                   'sample_utf8_bytes_by_file': dict(byte_counts),
+                   'normalizer': None, 'algorithm': 'byte-level BPE',
+                   'tokenizer_sha256': sha(out/'tokenizer.json')})
+    dump(out/'tokenizer_audit.json', report)
+    print('TOKENIZER_VERIFIED', report['vocab_size'], flush=True)
+    return fast
+
+def tokenizer_checks(fast):
+    samples = ['Hello, world!\n  Two spaces stay.',
+               'x² + y² = π; √2 ≈ 1.414; café — naïve 🚀',
+               'def add(a, b):\n    return a + b\n',
+               r'\frac{13}{6} = 2.1667; $x \\ y$',
+               '<|im_start|>assistant\n<think>\nCheck.\n</think>\n42<|im_end|>']
+    for text in samples:
+        ids = fast.encode(text, add_special_tokens=False)
+        assert fast.decode(ids, skip_special_tokens=False, clean_up_tokenization_spaces=False) == text
+        assert all(0 <= t < VOCAB for t in ids)
+    special_ids = {s: fast.encode(s, add_special_tokens=False) for s in SPECIALS}
+    assert special_ids == {s:[i] for i,s in enumerate(SPECIALS)}, special_ids
+    assert fast.model_max_length == CONTEXT
+    return {'vocab_size': len(fast), 'special_token_ids': special_ids,
+            'round_trip_cases_passed': len(samples), 'context': CONTEXT,
+            'no_unknown_token': fast.unk_token_id is None}
+
+class TokenWriter:
+    # 512 MB per shard: below the requested 7 GB ceiling.
+    def __init__(self, directory, split, shard_tokens=256_000_000):
+        self.directory = Path(directory); self.directory.mkdir(parents=True, exist_ok=True)
+        self.split = split; self.shard_tokens = shard_tokens
+        self.real_tokens = 0; self.total_tokens = 0; self.current_tokens = 0
+        self.files = []; self.handle = None; self.path = None
+    def add(self, ids, padding=False):
+        ids = np.asarray(ids, dtype='<u2')
+        if ids.size: assert int(ids.max()) < VOCAB
+        if not padding: self.real_tokens += len(ids)
+        pos = 0
+        while pos < len(ids):
+            if self.handle is None:
+                self.path = self.directory/f'{self.split}-{len(self.files):04d}.bin'
+                self.handle = self.path.open('wb'); self.current_tokens = 0
+            n = min(len(ids)-pos, self.shard_tokens-self.current_tokens)
+            self.handle.write(ids[pos:pos+n].tobytes())
+            self.current_tokens += n; self.total_tokens += n; pos += n
+            if self.current_tokens == self.shard_tokens: self.close_shard()
+    def close_shard(self):
+        if self.handle is None: return
+        self.handle.close(); self.handle = None
+        self.files.append({'name': self.path.name, 'tokens': self.current_tokens,
+                           'bytes': self.path.stat().st_size, 'sha256': sha(self.path)})
+    def finish(self):
+        if self.total_tokens:
+            # Each sample uses 1024 inputs and the next 1024 target tokens.
+            # Pad only the final tail; padding targets are ignored by the loss.
+            self.add([0] * ((1-self.total_tokens) % CONTEXT), padding=True)
+        self.close_shard()
+        return {'real_tokens': self.real_tokens, 'stored_tokens': self.total_tokens,
+                'padding_tokens': self.total_tokens-self.real_tokens,
+                'blocks': max(0,(self.total_tokens-1)//CONTEXT), 'files': self.files}
+
+def prepare(source, manifest_path, output, train_tokens, sample_bytes, verify=True, tokenizer_path=None):
+    source = Path(source); output = Path(output); output.mkdir(parents=True, exist_ok=True)
+    manifest = json.loads(Path(manifest_path).read_text(encoding='utf-8'))
+    assert manifest['total_bytes'] < MAX_SOURCE_BYTES
+    assert all(f['bytes'] < MAX_FILE_BYTES for f in manifest['files'])
+    for item in manifest['files']:
+        path = source/item['name']
+        assert path.stat().st_size == item['bytes'], path
+        if verify:
+            assert sha(path) == item['sha256'], path
+            print('SOURCE_HASH_OK', item['name'], flush=True)
+    dump(output/'source_manifest.json', manifest)
+    if tokenizer_path:
+        import shutil
+        from transformers import AutoTokenizer
+        shutil.copytree(tokenizer_path,output/'tokenizer',dirs_exist_ok=True)
+        fast=AutoTokenizer.from_pretrained(output/'tokenizer',local_files_only=True)
+        tokenizer_checks(fast)
+    else:
+        fast = train_tokenizer(source, manifest, output/'tokenizer', sample_bytes)
+    tokenizer = fast.backend_tokenizer
+    writers = {s:TokenWriter(output, s) for s in ['train','validation','test']}
+    selected = Counter(); kinds = Counter(); source_tokens = Counter()
+    sample_audit = []; duplicates = 0; scanned = 0
+    # Store compact hashes, not full documents. This set is bounded by selected records.
+    seen = set()
+    items = manifest['files'].copy(); random.Random(SEED).shuffle(items)
+    quota_basis = sum(x['bytes'] for x in items)
+    started = time.time()
+    for item in items:
+        quota = max(CONTEXT*4, round(train_tokens * item['bytes']/quota_basis))
+        file_train = 0; pending = []; pending_meta = []
+        def flush():
+            nonlocal file_train, duplicates
+            if not pending: return
+            encodings = tokenizer.encode_batch(pending, add_special_tokens=False)
+            for text, encoding, meta in zip(pending, encodings, pending_meta):
+                idx, row, split = meta
+                digest = __import__('hashlib').sha256(text.encode('utf-8')).digest()
+                if digest in seen:
+                    duplicates += 1; continue
+                seen.add(digest)
+                ids = encoding.ids
+                writers[split].add(ids)
+                selected[split] += 1; kinds[row['kind']] += 1
+                source_tokens[item['name'] + '/' + split] += len(ids)
+                if split == 'train': file_train += len(ids)
+                if len(sample_audit) < 100 or split != 'train':
+                    # No source text in audit: positions and hashes identify records.
+                    sample_audit.append({'file':item['name'], 'row':idx, 'split':split,
+                                         'prompt_sha256':prompt_key(row['prompt']).hex(),
+                                         'record_sha256':digest.hex(), 'tokens':len(ids)})
+            pending.clear(); pending_meta.clear()
+        for idx, row in rows(source/item['name']):
+            scanned += 1
+            text = render_row(row)
+            if text is None: continue
+            split = split_for(row['prompt'])
+            pending.append(text); pending_meta.append((idx,row,split))
+            if len(pending) >= 64:
+                flush()
+                if file_train >= quota: break
+        flush()
+        print('TOKENIZED_FILE', item['name'], 'train_tokens', file_train,
+              'total_train', writers['train'].real_tokens, 'seconds', round(time.time()-started), flush=True)
+    splits = {s:w.finish() for s,w in writers.items()}
+    assert all(splits[s]['blocks'] > 0 for s in splits), 'Each split needs data'
+    assert sum(x['bytes'] for s in splits.values() for x in s['files']) < MAX_SOURCE_BYTES
+    assert all(x['bytes'] < MAX_FILE_BYTES for s in splits.values() for x in s['files'])
+    result = {'created_utc':now(), 'seed':SEED, 'dtype':'little-endian uint16',
+              'context':CONTEXT, 'vocab_size':VOCAB,
+              'source_manifest_sha256':sha(manifest_path),
+              'tokenizer_sha256':sha(output/'tokenizer'/'tokenizer.json'),
+              'requested_train_tokens':train_tokens,
+              'sampling':'Deterministically shuffled row groups; per-file train-token quota proportional to compressed file bytes; retain whole records including final answers. Quotas may overshoot by one encoding batch.',
+              'split_rule':'SHA256(NFKC/casefold/whitespace-normalized prompt) modulo 1000: validation 0..4; test 5..9; train 10..999. Split before tokenizer fitting and packing.',
+              'packing':'Separate continuous token streams per split; EOS separates records; long documents continue into later windows. Attention resets every 1024 tokens; cross-document attention within a window is allowed.',
+              'loss':'All next-token targets except pad=0; no assistant-only loss at scratch-pretraining stage.',
+              'rows_scanned':scanned, 'selected_rows':dict(selected), 'exact_duplicate_rows_removed':duplicates,
+              'source_tokens':dict(source_tokens), 'kinds':dict(kinds), 'splits':splits}
+    dump(output/'split_record_audit.json', sample_audit)
+    dump(output/'corpus_manifest.json', result)
+    model_config().save_pretrained(output/'model_config')
+    print('PREPARE_COMPLETE', json.dumps({k:v['real_tokens'] for k,v in splits.items()}), flush=True)
+
+if __name__ == '__main__':
+    ap=argparse.ArgumentParser()
+    ap.add_argument('--source',required=True); ap.add_argument('--manifest',required=True)
+    ap.add_argument('--out',required=True)
+    ap.add_argument('--train-tokens',type=int,default=1_520_000_000)
+    ap.add_argument('--tokenizer-sample-bytes',type=int,default=128_000_000)
+    ap.add_argument('--tokenizer')
+    args=ap.parse_args()
+    prepare(args.source,args.manifest,args.out,args.train_tokens,args.tokenizer_sample_bytes,tokenizer_path=args.tokenizer)

code/requirements.txt

@@ -0,0 +1,8 @@
+torch>=2.5,<3
+transformers==4.56.2
+tokenizers==0.22.1
+huggingface-hub==0.35.3
+safetensors>=0.6,<1
+pyarrow>=18,<23
+numpy>=1.26,<3
+

code/stage_tokens.py

@@ -0,0 +1,41 @@
+"""Compress closed token shards for transfer; source and token hashes stay intact."""
+import argparse
+import gzip
+import json
+import shutil
+from pathlib import Path
+from common import dump, sha
+
+def stage():
+    source=Path('artifacts/corpus'); out=Path('artifacts/upload_tokens');out.mkdir(exist_ok=True)
+    complete=(source/'corpus_manifest.json').exists()
+    for split in ['train','validation','test']:
+        files=sorted(source.glob(split+'-*.bin'))
+        ready=files if complete else files[:-1]
+        for path in ready:
+            target=out/(path.name+'.gz')
+            if target.exists():continue
+            tmp=target.with_suffix('.gz.partial')
+            with path.open('rb') as inp,tmp.open('wb') as raw:
+                with gzip.GzipFile(filename='',mode='wb',compresslevel=1,fileobj=raw,mtime=0) as zipped:
+                    shutil.copyfileobj(inp,zipped,8*1024*1024)
+            tmp.replace(target)
+            print('COMPRESSED',target.name,path.stat().st_size,target.stat().st_size,flush=True)
+    if complete:
+        for name in ['corpus_manifest.json','source_manifest.json','split_record_audit.json']:
+            shutil.copy2(source/name,out/name)
+        for name in ['DATA_CARD.md','README.md','LICENSE']:
+            shutil.copy2(name,out/name)
+        shutil.copy2('artifacts/provenance.json',out/'provenance.json')
+        shutil.copy2('artifacts/tokenizer/tokenizer_audit.json',out/'tokenizer_audit.json')
+        import zipfile
+        with zipfile.ZipFile(out/'tokenizer.zip','w',zipfile.ZIP_DEFLATED) as z:
+            for p in Path('artifacts/tokenizer').glob('*'):z.write(p,'tokenizer/'+p.name)
+        total=sum(p.stat().st_size for p in out.iterdir() if p.is_file())
+        assert total<20_000_000_000
+        assert all(p.stat().st_size<7_000_000_000 for p in out.iterdir() if p.is_file())
+        dump(out/'transfer_manifest.json',{'bytes':total,'files':{
+            p.name:{'bytes':p.stat().st_size,'sha256':sha(p)} for p in out.iterdir() if p.is_file() and p.name!='transfer_manifest.json'}})
+        print('TRANSFER_SNAPSHOT_COMPLETE',total,flush=True)
+
+if __name__=='__main__':stage()

code/train.py

@@ -0,0 +1,376 @@
+"""Qwen3 72M scratch training with true two-GPU DDP and complete resume state."""
+from __future__ import annotations
+import argparse
+import bisect
+from contextlib import nullcontext
+from datetime import timedelta
+import json
+import math
+import os
+import random
+import shutil
+import signal
+import time
+from pathlib import Path
+import numpy as np
+import torch
+import torch.distributed as dist
+import torch.nn.functional as F
+from torch.nn.parallel import DistributedDataParallel as DDP
+from transformers import Qwen3ForCausalLM, AutoTokenizer
+from common import SEED, CONTEXT, VOCAB, dump, sha, now, model_config
+
+class PackedData:
+    def __init__(self, root, manifest, split):
+        self.arrays = [np.memmap(Path(root)/f['name'], mode='r', dtype='<u2')
+                       for f in manifest['splits'][split]['files']]
+        self.ends = np.cumsum([len(a) for a in self.arrays]).tolist()
+        self.blocks = (self.ends[-1]-1)//CONTEXT
+        assert self.blocks > 0
+    def get(self, index):
+        offset = index*CONTEXT
+        shard = bisect.bisect_right(self.ends, offset)
+        local = offset-(self.ends[shard-1] if shard else 0)
+        a = self.arrays[shard][local:local+CONTEXT+1]
+        if len(a) < CONTEXT+1:
+            a = np.concatenate([a, self.arrays[shard+1][:(CONTEXT+1-len(a))]])
+        return np.asarray(a, dtype=np.int64)
+    def batch(self, indices, device):
+        a = np.stack([self.get(i) for i in indices])
+        t = torch.from_numpy(a).to(device)
+        return t[:,:-1].contiguous(), t[:,1:].contiguous()
+
+def permutation(index, count, seed=SEED):
+    epoch, pos = divmod(index, count)
+    rng = random.Random(seed+epoch)
+    a = rng.randrange(1,max(2,count))
+    while math.gcd(a,count) != 1: a = (a+1)%count or 1
+    b = rng.randrange(count)
+    return (a*pos+b)%count
+
+def indices_for(step, micro, batch_size, accumulation, world, rank, count):
+    start = ((step*accumulation+micro)*world+rank)*batch_size
+    return [permutation(start+i,count) for i in range(batch_size)]
+
+def lr_for(step, total, warmup, peak, minimum):
+    if step < warmup: return peak*(step+1)/max(1,warmup)
+    progress = min(1.0,(step-warmup)/max(1,total-warmup))
+    return minimum+(peak-minimum)*0.5*(1+math.cos(math.pi*progress))
+
+def rng_state(device):
+    state = {'python':random.getstate(), 'numpy':np.random.get_state(),
+             'torch':torch.get_rng_state()}
+    if device.type == 'cuda': state['cuda'] = torch.cuda.get_rng_state(device)
+    return state
+
+def restore_rng(state,device):
+    random.setstate(state['python']); np.random.set_state(state['numpy'])
+    torch.set_rng_state(state['torch'].cpu())
+    if device.type == 'cuda': torch.cuda.set_rng_state(state['cuda'].cpu(),device)
+
+def save_checkpoint(out, model, optimizer, scaler, step, best, settings,
+                    rank, world, device, best_now=False):
+    states = [None]*world if rank == 0 else None
+    local = rng_state(device)
+    if world > 1: dist.gather_object(local, states, dst=0)
+    else: states = [local]
+    if rank == 0:
+        directory = out/'checkpoints'; directory.mkdir(parents=True,exist_ok=True)
+        path = directory/f'step-{step:08d}.pt'
+        payload = {'format_version':1, 'step':step, 'best_validation_loss':best,
+                   'model':model.state_dict(), 'optimizer':optimizer.state_dict(),
+                   'scaler':scaler.state_dict(), 'rng_by_rank':states,
+                   'settings':settings, 'data_cursor_global_blocks':step*settings['blocks_per_step'],
+                   'scheduler':{'type':'warmup_cosine','next_step':step},
+                   'created_utc':now()}
+        tmp = path.with_suffix('.pt.partial'); torch.save(payload,tmp); tmp.replace(path)
+        info = {'path':path.name,'step':step,'sha256':sha(path),'bytes':path.stat().st_size,
+                'corpus_manifest_sha256':settings['corpus_manifest_sha256']}
+        dump(directory/'latest.json',info)
+        if best_now:
+            best_path = directory/'best.pt'
+            shutil.copyfile(path,best_path.with_suffix('.pt.partial'))
+            best_path.with_suffix('.pt.partial').replace(best_path)
+            dump(directory/'best.json',{**info,'path':'best.pt','validation_loss':best})
+        # Delete only generated checkpoint files in this run's checked directory.
+        checked_root = directory.resolve()
+        for old in sorted(directory.glob('step-*.pt'))[:-2]:
+            assert old.resolve().parent == checked_root
+            old.unlink()
+        print('CHECKPOINT_SAVED',json.dumps(info),flush=True)
+        if os.environ.get('THINK72_CLOUD_RESUME')=='1':
+            try:
+                from checkpoint_cloud import backup_checkpoint
+                backup_checkpoint(out,path)
+            except Exception as exc:
+                # Retain local checkpoints and log only exception type, never secrets/URLs.
+                print('CLOUD_BACKUP_RETRY_AT_NEXT_CHECKPOINT',type(exc).__name__,flush=True)
+    if world > 1: dist.barrier()
+
+@torch.no_grad()
+def evaluate(model, data, device, rank, world, batches, batch_size, amp):
+    model.eval()
+    totals = torch.zeros(2,device=device,dtype=torch.float64)
+    for i in range(batches):
+        indices = [permutation((i*world+rank)*batch_size+j,data.blocks,SEED+71)
+                   for j in range(batch_size)]
+        x,y = data.batch(indices,device)
+        with torch.autocast(device_type=device.type,dtype=torch.float16,enabled=amp):
+            logits = model(input_ids=x,use_cache=False).logits
+            loss = F.cross_entropy(logits.reshape(-1,VOCAB).float(),y.reshape(-1),
+                                   ignore_index=0,reduction='sum')
+        totals[0] += loss.double(); totals[1] += (y != 0).sum()
+    if world > 1: dist.all_reduce(totals)
+    model.train()
+    nll = (totals[0]/totals[1]).item()
+    return {'loss':nll,'perplexity':math.exp(min(50,nll)),'evaluated_target_tokens':int(totals[1].item())}
+
+@torch.no_grad()
+def generate_samples(model, tokenizer, device, amp, max_new_tokens=96):
+    model.eval()
+    generations=[]
+    for prompt in ['What is 12 plus 7?', 'If a box has 3 rows of 4 apples, how many apples are there?',
+                   'Write a Python function that returns the square of a number.']:
+        inputs=tokenizer.apply_chat_template([{'role':'user','content':prompt}],
+                     add_generation_prompt=True,return_tensors='pt',return_dict=True,
+                     tokenizer_kwargs={'return_token_type_ids':False}).to(device)
+        with torch.autocast(device_type=device.type,dtype=torch.float16,enabled=amp):
+            generated=model.generate(**inputs,max_new_tokens=max_new_tokens,do_sample=False,use_cache=True,
+                                     pad_token_id=0,eos_token_id=2)
+        generations.append({'prompt':prompt,'response':tokenizer.decode(generated[0,inputs['input_ids'].shape[1]:],
+                                             skip_special_tokens=False,clean_up_tokenization_spaces=False)})
+    return generations
+
+def main():
+    ap=argparse.ArgumentParser()
+    ap.add_argument('--data',required=True); ap.add_argument('--out',required=True)
+    ap.add_argument('--resume'); ap.add_argument('--device',default='cuda',choices=['cpu','cuda'])
+    ap.add_argument('--require-two-t4',action='store_true')
+    ap.add_argument('--batch-size',type=int,default=8); ap.add_argument('--grad-accum',type=int,default=4)
+    ap.add_argument('--target-tokens',type=int,default=1_500_000_000)
+    ap.add_argument('--warmup-steps',type=int,default=150)
+    ap.add_argument('--lr',type=float,default=6e-4); ap.add_argument('--min-lr',type=float,default=6e-5)
+    ap.add_argument('--save-every',type=int,default=250); ap.add_argument('--save-seconds',type=int,default=600)
+    ap.add_argument('--eval-every',type=int,default=250); ap.add_argument('--eval-batches',type=int,default=16)
+    ap.add_argument('--max-hours',type=float,default=10.5)
+    ap.add_argument('--stop-after-steps',type=int,default=0)
+    ap.add_argument('--threads',type=int,default=2)
+    args=ap.parse_args()
+    rank=int(os.environ.get('RANK','0')); world=int(os.environ.get('WORLD_SIZE','1'))
+    local_rank=int(os.environ.get('LOCAL_RANK','0'))
+    torch.set_num_threads(args.threads)
+    if args.require_two_t4:
+        names=[torch.cuda.get_device_name(i) for i in range(torch.cuda.device_count())]
+        assert args.device=='cuda' and len(names)==2 and world==2 and all('T4' in n for n in names), (names,world)
+    device=torch.device(f'cuda:{local_rank}' if args.device=='cuda' else 'cpu')
+    if device.type=='cuda': torch.cuda.set_device(device)
+    if world>1:
+        dist.init_process_group('nccl' if device.type=='cuda' else 'gloo',timeout=timedelta(minutes=30))
+    random.seed(SEED); np.random.seed(SEED); torch.manual_seed(SEED)
+    if device.type=='cuda': torch.cuda.manual_seed_all(SEED)
+    out=Path(args.out); out.mkdir(parents=True,exist_ok=True)
+    manifest_path=Path(args.data)/'corpus_manifest.json'
+    manifest=json.loads(manifest_path.read_text())
+    assert manifest['context']==CONTEXT and manifest['vocab_size']==VOCAB
+    if rank==0:
+        for split in manifest['splits'].values():
+            for item in split['files']:
+                p=Path(args.data)/item['name']
+                assert p.stat().st_size==item['bytes'] and sha(p)==item['sha256'], p
+        assert sha(Path(args.data)/'tokenizer'/'tokenizer.json')==manifest['tokenizer_sha256']
+    if world>1: dist.barrier()
+    datasets={s:PackedData(args.data,manifest,s) for s in ['train','validation','test']}
+    config=model_config(); config._attn_implementation='sdpa'
+    model=Qwen3ForCausalLM(config).to(device)
+    assert sum(p.numel() for p in model.parameters())==72_000_000
+    decay=[p for p in model.parameters() if p.ndim>=2]
+    no_decay=[p for p in model.parameters() if p.ndim<2]
+    optimizer=torch.optim.AdamW([{'params':decay,'weight_decay':0.1},
+                                {'params':no_decay,'weight_decay':0.0}],
+                               lr=args.lr,betas=(0.9,0.95),eps=1e-8,
+                               fused=device.type=='cuda')
+    amp=device.type=='cuda'
+    scaler=torch.amp.GradScaler('cuda',enabled=amp)
+    blocks_per_step=args.batch_size*args.grad_accum*world
+    tokens_per_step=blocks_per_step*CONTEXT
+    total_steps=math.ceil(args.target_tokens/tokens_per_step)
+    settings={'parameters':72_000_000,'context':CONTEXT,'vocab':VOCAB,
+              'seed':SEED,'world_size':world,'batch_size':args.batch_size,'grad_accum':args.grad_accum,
+              'blocks_per_step':blocks_per_step,'tokens_per_step':tokens_per_step,
+              'target_tokens':args.target_tokens,'total_steps':total_steps,
+              'warmup_steps':args.warmup_steps,'lr':args.lr,'min_lr':args.min_lr,
+              'optimizer':'AdamW betas=(0.9,0.95) eps=1e-8 decay=0.1 matrices only',
+              'precision':'FP16 autocast, FP32 weights/optimizer, GradScaler' if amp else 'FP32',
+              'corpus_manifest_sha256':sha(manifest_path),'tokenizer_sha256':manifest['tokenizer_sha256'],
+              'model_config':config.to_dict()}
+    # Exclude library-version metadata from equality; training configuration remains strict.
+    settings['model_config'].pop('transformers_version',None)
+    step=0; best=math.inf
+    checkpoint=None
+    if args.resume:
+        checkpoint=torch.load(args.resume,map_location='cpu',weights_only=False)
+        assert checkpoint['settings']==settings, 'Resume settings or dataset changed'
+        model.load_state_dict(checkpoint['model']); optimizer.load_state_dict(checkpoint['optimizer'])
+        scaler.load_state_dict(checkpoint['scaler']); step=checkpoint['step']; best=checkpoint['best_validation_loss']
+        # Verify deserialization before any further training; GPU arithmetic after
+        # restart can differ slightly without deterministic CUDA kernels.
+        assert all(torch.equal(v.detach().cpu(),checkpoint['model'][k]) for k,v in model.state_dict().items())
+        loaded_optimizer=optimizer.state_dict()
+        for key, state in loaded_optimizer['state'].items():
+            for field, value in state.items():
+                expected=checkpoint['optimizer']['state'][key][field]
+                if torch.is_tensor(value):assert torch.equal(value.cpu(),expected.cpu()), (key,field)
+                else:assert value==expected, (key,field)
+        if rank==0:print('RESUME_STATE_VERIFIED_EXACT',step,flush=True)
+        assert checkpoint['data_cursor_global_blocks']==step*blocks_per_step
+        assert checkpoint['scheduler']['next_step']==step
+    wrapped=DDP(model,device_ids=[local_rank],broadcast_buffers=False,gradient_as_bucket_view=True) if world>1 else model
+    if checkpoint:
+        restore_rng(checkpoint['rng_by_rank'][rank],device)
+        del checkpoint
+    started=time.monotonic(); last_save=started; stopped=[False]
+    def request_stop(*_): stopped[0]=True
+    signal.signal(signal.SIGTERM,request_stop); signal.signal(signal.SIGINT,request_stop)
+    if rank==0:
+        hardware={'gpu_names':[torch.cuda.get_device_name(i) for i in range(torch.cuda.device_count())],
+                  'world_size':world,'torch':torch.__version__,'cuda':torch.version.cuda,
+                  'parameters':72_000_000,'resumed_step':step}
+        dump(out/'hardware.json',hardware); dump(out/'training_settings.json',settings)
+        print('HARDWARE_VERIFIED',json.dumps(hardware),flush=True)
+        dump(out/'run_status.json',{'status':'RUNNING','step':step,'total_steps':total_steps,'started_utc':now()})
+    initial=evaluate(model,datasets['validation'],device,rank,world,min(4,args.eval_batches),args.batch_size,amp)
+    if rank==0:
+        print('INITIAL_VALIDATION',json.dumps(initial),flush=True)
+        dump(out/'initial_validation.json',initial)
+    model.train(); start_step=step; overflows=0; latest_eval=None
+    while step<total_steps:
+        iter_started=time.monotonic(); optimizer.zero_grad(set_to_none=True)
+        lr=lr_for(step,total_steps,args.warmup_steps,args.lr,args.min_lr)
+        for group in optimizer.param_groups: group['lr']=lr
+        losses=torch.zeros(1,device=device)
+        for micro in range(args.grad_accum):
+            ids=indices_for(step,micro,args.batch_size,args.grad_accum,world,rank,datasets['train'].blocks)
+            x,y=datasets['train'].batch(ids,device)
+            sync=wrapped.no_sync() if world>1 and micro<args.grad_accum-1 else nullcontext()
+            with sync:
+                with torch.autocast(device_type=device.type,dtype=torch.float16,enabled=amp):
+                    logits=wrapped(input_ids=x,use_cache=False).logits
+                    loss=F.cross_entropy(logits.reshape(-1,VOCAB).float(),y.reshape(-1),ignore_index=0)
+                finite=torch.isfinite(loss.detach()).to(torch.int32)
+                if world>1: dist.all_reduce(finite,op=dist.ReduceOp.MIN)
+                if not finite.item(): raise FloatingPointError(f'Non-finite loss at step {step}')
+                losses+=loss.detach()/args.grad_accum
+                scaler.scale(loss/args.grad_accum).backward()
+                del logits,loss
+        scaler.unscale_(optimizer)
+        grad_norm=torch.nn.utils.clip_grad_norm_(model.parameters(),1.0)
+        before_scale=scaler.get_scale()
+        scaler.step(optimizer); scaler.update()
+        if scaler.get_scale()<before_scale:
+            overflows+=1
+            if rank==0: print('AMP_RETRY',step,'scale',scaler.get_scale(),flush=True)
+            if overflows>32: raise FloatingPointError('Repeated FP16 overflows')
+            continue
+        overflows=0; step+=1
+        if world>1: dist.all_reduce(losses); losses/=world
+        if step==1 or step%10==0:
+            elapsed=time.monotonic()-started
+            record={'step':step,'total_steps':total_steps,'tokens_seen':step*tokens_per_step,
+                    'train_loss':losses.item(),'lr':lr,'grad_norm':float(grad_norm),
+                    'step_seconds':time.monotonic()-iter_started,
+                    'tokens_per_second':(step-start_step)*tokens_per_step/max(1,elapsed),
+                    'gpu_memory_peak_gb':torch.cuda.max_memory_allocated(device)/1e9 if amp else 0}
+            if rank==0:
+                print('TRAIN',json.dumps(record),flush=True)
+                with (out/'metrics.jsonl').open('a',encoding='utf-8') as f:f.write(json.dumps(record)+'\n')
+                dump(out/'run_status.json',{'status':'RUNNING',**record,'updated_utc':now()})
+        best_now=False
+        if step%args.eval_every==0 or step==total_steps:
+            latest_eval=evaluate(model,datasets['validation'],device,rank,world,args.eval_batches,args.batch_size,amp)
+            best_now=latest_eval['loss']<best
+            if best_now: best=latest_eval['loss']
+            if rank==0:
+                print('VALIDATION',step,json.dumps(latest_eval),flush=True)
+                dump(out/'validation_latest.json',{'step':step,**latest_eval})
+        stop=torch.tensor(int(stopped[0] or time.monotonic()-started>=args.max_hours*3600
+                              or (args.stop_after_steps and step>=args.stop_after_steps)),device=device)
+        save=torch.tensor(int(step==1 or step%args.save_every==0 or time.monotonic()-last_save>=args.save_seconds
+                              or best_now or step==total_steps),device=device)
+        if world>1:
+            dist.all_reduce(stop,op=dist.ReduceOp.MAX); dist.all_reduce(save,op=dist.ReduceOp.MAX)
+        if save.item() or stop.item():
+            save_checkpoint(out,model,optimizer,scaler,step,best,settings,rank,world,device,best_now)
+            last_save=time.monotonic()
+        if stop.item(): break
+    final_eval=evaluate(model,datasets['test'],device,rank,world,args.eval_batches,args.batch_size,amp) if step>=total_steps else None
+    if rank==0:
+        export=out/'model'; export.mkdir(exist_ok=True)
+        model.config.use_cache=True
+        model.save_pretrained(export,max_shard_size='2GB',safe_serialization=True)
+        tokenizer=AutoTokenizer.from_pretrained(Path(args.data)/'tokenizer',local_files_only=True)
+        tokenizer.model_input_names=['input_ids','attention_mask']
+        tokenizer.init_kwargs['model_input_names']=tokenizer.model_input_names
+        tokenizer.save_pretrained(export)
+        generations=[]
+        sample_error=None
+        try:
+            generations=generate_samples(model,tokenizer,device,amp)
+        except Exception as exc:
+            # Optional previews must not prevent final status/backup after a saved checkpoint.
+            sample_error={'type':type(exc).__name__,'message':str(exc)}
+            dump(out/'sample_generation_error.json',sample_error)
+            print('SAMPLE_GENERATION_FAILED',type(exc).__name__,flush=True)
+        dump(out/'sample_generations.json',generations)
+        for name in ['source_manifest.json','corpus_manifest.json']:
+            shutil.copyfile(Path(args.data)/name,out/name)
+        status='COMPLETE' if step>=total_steps else 'CHECKPOINTED_NEEDS_RESUME'
+        card=f'''---
+language: en
+library_name: transformers
+pipeline_tag: text-generation
+license: other
+---
+# Think72 Qwen3-style scratch checkpoint
+
+Status: {status}. Completed optimizer updates: {step}/{total_steps}.
+Processed token positions: {step*tokens_per_step:,}. Context: 1024. Parameters: 72,000,000.
+This checkpoint uses 14 Qwen3 decoder layers, hidden size 512, intermediate size 1536,
+8 query heads and 4 KV heads of dimension 128, Q/K RMSNorm, RoPE, SwiGLU and tied embeddings.
+Weights and the 33,069-token byte-level BPE were trained from scratch; no pretrained Qwen weights were loaded.
+
+Data: https://www.kaggle.com/datasets/faithdemo/think72-english-tokens-v1
+Training run: https://www.kaggle.com/code/faithdemo/think72-qwen3-t4x2
+Exact corpus manifest SHA-256: {settings['corpus_manifest_sha256']}.
+Exact tokenizer SHA-256: {settings['tokenizer_sha256']}.
+See the adjacent source/corpus manifests, training_settings.json and DATA_CARD.md for provenance,
+schema, splits, loss objective, learning-rate schedule and third-party attribution.
+Original source revisions are unknown; local file hashes identify the consumed snapshot.
+
+Load with AutoTokenizer.from_pretrained(path) and AutoModelForCausalLM.from_pretrained(path).
+Use tokenizer.apply_chat_template(messages, add_generation_prompt=True) and keep the total
+prompt plus generated tokens at or below 1024. Optional reasoning text uses <think> and </think>.
+Intended language is English; tasks include experimental text continuation, math, science and code.
+Validation: {json.dumps(latest_eval)}. Held-out test sample: {json.dumps(final_eval)}.
+Perplexity is not a task-accuracy score; inspect sample_generations.json and evaluate independently.
+
+Limitations: tiny model capacity, short context, mostly reasoning/synthetic rather than broad
+language pretraining, possible overlap between semantically related questions, factual errors,
+repetition and unreliable reasoning. Qwen3 architecture does not confer Qwen3-0.6B capabilities.
+
+Copyright (c) 2026 sraivante covers original contributions and selection/arrangement only,
+under Apache License 2.0. Third-party source terms remain in force, including CC-BY-NC-4.0
+material; no blanket Apache or commercial-use license for these weights/data is asserted.
+'''
+        (export/'README.md').write_text(card,encoding='utf8')
+        if Path('DATA_CARD.md').exists():shutil.copy2('DATA_CARD.md',out/'DATA_CARD.md')
+        result={'status':status,
+                'step':step,'total_steps':total_steps,'tokens_seen':step*tokens_per_step,
+                'validation':latest_eval,'heldout_test':final_eval,'ended_utc':now(),
+                'sample_generation_error':sample_error,
+                'model_files':{p.name:{'bytes':p.stat().st_size,'sha256':sha(p)} for p in export.iterdir() if p.is_file()},
+                'limitations':'A scratch-trained 72M model with a 1024-token window; Qwen3 architecture does not imply Qwen3-0.6B reasoning ability. Task accuracy requires separate evaluation.'}
+        dump(out/'run_status.json',result)
+        print('TRAINING_SESSION_FINISHED',json.dumps(result),flush=True)
+    if world>1: dist.barrier(); dist.destroy_process_group()
+
+if __name__=='__main__': main()

code/verify_original_workspace_hellaswag.py

@@ -0,0 +1,137 @@
+"""Independently verify downloaded HellaSwag samples without rerunning inference."""
+import argparse
+import hashlib
+import json
+import math
+import re
+import zipfile
+from collections import Counter
+from datetime import datetime, timezone
+from pathlib import Path
+
+import pyarrow.parquet as pq
+
+ROOT = Path(__file__).resolve().parents[1]
+MODEL_SHA = "0f7bdaaae88077174d371c650759b71b83fce6e917638703b72466d59a2b3c9e"
+TOKENIZER_SHA = "27cb97f527f7eee80c7de32e04ba9cc32f8b8c2117d8664690443d53e228433b"
+DATA_SHA = "899813071e1e95efafec90f856e1987d2150fa4d020fc005df6962c259f660cd"
+WHEEL_SHA = "5daaa1973bf874005f64f28d3834b875f6886f0d6475878e6a6c821994a5286a"
+
+
+def sha(path):
+    with Path(path).open("rb") as handle:
+        return hashlib.file_digest(handle, "sha256").hexdigest()
+
+
+def require(condition, message):
+    if not condition:
+        raise ValueError(message)
+
+
+def preprocess(text):
+    # Independently reproduced from the pinned official HellaSwag task utility.
+    text = text.strip().replace(" [title]", ". ")
+    return re.sub(r"\[.*?\]", "", text).replace("  ", " ")
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--run", type=Path, default=ROOT / "downloads/evaluations/hellaswag-v1")
+    args = parser.parse_args()
+    run = args.run.resolve()
+    data = run / "hellaswag"
+    summary = json.loads((data / "summary.json").read_text(encoding="utf-8"))
+    native = json.loads((data / "results.json").read_text(encoding="utf-8"))
+    source = json.loads((run / "download_source.json").read_text(encoding="utf-8"))
+    model = ROOT / "downloads/run-v7/think72/run/model"
+    require(sha(model / "model.safetensors") == summary["model_sha256"] == MODEL_SHA,
+            "Final local model identity mismatch")
+    require(sha(model / "tokenizer.json") == summary["tokenizer_sha256"] == TOKENIZER_SHA,
+            "Tokenizer identity mismatch")
+    require(sha(data / "validation.parquet") == summary["dataset_file_sha256"] == DATA_SHA,
+            "Validation dataset identity mismatch")
+    require(sha(data / "hellaswag_worker.py") == sha(ROOT / "evaluation/hellaswag_worker.py"),
+            "Downloaded worker differs from submitted source")
+    wheel = ROOT / "artifacts/hellaswag/lm_eval-0.4.13-py3-none-any.whl"
+    require(sha(wheel) == WHEEL_SHA, "Official harness wheel identity mismatch")
+    with zipfile.ZipFile(wheel) as archive:
+        task_sha = hashlib.sha256(archive.read("lm_eval/tasks/hellaswag/hellaswag.yaml")).hexdigest()
+    require(task_sha == summary["task_yaml_sha256"], "Official task YAML mismatch")
+    require(summary["model_step"] == 22889 and summary["examples"] == 10042,
+            "Wrong checkpoint or incomplete evaluation")
+    require(summary["num_fewshot"] == 0 and summary["apply_chat_template"] is False
+            and summary["add_bos_token"] is False and summary["dtype"] == "float32",
+            "Unexpected evaluation protocol")
+
+    rows = pq.read_table(data / "validation.parquet").to_pylist()
+    require(len(rows) == 10042, "Incorrect validation split length")
+    seen = set()
+    correct = Counter(acc=0, acc_norm=0)
+    labels = Counter()
+    for line in (data / "samples.jsonl").read_text(encoding="utf-8").splitlines():
+        sample = json.loads(line)
+        index = sample["doc_id"]
+        require(isinstance(index, int) and 0 <= index < len(rows) and index not in seen,
+                f"Invalid or duplicate doc_id: {index}")
+        seen.add(index)
+        row, doc = rows[index], sample["doc"]
+        require(all(doc[key] == value for key, value in row.items()),
+                f"Source row mismatch: {index}")
+        gold = int(row["label"])
+        choices = [preprocess(ending) for ending in row["endings"]]
+        query = preprocess(row["activity_label"] + ": " + row["ctx_a"] + " " + row["ctx_b"].capitalize())
+        require(len(choices) == 4 and all(choices), f"Invalid choices: {index}")
+        require(doc["choices"] == choices and doc["query"] == query and doc["gold"] == gold
+                and int(sample["target"]) == gold, f"Preprocessing or label mismatch: {index}")
+        require(sample["arguments"] == [[query, " " + choice] for choice in choices],
+                f"Prompt or continuation mismatch: {index}")
+        responses = sample["filtered_resps"]
+        require(len(responses) == 4 and all(len(response) == 2 for response in responses),
+                f"Missing choice scores: {index}")
+        require(sample["resps"] == [[response] for response in responses],
+                f"Scores modified by filtering: {index}")
+        scores = [response[0] for response in responses]
+        require(all(math.isfinite(score) for score in scores), f"Nonfinite score: {index}")
+        raw_pred = max(range(4), key=lambda choice: scores[choice])
+        normalized_pred = max(range(4), key=lambda choice: scores[choice] / len(choices[choice]))
+        for metric, prediction in (("acc", raw_pred), ("acc_norm", normalized_pred)):
+            value = int(prediction == gold)
+            require(sample[metric] == value, f"Per-question {metric} mismatch: {index}")
+            correct[metric] += value
+        labels[str(gold)] += 1
+    require(seen == set(range(10042)), "Missing validation questions")
+    metrics = native["results"]["hellaswag"]
+    for metric in ("acc", "acc_norm"):
+        require(correct[metric] == summary[metric + "_correct"], f"Wrong {metric} count")
+        require(abs(correct[metric] / len(rows) - metrics[metric + ",none"]) < 1e-12,
+                f"Aggregate {metric} differs from independent recomputation")
+        require(metrics[metric + ",none"] == summary["metrics"][metric + ",none"],
+                f"Summary {metric} differs from harness result")
+    require(dict(labels) == summary["label_counts"], "Label counts mismatch")
+    audits = [json.loads((data / f"rank-{rank}-audit.json").read_text()) for rank in range(2)]
+    for rank, audit in enumerate(audits):
+        require(audit["rank"] == rank and audit["world_size"] == 2
+                and audit["choice_requests"] == 20084 and audit["truncated_requests"] == 0
+                and audit["max_pair_tokens"] <= 1024 and audit["device"] == f"cuda:{rank}",
+                f"Incomplete GPU rank audit: {rank}")
+    require(summary["gpu_names"] == ["Tesla T4", "Tesla T4"], "Wrong evaluation hardware")
+    verification = {
+        "status": "VERIFIED", "verified_utc": datetime.now(timezone.utc).isoformat(),
+        "source": source, "model_step": 22889, "model_sha256": MODEL_SHA,
+        "dataset_sha256": DATA_SHA, "validation_rows": len(rows), "unique_scored_rows": len(seen),
+        "source_rows_labels_prompts_and_all_choice_scores_verified": True,
+        "independently_recomputed_correct": dict(correct),
+        "independently_recomputed_accuracy": {key: value / len(rows) for key, value in correct.items()},
+        "random_choice_baseline": 0.25, "rank_audits": audits,
+        "files": {path.name: {"bytes": path.stat().st_size, "sha256": sha(path)}
+                  for path in sorted(data.iterdir()) if path.is_file()},
+        "limitations": "Recomputes metrics from recorded inference scores, not an independent second model inference. Training-data contamination has not been exhaustively audited.",
+    }
+    (run / "verification.json").write_text(json.dumps(verification, indent=2) + "\n", encoding="utf-8")
+    print(json.dumps({key: verification[key] for key in (
+        "status", "validation_rows", "unique_scored_rows", "independently_recomputed_correct",
+        "independently_recomputed_accuracy")}, indent=2))
+
+
+if __name__ == "__main__":
+    main()

config.json

@@ -0,0 +1,47 @@
+{
+  "architectures": [
+    "Qwen3ForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 1,
+  "dtype": "float32",
+  "eos_token_id": 2,
+  "head_dim": 128,
+  "hidden_act": "silu",
+  "hidden_size": 512,
+  "initializer_range": 0.02,
+  "intermediate_size": 1536,
+  "layer_types": [
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention"
+  ],
+  "max_position_embeddings": 1024,
+  "max_window_layers": 14,
+  "model_type": "qwen3",
+  "num_attention_heads": 8,
+  "num_hidden_layers": 14,
+  "num_key_value_heads": 4,
+  "pad_token_id": 0,
+  "rms_norm_eps": 1e-06,
+  "rope_scaling": null,
+  "rope_theta": 1000000.0,
+  "sliding_window": null,
+  "tie_word_embeddings": true,
+  "transformers_version": "4.56.2",
+  "use_cache": true,
+  "use_sliding_window": false,
+  "vocab_size": 33069
+}

evaluation/hellaswag/environment.txt

@@ -0,0 +1,932 @@
+a2a-sdk==1.2.1
+absl-py==1.4.0
+accelerate==1.10.1
+access==1.1.10.post3
+affine==3.0.1
+aiobotocore==2.26.0
+aiofiles==25.1.0
+aiohappyeyeballs==2.7.1
+aiohttp==3.14.3
+aioitertools==0.13.0
+aiosignal==1.4.0
+aiosqlite==0.22.1
+alabaster==1.0.0
+albucore==0.0.24
+albumentations==2.0.8
+ale-py==0.12.1
+alembic==1.20.0
+altair==5.5.0
+annotated-doc==0.0.5
+annotated-types==0.8.0
+antlr4-python3-runtime==4.9.3
+anyio==4.14.2
+anywidget==0.9.21
+apsw==3.53.4.0
+argon2-cffi==25.1.0
+argon2-cffi-bindings==26.1.0
+args==0.1.0
+array_record==0.8.3
+arrow==1.4.0
+arviz==0.22.0
+asn1crypto==1.5.1
+astropy==7.2.2
+astropy-iers-data==0.2026.8.31.0.57.9
+asttokens==3.0.2
+astunparse==1.6.3
+async-lru==2.3.0
+atpublic==5.1
+attrs==26.1.0
+audioop-lts==0.2.2
+audioread==3.1.0
+Authlib==1.8.0
+autograd==1.9.1
+babel==2.18.0
+backcall==0.2.0
+bayesian-optimization==3.3.0
+beartype==0.22.9
+beautifulsoup4==4.13.5
+betterproto==2.0.0b7
+bigframes==2.48.0
+bigquery-magics==0.14.0
+black==26.5.1
+blake3==1.0.10
+bleach==6.4.0
+blinker==1.9.0
+blis==1.3.3
+blobfile==3.3.0
+blosc2==4.12.0
+bokeh==3.8.2
+Boruta==0.4.3
+boto3==1.41.5
+botocore==1.41.5
+Bottleneck==1.4.2
+bqplot==0.12.47
+branca==0.8.2
+brotli==1.2.0
+build==1.6.1
+cachebox==5.2.3
+CacheControl==0.14.4
+cachetools==6.2.6
+Cartopy==0.26.0
+catalogue==2.0.10
+catboost==1.2.10
+category_encoders==2.11.1
+certifi==2026.7.22
+cesium==0.12.4
+cffi==2.1.1
+chardet==5.2.0
+charset-normalizer==3.4.9
+chex==0.1.90
+clarabel==0.11.1
+click==8.5.0
+click-plugins==1.1.1.2
+cligj==0.7.2
+clint==0.5.1
+cloudpathlib==0.25.0
+cloudpickle==3.1.2
+cmake==3.31.10
+cmdstanpy==1.3.0
+colorama==0.4.6
+colorcet==3.2.1
+colorlog==6.12.0
+colorlover==0.3.0
+comm==0.2.3
+community==1.0.0b1
+confection==1.3.3
+ConfigSpace==1.2.2
+cons==0.4.7
+contourpy==1.3.3
+coverage==7.16.2
+cramjam==2.12.1
+cryptography==50.0.1
+cucim-cu12 @ https://pypi.nvidia.com/cucim-cu12/cucim_cu12-26.2.1-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl
+cuda-bindings==12.9.7
+cuda-core==0.3.2
+cuda-pathfinder==1.8.0
+cuda-python==12.9.7
+cuda-toolkit==12.8.1
+cudf-cu12==26.2.1
+cudf-polars-cu12==26.2.1
+cufflinks==0.17.3
+cuml-cu12==26.2.0
+cupy-cuda12x==14.0.1
+curl_cffi==0.16.2
+cuvs-cu12 @ https://pypi.nvidia.com/cuvs-cu12/cuvs_cu12-26.2.0-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl
+cvxopt==1.3.2
+cvxpy==1.6.7
+cycler==0.12.1
+cyipopt==1.5.0
+cymem==2.0.13
+Cython==3.0.12
+cytoolz==1.1.0
+dacite==1.9.2
+dask==2026.1.1
+dask-cuda==26.2.0
+dask-cudf-cu12==26.2.1
+dask-expr==2.0.0
+dask-jobqueue==0.9.0
+dataclasses-json==0.6.7
+dataproc-spark-connect==1.1.0
+DataProperty==1.1.1
+datasets==4.1.1
+db-dtypes==1.7.1
+dbus-python==1.3.2
+deap==1.4.4
+debugpy==1.8.15
+decorator==5.3.1
+deepdiff==9.1.0
+defusedxml==0.7.1
+Deprecated==3.0.0
+deprecation==2.1.0
+diffusers==0.40.0
+dill==0.4.0
+dipy==1.12.1
+distributed==2026.1.1
+distributed-ucxx-cu12==0.48.0
+distro==1.9.0
+dlib==19.24.6
+dm-tree==0.1.10
+dnspython==2.8.0
+docker==7.2.0
+docstring-to-markdown==0.17
+docstring_parser==0.18.0
+docutils==0.21.2
+dopamine_rl==4.1.2
+duckdb==1.3.2
+earthengine-api==1.7.39
+easydict==1.13
+easyocr==1.7.2
+editdistance==0.8.1
+eerepr==0.1.2
+einops==0.8.2
+email-validator==2.3.0
+emoji==2.16.0
+en_core_web_sm @ https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.8.0/en_core_web_sm-3.8.0-py3-none-any.whl#sha256=1932429db727d4bff3deed6b34cfc05df17794f4a52eeb26cf8928f7c1a0fb85
+entrypoints==0.4
+esda==2.9.0
+et_xmlfile==2.0.0
+etils==1.14.0
+etuples==0.3.10
+evaluate==0.4.6
+execnb==0.3.3
+Farama-Notifications==0.0.6
+fastai==2.8.8
+fastapi==0.141.1
+fastcore==2.2.32
+fastdownload==0.0.7
+fastgit==0.1.4
+fastjsonschema==2.22.2
+fastprogress==1.1.6
+fastspec==0.2.5
+fasttext==0.9.3
+fasttransform==0.0.2
+fasttransport==0.0.4
+fastuuid==0.14.0
+featuretools==1.31.0
+filelock==3.32.5
+filetype==1.2.0
+fiona==1.10.1
+firebase-admin==6.9.0
+Flask==3.1.3
+flatbuffers==25.12.19
+flax==0.12.10
+folium==0.20.0
+fonttools==4.64.0
+formulaic==1.2.2
+fqdn==1.5.1
+frozendict==2.4.7
+frozenlist==1.8.0
+fsspec==2025.9.0
+func_timeout==4.3.5
+funcy==2.1
+fury==0.11.0
+future==1.0.0
+fuzzywuzzy==0.18.0
+gast==0.7.0
+gatspy==0.3
+gcsfs==2025.12.0
+GDAL==3.8.4
+gdown==5.2.2
+geemap==0.38.4
+gensim==4.4.0
+geocoder==1.38.1
+geographiclib==2.1
+geojson==3.3.0
+geopandas==1.1.4
+geopy==2.4.1
+gepa==0.1.4
+ghapi==2.1.4
+giddy==2.3.9
+gin-config==0.5.0
+glob2==0.7
+google==3.0.0
+google-adk==2.7.1
+google-ai-generativelanguage==0.6.15
+google-api-core==2.30.3
+google-api-python-client==2.200.0
+google-auth==2.49.0
+google-auth-httplib2==0.4.2
+google-auth-oauthlib==1.4.1
+google-cloud-aiplatform==2.3.0
+google-cloud-bigquery==3.44.0
+google-cloud-bigquery-connection==1.22.0
+google-cloud-core==2.7.0
+google-cloud-dataproc==5.28.0
+google-cloud-datastore==2.25.0
+google-cloud-firestore==2.27.0
+google-cloud-functions==1.24.0
+google-cloud-language==2.21.0
+google-cloud-monitoring==2.31.0
+google-cloud-resource-manager==1.18.0
+google-cloud-spanner==3.68.0
+google-cloud-storage==3.13.1
+google-cloud-storage-control==1.12.0
+google-cloud-texttospeech==2.37.0
+google-cloud-translate==3.27.0
+google-cloud-videointelligence==2.20.0
+google-cloud-vision==3.15.0
+google-colab @ file:///colabtools/dist/google_colab-1.0.0.tar.gz
+google-crc32c==1.8.0
+google-genai==2.12.1
+google-generativeai==0.8.6
+google-pasta==0.2.0
+google-resumable-media==2.10.2
+googleapis-common-protos==1.75.0
+googledrivedownloader==1.1.0
+gpxpy==1.6.2
+gradio==6.26.0
+gradio_client==2.6.1
+grain==0.2.18
+graphviz==0.21
+greenlet==3.5.5
+groovy==0.1.2
+grpc-google-iam-v1==0.14.4
+grpc-interceptor==0.15.4
+grpcio==1.84.0
+grpcio-status==1.71.2
+grpcio-tools==1.71.2
+grpclib==0.4.9
+gspread==6.2.1
+gspread-dataframe==4.0.0
+gym==0.26.2
+gym-notices==0.1.0
+gymnasium==1.2.0
+gymnax==0.0.8
+h11==0.16.0
+h2==4.4.1
+h2o==3.46.0.12
+h5netcdf==1.8.1
+h5py==3.16.0
+haversine==2.9.0
+hdbscan==0.8.44
+hep_ml==0.8.0
+hf-gradio==0.4.1
+hf-xet==1.6.0
+highspy==1.15.1
+holidays==0.103
+holoviews==1.23.2
+hpack==4.2.0
+html5lib==1.1
+httpcore==1.0.9
+httpcore2==2.12.0
+httpimport==1.4.1
+httplib2==0.32.0
+httptools==0.8.0
+httpx==0.28.1
+httpx2==2.12.0
+huggingface-hub==0.35.3
+humanize==4.16.0
+hyperframe==6.1.0
+hyperopt==0.3.0
+ibis-framework==9.5.0
+id==1.6.1
+idna==3.19
+igraph==1.0.0
+ImageHash==4.3.2
+ImageIO==2.37.4
+imageio-ffmpeg==0.6.0
+imagesize==2.0.1
+imbalanced-learn==0.14.2
+immutabledict==4.3.1
+importlib_metadata==8.9.0
+importlib_resources==7.1.0
+imutils==0.5.4
+in-toto-attestation==0.9.3
+inequality==1.1.2
+inflect==7.5.0
+iniconfig==2.3.0
+intel-cmplr-lib-ur==2025.3.3
+intel-openmp==2025.3.3
+interface_meta==2.0.1
+ipyevents==2.0.4
+ipyfilechooser==0.6.0
+ipykernel==6.17.1
+ipyleaflet==0.20.0
+ipympl==0.10.0
+ipyparallel==8.8.0
+ipython==7.34.0
+ipython-genutils==0.2.0
+ipython-sql==0.5.0
+ipython_pygments_lexers==1.1.1
+ipywidgets==8.1.5
+isoduration==20.11.0
+isoweek==1.3.3
+itsdangerous==2.2.0
+Janome==0.5.0
+jaraco.classes==3.4.0
+jaraco.context==6.1.2
+jaraco.functools==4.6.0
+jax==0.11.1
+jax-cuda12-pjrt==0.11.1
+jax-cuda12-plugin==0.11.1
+jaxlib==0.11.1
+jedi==0.20.0
+jeepney==0.9.0
+jieba==0.42.1
+Jinja2==3.1.6
+jiter==0.10.0
+jmespath==1.1.0
+joblib==1.6.0
+joserfc==1.7.5
+json-rpc==1.15.0
+json5==0.15.0
+jsonpatch==1.33
+jsonpickle==4.1.2
+jsonpointer==3.1.1
+jsonschema==4.26.0
+jsonschema-specifications==2025.9.1
+jupyter-console==6.6.3
+jupyter-events==0.12.1
+jupyter-leaflet==0.20.0
+jupyter-lsp==2.3.1
+jupyter_client==7.4.9
+jupyter_core==5.9.1
+jupyter_kernel_gateway @ git+https://github.com/googlecolab/kernel_gateway@b134e9945df25c2dcb98ade9129399be10788671
+jupyter_server==2.12.5
+jupyter_server_proxy==4.6.0
+jupyter_server_terminals==0.5.4
+jupyterlab==4.5.11
+jupyterlab-lsp==5.3.0
+jupyterlab_pygments==0.3.0
+jupyterlab_server==2.28.1
+jupyterlab_widgets==3.0.17
+jupytext==1.19.5
+kaggle==2.0.2
+kaggle-environments==1.27.3
+kagglehub==1.0.2
+kagglesdk==0.1.37
+keras==3.13.2
+keras-core==0.1.7
+keras-cv==0.9.0
+keras-hub==0.26.0
+keras-nlp==0.26.0
+keras-tuner==1.4.8
+keyring==25.7.0
+keyrings.google-artifactregistry-auth==1.1.2
+kiwisolver==1.5.1
+kornia==0.8.3
+kornia_rs==0.2.0
+kt-legacy==1.0.5
+langchain==1.3.18
+langchain-core==1.6.1
+langchain-protocol==0.0.19
+langgraph==1.2.11
+langgraph-checkpoint==4.2.0
+langgraph-prebuilt==1.1.0
+langgraph-sdk==0.4.4
+langid==1.1.6
+langsmith==0.12.1
+lark==1.3.1
+launchpadlib==1.11.0
+lazr.restfulclient==0.14.6
+lazr.uri==1.0.6
+lazy-loader==0.5
+learntools @ git+https://github.com/Kaggle/learntools@e9ba66dc899d3e17b80d3a1b9a5d29f430d310cb
+libclang==18.1.1
+libcudf-cu12==26.2.1
+libcugraph-cu12==26.2.0
+libcuml-cu12==26.2.0
+libcuvs-cu12==26.2.0
+libkvikio-cu12==26.2.0
+libpysal==4.14.1
+libraft-cu12==26.2.0
+librmm-cu12==26.2.0
+librosa==0.11.0
+libucx-cu12==1.19.0
+libucxx-cu12==0.48.0
+lightgbm==4.6.0
+lightning-utilities==0.15.3
+lime==0.2.0.1
+line_profiler==5.0.2
+linkify-it-py==2.2.0
+litellm==1.85.7
+llvmlite==0.44.0
+lm_eval @ file:///tmp/lm_eval-0.4.13-py3-none-any.whl#sha256=5daaa1973bf874005f64f28d3834b875f6886f0d6475878e6a6c821994a5286a
+lml==0.2.0
+locket==1.0.0
+logical-unification==0.4.7
+lxml==6.1.2
+Mako==1.4.3
+mamba==0.11.3
+mapclassify==2.10.0
+Markdown==3.10.3
+markdown-it-py==4.2.0
+MarkupSafe==3.0.3
+marshmallow==3.26.2
+matplotlib==3.10.0
+matplotlib-inline==0.2.2
+matplotlib-venn==1.1.2
+mbstrdecoder==1.1.5
+mdit-py-plugins==0.6.1
+mdurl==0.1.2
+mgwr==2.2.1
+minify_html==0.18.1
+miniKanren==1.0.5
+missingno==0.5.2
+mistune==0.8.4
+mizani==0.13.5
+mkl==2025.3.1
+ml_collections==1.1.0
+ml_dtypes==0.6.0
+mlcrate==0.2.0
+mlxtend==0.23.4
+mmh3==5.3.0
+mne==1.13.2
+model-signing==1.1.1
+momepy==0.11.0
+moocore==0.3.2
+more-itertools==10.8.0
+moviepy==1.0.3
+mpld3==0.5.12
+mpmath==1.3.0
+msgpack==1.2.2
+multidict==6.7.1
+multimethod==1.12
+multipledispatch==1.0.0
+multiprocess==0.70.16
+multitasking==0.0.13
+murmurhash==1.0.15
+music21==9.9.2
+mypy_extensions==1.1.0
+namex==0.1.0
+narwhals==2.25.0
+natsort==8.4.0
+nbclassic==1.3.3
+nbclient==0.5.13
+nbconvert==6.4.5
+nbdev==3.3.24
+nbformat==5.11.1
+ndindex==1.10.1
+nest-asyncio==1.6.0
+networkx==3.6.1
+nh3==0.3.7
+nibabel==5.4.2
+nilearn==0.14.1
+ninja==1.13.2
+nltk==3.9.1
+notebook==6.5.7
+notebook_shim==0.2.4
+numba==0.61.2
+numba-cuda==0.22.2
+numexpr==2.14.2
+numpy==2.1.3
+nvidia-cublas-cu12==12.8.4.1
+nvidia-cuda-cccl-cu12==12.9.27
+nvidia-cuda-cupti-cu12==12.8.90
+nvidia-cuda-nvcc-cu12==12.8.93
+nvidia-cuda-nvrtc-cu12==12.8.93
+nvidia-cuda-runtime-cu12==12.8.90
+nvidia-cudnn-cu12==9.19.0.56
+nvidia-cufft-cu12==11.3.3.83
+nvidia-cufile-cu12==1.13.1.3
+nvidia-curand-cu12==10.3.9.90
+nvidia-cusolver-cu12==11.7.3.90
+nvidia-cusparse-cu12==12.5.8.93
+nvidia-cusparselt-cu12==0.7.1
+nvidia-libnvcomp-cu12==5.1.0.21
+nvidia-ml-py==13.610.43
+nvidia-nccl-cu12==2.28.9
+nvidia-nccl-cu13==2.31.2
+nvidia-nvimgcodec-cu12==0.7.0.11
+nvidia-nvjitlink-cu12==12.8.93
+nvidia-nvshmem-cu12==3.4.5
+nvidia-nvtx-cu12==12.8.90
+nvtx==0.2.16
+nx-cugraph-cu12 @ https://pypi.nvidia.com/nx-cugraph-cu12/nx_cugraph_cu12-26.2.0-py3-none-any.whl
+oauth2client==4.1.3
+oauthlib==3.3.1
+odfpy==1.4.1
+olefile==0.47
+omegaconf==2.3.1
+onemkl-license==2025.3.1
+onnx==1.22.0
+open-spiel==2.0.2
+openai==2.54.0
+opencv-contrib-python==4.14.0.94
+opencv-python==4.14.0.94
+opencv-python-headless==4.14.0.94
+openpyxl==3.1.5
+openslide-bin==4.0.1.2
+openslide-python==1.4.6
+opentelemetry-api==1.42.1
+opentelemetry-resourcedetector-gcp==1.14.0
+opentelemetry-sdk==1.42.1
+opentelemetry-semantic-conventions==0.63b1
+opt_einsum==3.4.0
+optax==0.2.8
+optree==0.20.0
+optuna==5.0.0
+orbax-checkpoint==0.12.4
+orderly-set==5.5.0
+orjson==3.12.0
+ormsgpack==1.12.2
+osqp==1.1.3
+overrides==7.7.0
+packaging==26.3
+pandas==2.3.3
+pandas-datareader==0.11.1
+pandas-gbq==0.30.0
+pandas-profiling==3.6.6
+pandas-stubs==2.2.3.250527
+pandasql==0.7.3
+pandocfilters==1.5.1
+panel==1.9.4
+panel-material-ui==0.14.2
+papermill==2.7.0
+param==2.4.1
+parso==0.8.7
+parsy==2.2
+partd==1.4.2
+path==17.1.1
+path.py==12.5.0
+pathlib_abc==0.5.2
+pathos==0.3.5
+pathspec==1.1.1
+pathvalidate==3.3.1
+patsy==1.0.3
+pdf2image==1.17.0
+peewee==4.4.0
+peft==0.17.1
+pettingzoo==1.24.0
+pexpect==4.9.0
+phik==0.12.5
+pickleshare==0.7.5
+pillow==12.3.0
+platformdirs==4.11.7
+plotly==5.24.1
+plotly-express==0.4.1
+plotnine==0.14.5
+pluggy==1.6.0
+plum-dispatch==2.9.0
+pointpats==2.5.5
+pokerkit==0.6.3
+polars==1.35.2
+polars-runtime-32==1.35.2
+pooch==1.9.0
+portalocker==4.4.1
+portpicker==1.5.2
+POT==0.9.7.post1
+pox==0.3.7
+ppft==1.7.8
+preshed==3.0.13
+prettytable==3.18.0
+proglog==0.1.12
+progressbar2==4.5.0
+prometheus_client==0.26.0
+promise==2.3
+prompt_toolkit==3.0.53
+propcache==0.5.2
+prophet==1.4.0
+proto-plus==1.28.2
+protobuf==5.29.5
+psutil==5.9.5
+psycopg2==2.9.12
+psygnal==0.15.1
+ptyprocess==0.7.0
+pudb==2025.1.5
+PuLP==3.3.2
+puremagic==2.2.0
+py-cpuinfo==9.0.0
+py4j==0.10.9.9
+pyaml==26.7.0
+PyArabic==0.6.15
+pyarrow==23.0.1
+pyasn1==0.6.4
+pyasn1_modules==0.4.2
+pybind11==3.1.0
+pycairo==1.29.1
+pyclipper==1.4.0
+pycocotools==2.0.11
+pycparser==3.0
+pycryptodome==3.23.0
+pycryptodomex==3.23.0
+pycuda==2026.1
+pydantic==2.13.5
+pydantic_core==2.46.5
+pydata-google-auth==1.9.1
+pydicom==3.0.2
+pydot==4.0.1
+pydotplus==2.0.2
+PyDrive2==1.21.1
+pydub==0.25.1
+pyemd==2.0.0
+pyerfa==2.0.1.5
+pyexcel-io==0.6.8
+pyexcel-ods==0.6.0
+pygame==2.6.1
+pygit2==1.20.0
+pygltflib==1.16.5
+Pygments==2.21.0
+PyGObject==3.48.2
+pyiceberg==0.12.0
+pyjson5==2.0.1
+PyJWT==2.13.0
+pyLDAvis==3.4.1
+pylibcudf-cu12==26.2.1
+pylibcugraph-cu12==26.2.0
+pylibraft-cu12==26.2.0
+pymc==5.28.5
+pymongo==4.18.2
+Pympler==1.1
+pynndescent==0.6.0
+pyogrio==0.13.0
+pyomo==6.10.1
+PyOpenGL==3.1.10
+pyOpenSSL==26.4.0
+pyparsing==3.3.2
+pypdf==6.19.0
+pyperclip==1.11.0
+pyproj==3.7.2
+pyproject_hooks==1.3.3
+pyroaring==1.1.0
+pysal==25.7
+pyshp==3.1.6
+PySocks==1.7.1
+pyspark==4.0.4
+pytablewriter==1.2.1
+pytensor==2.38.3
+pytesseract==0.3.13
+pytest==8.4.2
+python-apt==0.0.0
+python-bidi==0.6.11
+python-box==7.4.1
+python-dateutil==2.9.0.post0
+python-dotenv==1.2.3
+python-fasthtml==0.14.12
+python-json-logger==4.2.0
+python-louvain==0.16
+python-lsp-jsonrpc==1.1.2
+python-lsp-server==1.15.0
+python-multipart==0.0.32
+python-slugify==8.0.4
+python-snappy==0.7.3
+python-utils==4.0.1
+pytokens==0.4.1
+pytools==2026.1.1
+pytorch-ignite==0.5.5
+pytorch-lightning==2.6.6
+pytz==2025.2
+PyUpSet==0.1.1.post7
+pyviz_comms==3.0.6
+PyWavelets==1.9.0
+PyYAML==6.0.3
+pyzmq==26.2.1
+qgrid==1.3.1
+qtconsole==5.7.2
+QtPy==2.4.3
+quantecon==0.11.4
+raft-dask-cu12==26.2.0
+rapids-dask-dependency==26.2.0
+rapids-logger==0.2.3
+rasterio==1.5.1
+rasterstats==0.21.0
+ratelim==0.1.6
+ray==2.58.0
+referencing==0.37.0
+regex==2025.11.3
+requests==2.34.2
+requests-oauthlib==2.0.0
+requests-toolbelt==1.0.0
+requirements-parser==0.9.0
+rfc3161-client==1.0.9
+rfc3339-validator==0.1.4
+rfc3986-validator==0.1.1
+rfc3987-syntax==1.1.0
+rfc8785==0.1.4
+rgf-python==3.12.0
+rich==13.9.4
+rmm-cu12==26.2.0
+roman-numerals==4.1.0
+roman-numerals-py==4.1.0
+rouge_score==0.1.2
+rpds-py==2026.6.3
+rpy2==3.5.17
+rsa==4.9.1
+rtree==1.4.1
+ruamel.yaml==0.19.1
+s3fs==2025.12.0
+s3transfer==0.15.0
+sacrebleu==2.6.0
+safehttpx==0.1.7
+safetensors==0.8.0
+scikit-image==0.25.2
+scikit-learn==1.6.1
+scikit-multilearn==0.2.0
+scikit-optimize==0.10.2
+scikit-plot==0.3.7
+scikit-surprise==1.1.5
+scipy==1.16.3
+scooby==0.11.2
+scs==3.2.11
+seaborn==0.13.2
+SecretStorage==3.5.0
+securesystemslib==1.5.1
+segment_anything @ git+https://github.com/facebookresearch/segment-anything.git@dca509fe793f601edb92606367a655c15ac00fdf
+segregation==2.5.4
+semantic-version==2.10.0
+Send2Trash==2.1.0
+sentence-transformers==5.7.0
+sentencepiece==0.2.2
+sentry-sdk==2.68.1
+setuptools==81.0.0
+shap==0.52.0
+shapely==2.1.2
+shellingham==1.5.4
+Shimmy==2.0.1
+sigstore==4.5.0
+sigstore-models==0.0.6
+sigstore-rekor-types==0.0.18
+simpervisor==1.0.0
+simple-parsing==0.1.9
+simpleitk==2.5.6
+simplejson==4.1.2
+simsimd==6.5.16
+siphash24==1.9
+six==1.17.0
+sklearn-compat==0.1.6
+sklearn-pandas==2.2.0
+slicer==0.0.8
+smart_open==8.0.1
+sniffio==1.3.1
+snowballstemmer==3.1.1
+sortedcontainers==2.4.0
+soundfile==0.14.0
+soupsieve==2.9.2
+soxr==1.1.0
+spacy==3.8.16
+spacy-legacy==3.0.12
+spacy-loggers==1.0.5
+spaghetti==1.7.6
+spanner-graph-notebook==1.1.10
+spglm==1.1.0
+Sphinx==8.2.3
+sphinxcontrib-applehelp==2.0.0
+sphinxcontrib-devhelp==2.0.0
+sphinxcontrib-htmlhelp==2.1.0
+sphinxcontrib-jsmath==1.0.1
+sphinxcontrib-qthelp==2.0.0
+sphinxcontrib-serializinghtml==2.0.0
+spint==1.1.0
+splot==1.1.7
+spopt==0.7.0
+spreg==1.9.0
+SQLAlchemy==2.0.52
+sqlglot==25.20.2
+sqlitedict==2.1.0
+sqlparse==0.6.0
+squarify==0.4.5
+srsly==2.5.3
+standard-aifc==3.13.0
+standard-chunk==3.13.0
+standard-sunau==3.13.0
+stanio==0.5.1
+starlette==1.6.0
+statsmodels==0.15.0
+stopit==1.1.2
+strictyaml==1.7.3
+stringzilla==5.1.2
+stumpy==1.14.1
+sympy==1.14.0
+tabledata==1.3.5
+tables==3.10.2
+tabulate==0.9.0
+tbb==2022.3.1
+tblib==3.2.2
+tcmlib==1.5.0
+tcolorpy==0.1.7
+tenacity==9.1.4
+tensorboard==2.20.0
+tensorboard-data-server==0.7.2
+tensorflow==2.20.0
+tensorflow-datasets==4.9.10
+tensorflow-hub==0.16.1
+tensorflow-metadata==1.21.0
+tensorflow-probability==0.25.0
+tensorflow-text==2.20.1
+tensorstore==0.1.85
+termcolor==3.3.0
+terminado==0.18.1
+testpath==0.6.0
+text-unidecode==1.3
+textblob==0.19.0
+texttable==1.7.0
+tf-slim==1.1.0
+tf_keras==2.20.1
+thinc==8.3.13
+threadpoolctl==3.6.0
+tifffile==2026.8.23
+tiktoken==0.14.0
+timm==1.0.29
+tinycss2==1.5.1
+tobler==0.14.0
+tokenizers==0.22.1
+toml==0.10.2
+tomlkit==0.14.0
+toolz==0.12.1
+torch==2.11.0+cu128
+torchao==0.10.0
+torchaudio==2.11.0+cu128
+torchcodec==0.11.0+cu128
+torchdata==0.11.0
+torchinfo==1.8.0
+torchmetrics==1.9.0
+torchsummary==1.5.1
+torchtune==0.6.1
+torchvision==0.26.0+cu128
+tornado==6.5.7
+TPOT==1.1.0
+tqdm==4.67.3
+traitlets==5.16.1
+traittypes==0.2.3
+transformers==4.56.2
+treelite==4.7.1
+treescope==0.1.10
+triton==3.6.0
+truststore==0.10.4
+trx-python==0.6
+tsfresh==0.21.2
+tuf==7.0.1
+tweepy==4.17.0
+typeguard==4.6.0
+typepy==1.3.5
+typer==0.27.2
+types-pytz==2026.3.1.20260727
+types-setuptools==84.0.0.20260812
+typing-inspect==0.9.0
+typing-inspection==0.4.4
+typing_extensions==4.16.0
+tzdata==2026.3
+tzlocal==5.4.4
+ucxx-cu12==0.48.0
+ujson==6.0.0
+umap-learn==0.5.12
+umf==1.0.3
+universal_pathlib==0.3.10
+update_checker==1.0.0
+uri-template==1.3.0
+uritemplate==4.2.0
+urllib3==2.8.0
+urwid==4.2.4
+urwid_readline==0.15.1
+uuid_utils==0.17.0
+uvicorn==0.52.4
+uvloop==0.22.1
+vega-datasets==0.9.0
+visions==0.8.1
+vtk==9.7.1
+wadllib==1.3.6
+Wand==0.7.2
+wandb==0.28.1
+wasabi==1.1.3
+watchdog==6.0.0
+watchfiles==1.2.0
+wavio==0.0.9
+wcwidth==0.8.3
+weasel==1.0.0
+webcolors==25.10.0
+webencodings==0.6.1
+websocket-client==1.9.2
+websockets==15.0.1
+Werkzeug==3.1.8
+wheel==0.48.0
+widgetsnbextension==4.0.16
+woodwork==0.31.0
+word2number==1.1
+wordcloud==1.9.6
+wrapt==1.17.3
+xarray==2025.12.0
+xarray-einstats==0.11.0
+xgboost==3.4.1
+xlrd==2.0.2
+xvfbwrapper==0.2.34
+xxhash==4.0.1
+xyzservices==2026.3.0
+yarl==1.24.5
+ydata-profiling==4.18.4
+ydf==0.15.0
+ydf_tf==2.20.0
+yellowbrick==1.5
+yfinance==0.2.66
+zict==3.0.0
+zipp==4.1.0
+zstandard==0.25.0

evaluation/hellaswag/hellaswag_worker.py

@@ -0,0 +1,175 @@
+"""Official lm-evaluation-harness HellaSwag, pinned data and final Think72 export."""
+import argparse
+import hashlib
+import importlib.metadata
+import json
+import math
+import os
+import time
+from collections import Counter
+from datetime import datetime, timezone
+from pathlib import Path
+
+os.environ.setdefault('TOKENIZERS_PARALLELISM', 'false')
+os.environ.setdefault('HF_HUB_DISABLE_TELEMETRY', '1')
+
+import numpy as np
+import torch
+import yaml
+import lm_eval
+from lm_eval import evaluator
+from lm_eval.api.task import ConfigurableTask
+from lm_eval.models.huggingface import HFLM
+from lm_eval.tasks.hellaswag.utils import process_docs
+
+MODEL_SHA = '0f7bdaaae88077174d371c650759b71b83fce6e917638703b72466d59a2b3c9e'
+TOKENIZER_SHA = '27cb97f527f7eee80c7de32e04ba9cc32f8b8c2117d8664690443d53e228433b'
+DATA_SHA = '899813071e1e95efafec90f856e1987d2150fa4d020fc005df6962c259f660cd'
+DATA_REVISION = '218ec52e09a7e7462a5400043bb9a69a41d06b76'
+
+
+def sha(path):
+    with Path(path).open('rb') as handle:
+        return hashlib.file_digest(handle, 'sha256').hexdigest()
+
+
+def json_default(value):
+    if isinstance(value, np.generic):
+        return value.item()
+    if isinstance(value, np.ndarray):
+        return value.tolist()
+    if callable(value):
+        return value.__module__ + '.' + value.__qualname__
+    return str(value)
+
+
+def save(path, value):
+    Path(path).write_text(json.dumps(value, indent=2, default=json_default) + '\n', encoding='utf-8')
+
+
+def wilson(correct, count):
+    p = correct / count
+    z = 1.959963984540054
+    denominator = 1 + z*z/count
+    centre = (p + z*z/(2*count)) / denominator
+    half = z * math.sqrt(p*(1-p)/count + z*z/(4*count*count)) / denominator
+    return [centre-half, centre+half]
+
+
+class AuditedHFLM(HFLM):
+    """Add input-length/progress checks without changing official scoring."""
+    def loglikelihood(self, requests, disable_tqdm=False):
+        maximum = 0
+        for request in requests:
+            context, continuation = self._encode_pair(*request.args)
+            assert context and continuation
+            length = len(context) + len(continuation)
+            maximum = max(maximum, length)
+            assert length <= 1024, 'Benchmark would exceed the trained context; stop instead of silently truncating.'
+        audit = {'rank': self.rank, 'world_size': self.world_size,
+                 'choice_requests': len(requests), 'max_pair_tokens': maximum,
+                 'truncated_requests': 0, 'device': str(self.device)}
+        save(Path(os.environ['THINK72_EVAL_OUT']) / f'rank-{self.rank}-audit.json', audit)
+        print('EVAL_TOKENIZATION_VERIFIED', json.dumps(audit), flush=True)
+        return super().loglikelihood(requests, disable_tqdm=disable_tqdm)
+
+    def _model_call(self, *args, **kwargs):
+        output = super()._model_call(*args, **kwargs)
+        self.audit_batches = getattr(self, 'audit_batches', 0) + 1
+        if self.audit_batches % 50 == 0:
+            print('EVAL_FORWARD_PROGRESS', json.dumps({'rank': self.rank, 'batches': self.audit_batches}), flush=True)
+        return output
+
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--model', type=Path, required=True)
+    parser.add_argument('--data', type=Path, required=True)
+    parser.add_argument('--out', type=Path, required=True)
+    parser.add_argument('--batch-size', type=int, default=32)
+    args = parser.parse_args()
+    args.out.mkdir(parents=True, exist_ok=True)
+    os.environ['THINK72_EVAL_OUT'] = str(args.out)
+    assert sha(args.model/'model.safetensors') == MODEL_SHA
+    assert sha(args.model/'tokenizer.json') == TOKENIZER_SHA
+    assert sha(args.data) == DATA_SHA
+    assert importlib.metadata.version('lm_eval') == '0.4.13'
+    assert torch.cuda.device_count() == 2
+    assert all('T4' in torch.cuda.get_device_name(i) for i in range(2))
+    torch.set_num_threads(2)
+    torch.backends.cuda.matmul.allow_tf32 = False
+    torch.backends.cudnn.allow_tf32 = False
+
+    # Read the installed official task definition. Only its dataset transport is
+    # replaced with the byte-verified Parquet from the same pinned Rowan revision.
+    task_yaml = Path(lm_eval.__file__).parent/'tasks/hellaswag/hellaswag.yaml'
+    class TaskLoader(yaml.SafeLoader):
+        pass
+    def task_function(loader, node):
+        assert loader.construct_scalar(node) == 'utils.process_docs'
+        return process_docs
+    TaskLoader.add_constructor('!function', task_function)
+    config = yaml.load(task_yaml.read_text(encoding='utf-8'), Loader=TaskLoader)
+    config.update(dataset_path='parquet', dataset_kwargs={'data_files': {'validation': str(args.data)}},
+                  training_split=None, num_fewshot=0)
+    task = ConfigurableTask(config=config)
+    task.EVAL_HARNESS_NAME = 'hellaswag'
+    docs = task.validation_docs()
+    assert len(docs) == 10042
+    assert all(len(task.doc_to_choice(doc)) == 4 and task.doc_to_target(doc) in range(4) for doc in docs)
+
+    model = AuditedHFLM(pretrained=str(args.model), tokenizer=str(args.model),
+                        backend='causal', device='cuda', dtype='float32',
+                        softmax_dtype='float32', batch_size=args.batch_size,
+                        max_length=1024, add_bos_token=False,
+                        trust_remote_code=False, attn_implementation='sdpa')
+    assert model.world_size == 2
+    assert sum(p.numel() for p in model.model.parameters()) == 72_000_000
+    print('EVAL_MODEL_VERIFIED', json.dumps({'rank': model.rank, 'device': str(model.device),
+          'step': 22889, 'sha256': MODEL_SHA, 'parameters': 72000000}), flush=True)
+    start = time.monotonic()
+    result = evaluator.simple_evaluate(model=model, tasks=[task], num_fewshot=0,
+                                      limit=None, log_samples=True, bootstrap_iters=1000,
+                                      apply_chat_template=False, random_seed=0,
+                                      numpy_random_seed=1234, torch_random_seed=1234,
+                                      fewshot_random_seed=1234)
+    if model.rank == 0:
+        assert result is not None
+        samples = result.pop('samples')['hellaswag']
+        assert len(samples) == 10042
+        assert len({x['doc_id'] for x in samples}) == 10042
+        with (args.out/'samples.jsonl').open('w', encoding='utf-8') as output:
+            for sample in samples:
+                output.write(json.dumps(sample, default=json_default) + '\n')
+        save(args.out/'results.json', result)
+        metrics = result['results']['hellaswag']
+        summary = {
+            'status': 'COMPLETE', 'completed_utc': datetime.now(timezone.utc).isoformat(),
+            'model': 'Think72 final export', 'model_step': 22889,
+            'training_tokens': 1500053504, 'model_sha256': MODEL_SHA,
+            'tokenizer_sha256': TOKENIZER_SHA, 'parameters': 72000000,
+            'context': 1024, 'dataset': 'Rowan/hellaswag', 'split': 'validation',
+            'dataset_revision': DATA_REVISION, 'dataset_file_sha256': DATA_SHA,
+            'examples': 10042, 'num_fewshot': 0, 'apply_chat_template': False,
+            'dtype': 'float32', 'quantized': False, 'add_bos_token': False,
+            'method': 'Official lm-evaluation-harness four-choice continuation log-likelihood',
+            'acc_norm_denominator': 'Length in characters of the official preprocessed ending',
+            'lm_eval_version': importlib.metadata.version('lm_eval'),
+            'transformers_version': importlib.metadata.version('transformers'),
+            'torch_version': torch.__version__, 'task_yaml_sha256': sha(task_yaml),
+            'world_size': model.world_size, 'gpu_names': [torch.cuda.get_device_name(i) for i in range(2)],
+            'batch_size_per_gpu': args.batch_size, 'elapsed_seconds': time.monotonic()-start,
+            'random_choice_baseline': 0.25, 'metrics': metrics,
+            'label_counts': dict(Counter(int(doc['label']) for doc in docs)),
+            'limitations': 'Zero-shot continuation benchmark, not a chat or generated-reasoning test. Training-data contamination has not been exhaustively audited.'}
+        for key in ('acc', 'acc_norm'):
+            correct = int(sum(sample[key] for sample in samples))
+            assert abs(correct/10042 - metrics[key+',none']) < 1e-12
+            summary[key+'_correct'] = correct
+            summary[key+'_wilson_95_ci'] = wilson(correct, 10042)
+        save(args.out/'summary.json', summary)
+        print('HELLASWAG_FINISHED', json.dumps(summary, default=json_default), flush=True)
+
+
+if __name__ == '__main__':
+    main()

evaluation/hellaswag/rank-0-audit.json

@@ -0,0 +1,8 @@
+{
+  "rank": 0,
+  "world_size": 2,
+  "choice_requests": 20084,
+  "max_pair_tokens": 170,
+  "truncated_requests": 0,
+  "device": "cuda:0"
+}

evaluation/hellaswag/rank-1-audit.json

@@ -0,0 +1,8 @@
+{
+  "rank": 1,
+  "world_size": 2,
+  "choice_requests": 20084,
+  "max_pair_tokens": 158,
+  "truncated_requests": 0,
+  "device": "cuda:1"
+}

evaluation/hellaswag/results.json

@@ -0,0 +1,126 @@
+{
+  "results": {
+    "hellaswag": {
+      "name": "hellaswag",
+      "alias": "hellaswag",
+      "sample_len": 10042,
+      "acc,none": 0.2713602867954591,
+      "acc_stderr,none": 0.0044375277328504974,
+      "acc_norm,none": 0.28360884285998805,
+      "acc_norm_stderr,none": 0.004498280244494868
+    }
+  },
+  "group_subtasks": {},
+  "configs": {
+    "hellaswag": {
+      "task": "hellaswag",
+      "tag": [
+        "multiple_choice"
+      ],
+      "dataset_path": "parquet",
+      "dataset_kwargs": {
+        "data_files": {
+          "validation": "/kaggle/working/hellaswag/validation.parquet"
+        }
+      },
+      "validation_split": "validation",
+      "process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n    def _process_doc(doc):\n        ctx = doc[\"ctx_a\"] + \" \" + doc[\"ctx_b\"].capitalize()\n        out_doc = {\n            \"query\": preprocess(doc[\"activity_label\"] + \": \" + ctx),\n            \"choices\": [preprocess(ending) for ending in doc[\"endings\"]],\n            \"gold\": int(doc[\"label\"]),\n        }\n        return out_doc\n\n    return dataset.map(_process_doc)\n",
+      "doc_to_text": "{{query}}",
+      "doc_to_target": "{{label}}",
+      "unsafe_code": false,
+      "doc_to_choice": "choices",
+      "description": "",
+      "target_delimiter": " ",
+      "fewshot_delimiter": "\n\n",
+      "fewshot_config": {
+        "sampler": "default",
+        "split": null,
+        "process_docs": "lm_eval.tasks.hellaswag.utils.process_docs",
+        "fewshot_indices": null,
+        "samples": null,
+        "doc_to_text": "{{query}}",
+        "doc_to_choice": "choices",
+        "doc_to_target": "{{label}}",
+        "gen_prefix": null,
+        "fewshot_delimiter": "\n\n",
+        "target_delimiter": " "
+      },
+      "num_fewshot": 0,
+      "metric_list": [
+        {
+          "metric": "acc",
+          "aggregation": "mean",
+          "higher_is_better": true
+        },
+        {
+          "metric": "acc_norm",
+          "aggregation": "mean",
+          "higher_is_better": true
+        }
+      ],
+      "output_type": "multiple_choice",
+      "repeats": 1,
+      "should_decontaminate": false,
+      "metadata": {
+        "version": 1.0
+      }
+    }
+  },
+  "versions": {
+    "hellaswag": 1.0
+  },
+  "n-shot": {
+    "hellaswag": 0
+  },
+  "higher_is_better": {
+    "hellaswag": {
+      "acc": true,
+      "acc_norm": true
+    }
+  },
+  "n-samples": {
+    "hellaswag": {
+      "original": 10042,
+      "effective": 10042
+    }
+  },
+  "config": {
+    "model": "/kaggle/input/notebooks/faithdemo/think72-qwen3-t4x2/think72/run/model",
+    "model_args": null,
+    "model_num_parameters": 72000000,
+    "model_dtype": "torch.float32",
+    "model_revision": "main",
+    "model_sha": "",
+    "batch_size": null,
+    "batch_sizes": [],
+    "device": null,
+    "use_cache": null,
+    "limit": null,
+    "bootstrap_iters": 1000,
+    "gen_kwargs": null,
+    "random_seed": 0,
+    "numpy_seed": 1234,
+    "torch_seed": 1234,
+    "fewshot_seed": 1234
+  },
+  "git_hash": null,
+  "date": 1792293424.3973808,
+  "pretty_env_info": "PyTorch version: 2.11.0+cu128\nIs debug build: False\nCUDA used to build PyTorch: 12.8\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 24.04.1 LTS (x86_64)\nGCC version: (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0\nClang version: Could not collect\nCMake version: version 3.31.10\nLibc version: glibc-2.39\n\nPython version: 3.13.15 (main, Aug  6 2026, 11:06:22) [GCC 13.3.0] (64-bit runtime)\nPython platform: Linux-6.18.48+-x86_64-with-glibc2.39\nIs CUDA available: True\nCUDA runtime version: 12.8.93\nCUDA_MODULE_LOADING set to: \nGPU models and configuration: \nGPU 0: Tesla T4\nGPU 1: Tesla T4\n\nNvidia driver version: 580.178.04\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.9.8.0\n/usr/lib/x86_64-linux-gnu/libcudnn_adv.so.9.8.0\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn.so.9.8.0\n/usr/lib/x86_64-linux-gnu/libcudnn_engines_precompiled.so.9.8.0\n/usr/lib/x86_64-linux-gnu/libcudnn_engines_runtime_compiled.so.9.8.0\n/usr/lib/x86_64-linux-gnu/libcudnn_graph.so.9.8.0\n/usr/lib/x86_64-linux-gnu/libcudnn_heuristic.so.9.8.0\n/usr/lib/x86_64-linux-gnu/libcudnn_ops.so.9.8.0\nIs XPU available: False\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\nCaching allocator config: N/A\n\nCPU:\nArchitecture:                            x86_64\nCPU op-mode(s):                          32-bit, 64-bit\nAddress sizes:                           46 bits physical, 48 bits virtual\nByte Order:                              Little Endian\nCPU(s):                                  4\nOn-line CPU(s) list:                     0-3\nVendor ID:                               GenuineIntel\nModel name:                              Intel(R) Xeon(R) CPU @ 2.00GHz\nCPU family:                              6\nModel:                                   85\nThread(s) per core:                      2\nCore(s) per socket:                      2\nSocket(s):                               1\nStepping:                                3\nBogoMIPS:                                4000.39\nFlags:                                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch pti ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves arat md_clear arch_capabilities\nHypervisor vendor:                       KVM\nVirtualization type:                     full\nL1d cache:                               64 KiB (2 instances)\nL1i cache:                               64 KiB (2 instances)\nL2 cache:                                2 MiB (2 instances)\nL3 cache:                                38.5 MiB (1 instance)\nNUMA node(s):                            1\nNUMA node0 CPU(s):                       0-3\nVulnerability Gather data sampling:      Not affected\nVulnerability Ghostwrite:                Not affected\nVulnerability Indirect target selection: Mitigation; Aligned branch/return thunks\nVulnerability Itlb multihit:             Not affected\nVulnerability L1tf:                      Mitigation; PTE Inversion\nVulnerability Mds:                       Mitigation; Clear CPU buffers; SMT Host state unknown\nVulnerability Meltdown:                  Mitigation; PTI\nVulnerability Mmio stale data:           Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown\nVulnerability Old microcode:             Not affected\nVulnerability Reg file data sampling:    Not affected\nVulnerability Retbleed:                  Mitigation; IBRS\nVulnerability Spec rstack overflow:      Not affected\nVulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl\nVulnerability Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2:                Mitigation; IBRS; IBPB conditional; STIBP conditional; RSB filling; PBRSB-eIBRS Not affected; BHI SW loop, KVM SW loop\nVulnerability Srbds:                     Not affected\nVulnerability Tsa:                       Not affected\nVulnerability Tsx async abort:           Mitigation; Clear CPU buffers; SMT Host state unknown\nVulnerability Vmscape:                   Not affected\n\nVersions of relevant libraries:\n[pip3] intel-cmplr-lib-ur==2025.3.3\n[pip3] intel-openmp==2025.3.3\n[pip3] mkl==2025.3.1\n[pip3] mypy_extensions==1.1.0\n[pip3] numpy==2.1.3\n[pip3] nvidia-cublas-cu12==12.8.4.1\n[pip3] nvidia-cuda-cupti-cu12==12.8.90\n[pip3] nvidia-cuda-nvrtc-cu12==12.8.93\n[pip3] nvidia-cuda-runtime-cu12==12.8.90\n[pip3] nvidia-cudnn-cu12==9.19.0.56\n[pip3] nvidia-cufft-cu12==11.3.3.83\n[pip3] nvidia-curand-cu12==10.3.9.90\n[pip3] nvidia-cusolver-cu12==11.7.3.90\n[pip3] nvidia-cusparse-cu12==12.5.8.93\n[pip3] nvidia-cusparselt-cu12==0.7.1\n[pip3] nvidia-nccl-cu12==2.28.9\n[pip3] nvidia-nccl-cu13==2.31.2\n[pip3] nvidia-nvjitlink-cu12==12.8.93\n[pip3] nvidia-nvtx-cu12==12.8.90\n[pip3] nvtx==0.2.16\n[pip3] onemkl-license==2025.3.1\n[pip3] onnx==1.22.0\n[pip3] optree==0.20.0\n[pip3] pytorch-ignite==0.5.5\n[pip3] pytorch-lightning==2.6.6\n[pip3] tbb==2022.3.1\n[pip3] tcmlib==1.5.0\n[pip3] torch==2.11.0+cu128\n[pip3] torchao==0.10.0\n[pip3] torchaudio==2.11.0+cu128\n[pip3] torchcodec==0.11.0+cu128\n[pip3] torchdata==0.11.0\n[pip3] torchinfo==1.8.0\n[pip3] torchmetrics==1.9.0\n[pip3] torchsummary==1.5.1\n[pip3] torchtune==0.6.1\n[pip3] torchvision==0.26.0+cu128\n[pip3] triton==3.6.0\n[pip3] umf==1.0.3\n[conda] Could not collect",
+  "transformers_version": "4.56.2",
+  "lm_eval_version": "0.4.13",
+  "upper_git_hash": null,
+  "tokenizer_pad_token": [
+    "<|pad|>",
+    "0"
+  ],
+  "tokenizer_eos_token": [
+    "<|im_end|>",
+    "2"
+  ],
+  "tokenizer_bos_token": [
+    "<|endoftext|>",
+    "1"
+  ],
+  "eot_token_id": 2,
+  "max_length": 1024
+}

evaluation/hellaswag/summary.json

@@ -0,0 +1,61 @@
+{
+  "status": "COMPLETE",
+  "completed_utc": "2026-10-18T03:19:43.570646+00:00",
+  "model": "Think72 final export",
+  "model_step": 22889,
+  "training_tokens": 1500053504,
+  "model_sha256": "0f7bdaaae88077174d371c650759b71b83fce6e917638703b72466d59a2b3c9e",
+  "tokenizer_sha256": "27cb97f527f7eee80c7de32e04ba9cc32f8b8c2117d8664690443d53e228433b",
+  "parameters": 72000000,
+  "context": 1024,
+  "dataset": "Rowan/hellaswag",
+  "split": "validation",
+  "dataset_revision": "218ec52e09a7e7462a5400043bb9a69a41d06b76",
+  "dataset_file_sha256": "899813071e1e95efafec90f856e1987d2150fa4d020fc005df6962c259f660cd",
+  "examples": 10042,
+  "num_fewshot": 0,
+  "apply_chat_template": false,
+  "dtype": "float32",
+  "quantized": false,
+  "add_bos_token": false,
+  "method": "Official lm-evaluation-harness four-choice continuation log-likelihood",
+  "acc_norm_denominator": "Length in characters of the official preprocessed ending",
+  "lm_eval_version": "0.4.13",
+  "transformers_version": "4.56.2",
+  "torch_version": "2.11.0+cu128",
+  "task_yaml_sha256": "e9ef8ac3fed02bf283777d946ae76cbf906152e9c92533c9794e5a9dad78d1cf",
+  "world_size": 2,
+  "gpu_names": [
+    "Tesla T4",
+    "Tesla T4"
+  ],
+  "batch_size_per_gpu": 32,
+  "elapsed_seconds": 159.177716964,
+  "random_choice_baseline": 0.25,
+  "metrics": {
+    "name": "hellaswag",
+    "alias": "hellaswag",
+    "sample_len": 10042,
+    "acc,none": 0.2713602867954591,
+    "acc_stderr,none": 0.0044375277328504974,
+    "acc_norm,none": 0.28360884285998805,
+    "acc_norm_stderr,none": 0.004498280244494868
+  },
+  "label_counts": {
+    "3": 2458,
+    "2": 2584,
+    "1": 2485,
+    "0": 2515
+  },
+  "limitations": "Zero-shot continuation benchmark, not a chat or generated-reasoning test. Training-data contamination has not been exhaustively audited.",
+  "acc_correct": 2725,
+  "acc_wilson_95_ci": [
+    0.2627519789799681,
+    0.28014345503387256
+  ],
+  "acc_norm_correct": 2848,
+  "acc_norm_wilson_95_ci": [
+    0.2748768583926779,
+    0.2925063202275641
+  ]
+}

evaluation/hellaswag/verification.json

@@ -0,0 +1,78 @@
+{
+  "status": "VERIFIED",
+  "verified_utc": "2026-10-18T03:24:41.562026+00:00",
+  "source": {
+    "kernel": "faithdemo/think72-final-hellaswag-t4x2",
+    "version": 1,
+    "script_version_id": 359563596,
+    "model_step": 22889
+  },
+  "model_step": 22889,
+  "model_sha256": "0f7bdaaae88077174d371c650759b71b83fce6e917638703b72466d59a2b3c9e",
+  "dataset_sha256": "899813071e1e95efafec90f856e1987d2150fa4d020fc005df6962c259f660cd",
+  "validation_rows": 10042,
+  "unique_scored_rows": 10042,
+  "source_rows_labels_prompts_and_all_choice_scores_verified": true,
+  "independently_recomputed_correct": {
+    "acc": 2725,
+    "acc_norm": 2848
+  },
+  "independently_recomputed_accuracy": {
+    "acc": 0.2713602867954591,
+    "acc_norm": 0.28360884285998805
+  },
+  "random_choice_baseline": 0.25,
+  "rank_audits": [
+    {
+      "rank": 0,
+      "world_size": 2,
+      "choice_requests": 20084,
+      "max_pair_tokens": 170,
+      "truncated_requests": 0,
+      "device": "cuda:0"
+    },
+    {
+      "rank": 1,
+      "world_size": 2,
+      "choice_requests": 20084,
+      "max_pair_tokens": 158,
+      "truncated_requests": 0,
+      "device": "cuda:1"
+    }
+  ],
+  "files": {
+    "environment.txt": {
+      "bytes": 18374,
+      "sha256": "985d80012c8ffc6a8eb64f247f06112838b44788ab73b054a871ee1dbc2af760"
+    },
+    "hellaswag_worker.py": {
+      "bytes": 8555,
+      "sha256": "597f59452a387041f6e0b457df5d79a104a3864edf502f963096d7aeee939138"
+    },
+    "rank-0-audit.json": {
+      "bytes": 138,
+      "sha256": "cbde996d6e71f0e9b67692ba069428482bc130b3e73681a97244359475599ba8"
+    },
+    "rank-1-audit.json": {
+      "bytes": 138,
+      "sha256": "b7c0a5cf66d64d3bf56d5bf99f23653d0808e286b837235e487dce8f94e63c00"
+    },
+    "results.json": {
+      "bytes": 9185,
+      "sha256": "f87fa8a086b366164f2508ae8eec32128043dd15bbb132617c87ced82c81846d"
+    },
+    "samples.jsonl": {
+      "bytes": 42048156,
+      "sha256": "5e70b2d098307c2b84632966bcf5b123da7c4591cfbf19d50a24f46453b7c069"
+    },
+    "summary.json": {
+      "bytes": 2006,
+      "sha256": "197021e4284a188dce7a2015a4d9a424796ea67acf1309ee9810a8f0211b9081"
+    },
+    "validation.parquet": {
+      "bytes": 6315951,
+      "sha256": "899813071e1e95efafec90f856e1987d2150fa4d020fc005df6962c259f660cd"
+    }
+  },
+  "limitations": "Recomputes metrics from recorded inference scores, not an independent second model inference. Training-data contamination has not been exhaustively audited."
+}

generation_config.json

@@ -0,0 +1,8 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "pad_token_id": 0,
+  "transformers_version": "4.56.2",
+  "use_cache": false
+}

licenses/Apache-2.0.txt

@@ -0,0 +1,202 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0f7bdaaae88077174d371c650759b71b83fce6e917638703b72466d59a2b3c9e
+size 288017256

provenance/corpus_manifest.json

@@ -0,0 +1,280 @@
+{
+  "created_utc": "2026-10-16T20:20:23.840418+00:00",
+  "seed": 20261017,
+  "dtype": "little-endian uint16",
+  "context": 1024,
+  "vocab_size": 33069,
+  "source_manifest_sha256": "8457a299e8602fb874b51ffac7c5a93460ba93e26d5f616b40991d52d33a0923",
+  "tokenizer_sha256": "27cb97f527f7eee80c7de32e04ba9cc32f8b8c2117d8664690443d53e228433b",
+  "requested_train_tokens": 1520000000,
+  "sampling": "Deterministically shuffled row groups; per-file train-token quota proportional to compressed file bytes; retain whole records including final answers. Quotas may overshoot by one encoding batch.",
+  "split_rule": "SHA256(NFKC/casefold/whitespace-normalized prompt) modulo 1000: validation 0..4; test 5..9; train 10..999. Split before tokenizer fitting and packing.",
+  "packing": "Separate continuous token streams per split; EOS separates records; long documents continue into later windows. Attention resets every 1024 tokens; cross-document attention within a window is allowed.",
+  "loss": "All next-token targets except pad=0; no assistant-only loss at scratch-pretraining stage.",
+  "rows_scanned": 692288,
+  "selected_rows": {
+    "train": 670031,
+    "test": 3389,
+    "validation": 3348
+  },
+  "exact_duplicate_rows_removed": 15520,
+  "source_tokens": {
+    "simplescaling__s1K-1.1__data__train-00000-of-00001.parquet/train": 907355,
+    "open-r1__OpenR1-Math-220k__extended__train-00005-of-00010.parquet/train": 6329669,
+    "open-r1__OpenR1-Math-220k__extended__train-00005-of-00010.parquet/test": 22996,
+    "open-r1__OpenR1-Math-220k__extended__train-00005-of-00010.parquet/validation": 27831,
+    "open-thoughts__OpenThoughts-114k__data__train-00003-of-00006.parquet/train": 16828088,
+    "open-thoughts__OpenThoughts-114k__data__train-00003-of-00006.parquet/test": 95336,
+    "open-thoughts__OpenThoughts-114k__data__train-00003-of-00006.parquet/validation": 51167,
+    "open-r1__Mixture-of-Thoughts__math__train-00001-of-00004.parquet/train": 16914253,
+    "open-r1__Mixture-of-Thoughts__math__train-00001-of-00004.parquet/validation": 62564,
+    "open-r1__Mixture-of-Thoughts__math__train-00001-of-00004.parquet/test": 63648,
+    "open-thoughts__OpenThoughts-114k__data__train-00004-of-00006.parquet/train": 14645665,
+    "open-thoughts__OpenThoughts-114k__data__train-00004-of-00006.parquet/test": 70829,
+    "open-thoughts__OpenThoughts-114k__data__train-00004-of-00006.parquet/validation": 56913,
+    "nvidia__Llama-Nemotron-Post-Training-Dataset__SFT__safety__safety.parquet/train": 979612,
+    "nvidia__Llama-Nemotron-Post-Training-Dataset__SFT__safety__safety.parquet/test": 6711,
+    "nvidia__Llama-Nemotron-Post-Training-Dataset__SFT__safety__safety.parquet/validation": 6135,
+    "nvidia__Llama-Nemotron-Post-Training-Dataset__SFT__science__science.parquet/train": 254150591,
+    "nvidia__Llama-Nemotron-Post-Training-Dataset__SFT__science__science.parquet/test": 1328102,
+    "nvidia__Llama-Nemotron-Post-Training-Dataset__SFT__science__science.parquet/validation": 1234004,
+    "meta-math__MetaMathQA__MetaMathQA-395K.parquet/train": 13204543,
+    "meta-math__MetaMathQA__MetaMathQA-395K.parquet/test": 65737,
+    "meta-math__MetaMathQA__MetaMathQA-395K.parquet/validation": 68681,
+    "AI-MO__NuminaMath-CoT__data__train-00004-of-00005.parquet/train": 11779246,
+    "AI-MO__NuminaMath-CoT__data__train-00004-of-00005.parquet/test": 56193,
+    "AI-MO__NuminaMath-CoT__data__train-00004-of-00005.parquet/validation": 54188,
+    "open-r1__Mixture-of-Thoughts__math__train-00000-of-00004.parquet/train": 16935017,
+    "open-r1__Mixture-of-Thoughts__math__train-00000-of-00004.parquet/test": 108656,
+    "open-r1__Mixture-of-Thoughts__math__train-00000-of-00004.parquet/validation": 96225,
+    "open-r1__Mixture-of-Thoughts__math__train-00003-of-00004.parquet/train": 16819808,
+    "open-r1__Mixture-of-Thoughts__math__train-00003-of-00004.parquet/test": 75752,
+    "open-r1__Mixture-of-Thoughts__math__train-00003-of-00004.parquet/validation": 99972,
+    "AI-MO__NuminaMath-CoT__data__train-00002-of-00005.parquet/train": 11770760,
+    "AI-MO__NuminaMath-CoT__data__train-00002-of-00005.parquet/test": 54142,
+    "AI-MO__NuminaMath-CoT__data__train-00002-of-00005.parquet/validation": 67014,
+    "AI-MO__NuminaMath-CoT__data__train-00001-of-00005.parquet/train": 11765808,
+    "AI-MO__NuminaMath-CoT__data__train-00001-of-00005.parquet/test": 61146,
+    "AI-MO__NuminaMath-CoT__data__train-00001-of-00005.parquet/validation": 67511,
+    "open-r1__OpenR1-Math-220k__extended__train-00002-of-00010.parquet/train": 6430762,
+    "open-r1__OpenR1-Math-220k__extended__train-00002-of-00010.parquet/test": 45143,
+    "open-r1__OpenR1-Math-220k__extended__train-00002-of-00010.parquet/validation": 28023,
+    "open-r1__Mixture-of-Thoughts__code__train-00003-of-00008.parquet/train": 18069607,
+    "open-r1__Mixture-of-Thoughts__code__train-00003-of-00008.parquet/test": 101545,
+    "open-r1__Mixture-of-Thoughts__code__train-00003-of-00008.parquet/validation": 79791,
+    "open-r1__Mixture-of-Thoughts__code__train-00004-of-00008.parquet/train": 17954395,
+    "open-r1__Mixture-of-Thoughts__code__train-00004-of-00008.parquet/validation": 93818,
+    "open-r1__Mixture-of-Thoughts__code__train-00004-of-00008.parquet/test": 85448,
+    "open-r1__Mixture-of-Thoughts__code__train-00000-of-00008.parquet/train": 18055726,
+    "open-r1__Mixture-of-Thoughts__code__train-00000-of-00008.parquet/validation": 108580,
+    "open-r1__Mixture-of-Thoughts__code__train-00000-of-00008.parquet/test": 134291,
+    "open-thoughts__OpenThoughts-114k__data__train-00002-of-00006.parquet/train": 16722292,
+    "open-thoughts__OpenThoughts-114k__data__train-00002-of-00006.parquet/test": 102951,
+    "open-thoughts__OpenThoughts-114k__data__train-00002-of-00006.parquet/validation": 92132,
+    "open-r1__OpenR1-Math-220k__data__train-00003-of-00010.parquet/train": 6891434,
+    "open-r1__OpenR1-Math-220k__data__train-00003-of-00010.parquet/test": 38141,
+    "open-r1__OpenR1-Math-220k__data__train-00003-of-00010.parquet/validation": 31670,
+    "nvidia__Llama-Nemotron-Post-Training-Dataset__SFT__chat__chat.parquet/train": 6089771,
+    "nvidia__Llama-Nemotron-Post-Training-Dataset__SFT__chat__chat.parquet/validation": 27430,
+    "nvidia__Llama-Nemotron-Post-Training-Dataset__SFT__chat__chat.parquet/test": 32955,
+    "open-r1__Mixture-of-Thoughts__science__train-00002-of-00004.parquet/train": 14083741,
+    "open-r1__Mixture-of-Thoughts__science__train-00002-of-00004.parquet/test": 48934,
+    "open-r1__Mixture-of-Thoughts__science__train-00002-of-00004.parquet/validation": 51822,
+    "open-r1__OpenR1-Math-220k__data__train-00008-of-00010.parquet/train": 6850762,
+    "open-r1__OpenR1-Math-220k__data__train-00008-of-00010.parquet/validation": 30554,
+    "open-r1__OpenR1-Math-220k__data__train-00008-of-00010.parquet/test": 79278,
+    "AI-MO__NuminaMath-CoT__data__train-00000-of-00005.parquet/train": 11766066,
+    "AI-MO__NuminaMath-CoT__data__train-00000-of-00005.parquet/test": 60793,
+    "AI-MO__NuminaMath-CoT__data__train-00000-of-00005.parquet/validation": 59237,
+    "simplescaling__s1K__data__train-00000-of-00001.parquet/train": 676159,
+    "open-thoughts__OpenThoughts-114k__data__train-00005-of-00006.parquet/train": 14483195,
+    "open-thoughts__OpenThoughts-114k__data__train-00005-of-00006.parquet/test": 66008,
+    "open-thoughts__OpenThoughts-114k__data__train-00005-of-00006.parquet/validation": 74267,
+    "open-r1__OpenR1-Math-220k__extended__train-00000-of-00010.parquet/train": 6465783,
+    "open-r1__OpenR1-Math-220k__extended__train-00000-of-00010.parquet/validation": 43140,
+    "open-r1__OpenR1-Math-220k__extended__train-00000-of-00010.parquet/test": 26437,
+    "open-r1__OpenR1-Math-220k__data__train-00007-of-00010.parquet/train": 6934280,
+    "open-r1__OpenR1-Math-220k__data__train-00007-of-00010.parquet/test": 16975,
+    "open-r1__OpenR1-Math-220k__data__train-00007-of-00010.parquet/validation": 37630,
+    "open-r1__OpenR1-Math-220k__data__train-00009-of-00010.parquet/train": 6896161,
+    "open-r1__OpenR1-Math-220k__data__train-00009-of-00010.parquet/test": 25234,
+    "open-r1__OpenR1-Math-220k__data__train-00009-of-00010.parquet/validation": 24626,
+    "a-m-team__AM-DeepSeek-R1-Distilled-1.4M__am_0.5M.parquet/train": 284673619,
+    "a-m-team__AM-DeepSeek-R1-Distilled-1.4M__am_0.5M.parquet/validation": 1431121,
+    "a-m-team__AM-DeepSeek-R1-Distilled-1.4M__am_0.5M.parquet/test": 1311833,
+    "open-r1__Mixture-of-Thoughts__science__train-00000-of-00004.parquet/train": 13937093,
+    "open-r1__Mixture-of-Thoughts__science__train-00000-of-00004.parquet/test": 51476,
+    "open-r1__Mixture-of-Thoughts__science__train-00000-of-00004.parquet/validation": 49754,
+    "open-r1__Mixture-of-Thoughts__code__train-00007-of-00008.parquet/train": 17919163,
+    "open-r1__Mixture-of-Thoughts__code__train-00007-of-00008.parquet/test": 93156,
+    "open-r1__Mixture-of-Thoughts__code__train-00007-of-00008.parquet/validation": 59786,
+    "GAIR__LIMO__limo.parquet/train": 779529,
+    "GAIR__LIMO__limo.parquet/test": 10417,
+    "openai__gsm8k__socratic__train-00000-of-00001.parquet/train": 241220,
+    "openai__gsm8k__socratic__train-00000-of-00001.parquet/test": 867,
+    "openai__gsm8k__socratic__train-00000-of-00001.parquet/validation": 1003,
+    "open-r1__Mixture-of-Thoughts__code__train-00005-of-00008.parquet/train": 17968677,
+    "open-r1__Mixture-of-Thoughts__code__train-00005-of-00008.parquet/test": 87012,
+    "open-r1__Mixture-of-Thoughts__code__train-00005-of-00008.parquet/validation": 57292,
+    "open-r1__Mixture-of-Thoughts__code__train-00001-of-00008.parquet/train": 17648883,
+    "open-r1__Mixture-of-Thoughts__code__train-00001-of-00008.parquet/test": 115190,
+    "open-r1__Mixture-of-Thoughts__code__train-00001-of-00008.parquet/validation": 135718,
+    "microsoft__orca-math-word-problems-200k__data__train-00000-of-00001.parquet/train": 7790328,
+    "microsoft__orca-math-word-problems-200k__data__train-00000-of-00001.parquet/test": 42587,
+    "microsoft__orca-math-word-problems-200k__data__train-00000-of-00001.parquet/validation": 41531,
+    "open-r1__OpenR1-Math-220k__extended__train-00004-of-00010.parquet/train": 6329059,
+    "open-r1__OpenR1-Math-220k__extended__train-00004-of-00010.parquet/validation": 21178,
+    "open-r1__OpenR1-Math-220k__extended__train-00004-of-00010.parquet/test": 26995,
+    "open-r1__OpenR1-Math-220k__extended__train-00003-of-00010.parquet/train": 6351532,
+    "open-r1__OpenR1-Math-220k__extended__train-00003-of-00010.parquet/test": 50760,
+    "open-r1__OpenR1-Math-220k__extended__train-00003-of-00010.parquet/validation": 28180,
+    "AI-MO__NuminaMath-CoT__data__train-00003-of-00005.parquet/train": 11779056,
+    "AI-MO__NuminaMath-CoT__data__train-00003-of-00005.parquet/validation": 57724,
+    "AI-MO__NuminaMath-CoT__data__train-00003-of-00005.parquet/test": 60218,
+    "open-r1__OpenR1-Math-220k__data__train-00005-of-00010.parquet/train": 6676032,
+    "open-r1__OpenR1-Math-220k__data__train-00005-of-00010.parquet/test": 38216,
+    "open-r1__OpenR1-Math-220k__data__train-00005-of-00010.parquet/validation": 8539,
+    "open-r1__OpenR1-Math-220k__extended__train-00006-of-00010.parquet/train": 6405403,
+    "open-r1__OpenR1-Math-220k__extended__train-00006-of-00010.parquet/test": 34496,
+    "open-r1__OpenR1-Math-220k__extended__train-00006-of-00010.parquet/validation": 43472,
+    "open-r1__Mixture-of-Thoughts__math__train-00002-of-00004.parquet/train": 16682931,
+    "open-r1__Mixture-of-Thoughts__math__train-00002-of-00004.parquet/test": 96981,
+    "open-r1__Mixture-of-Thoughts__math__train-00002-of-00004.parquet/validation": 94605,
+    "open-r1__OpenR1-Math-220k__data__train-00001-of-00010.parquet/train": 6751061,
+    "open-r1__OpenR1-Math-220k__data__train-00001-of-00010.parquet/validation": 4413,
+    "open-r1__OpenR1-Math-220k__data__train-00001-of-00010.parquet/test": 34371,
+    "open-r1__Mixture-of-Thoughts__science__train-00003-of-00004.parquet/train": 17976925,
+    "open-r1__Mixture-of-Thoughts__science__train-00003-of-00004.parquet/validation": 39085,
+    "open-r1__Mixture-of-Thoughts__science__train-00003-of-00004.parquet/test": 146403,
+    "openai__gsm8k__main__train-00000-of-00001.parquet/train": 214777,
+    "openai__gsm8k__main__train-00000-of-00001.parquet/test": 820,
+    "openai__gsm8k__main__train-00000-of-00001.parquet/validation": 805,
+    "open-r1__OpenR1-Math-220k__data__train-00006-of-00010.parquet/train": 6640434,
+    "open-r1__OpenR1-Math-220k__data__train-00006-of-00010.parquet/validation": 51143,
+    "open-r1__OpenR1-Math-220k__data__train-00006-of-00010.parquet/test": 48048,
+    "open-r1__Mixture-of-Thoughts__code__train-00006-of-00008.parquet/train": 18591554,
+    "open-r1__Mixture-of-Thoughts__code__train-00006-of-00008.parquet/test": 148827,
+    "open-r1__Mixture-of-Thoughts__code__train-00006-of-00008.parquet/validation": 61471,
+    "open-r1__OpenR1-Math-220k__extended__train-00008-of-00010.parquet/train": 6243132,
+    "open-r1__OpenR1-Math-220k__extended__train-00008-of-00010.parquet/test": 24500,
+    "open-r1__OpenR1-Math-220k__extended__train-00008-of-00010.parquet/validation": 58794,
+    "open-r1__OpenR1-Math-220k__data__train-00004-of-00010.parquet/train": 7022711,
+    "open-r1__OpenR1-Math-220k__data__train-00004-of-00010.parquet/test": 41320,
+    "open-r1__OpenR1-Math-220k__data__train-00004-of-00010.parquet/validation": 6905,
+    "open-r1__OpenR1-Math-220k__extended__train-00007-of-00010.parquet/train": 6308278,
+    "open-r1__OpenR1-Math-220k__extended__train-00007-of-00010.parquet/validation": 27762,
+    "open-r1__OpenR1-Math-220k__extended__train-00007-of-00010.parquet/test": 53536,
+    "open-thoughts__OpenThoughts-114k__data__train-00000-of-00006.parquet/train": 23327800,
+    "open-thoughts__OpenThoughts-114k__data__train-00000-of-00006.parquet/test": 117044,
+    "open-thoughts__OpenThoughts-114k__data__train-00000-of-00006.parquet/validation": 143732,
+    "open-thoughts__OpenThoughts-114k__data__train-00001-of-00006.parquet/train": 16844678,
+    "open-thoughts__OpenThoughts-114k__data__train-00001-of-00006.parquet/test": 84367,
+    "open-thoughts__OpenThoughts-114k__data__train-00001-of-00006.parquet/validation": 51259,
+    "open-r1__OpenR1-Math-220k__data__train-00002-of-00010.parquet/validation": 49517,
+    "open-r1__OpenR1-Math-220k__data__train-00002-of-00010.parquet/train": 6889947,
+    "open-r1__OpenR1-Math-220k__data__train-00002-of-00010.parquet/test": 14586,
+    "open-r1__OpenR1-Math-220k__extended__train-00009-of-00010.parquet/train": 6351371,
+    "open-r1__OpenR1-Math-220k__extended__train-00009-of-00010.parquet/test": 35161,
+    "open-r1__OpenR1-Math-220k__extended__train-00009-of-00010.parquet/validation": 22551,
+    "open-r1__OpenR1-Math-220k__data__train-00000-of-00010.parquet/train": 6780881,
+    "open-r1__OpenR1-Math-220k__data__train-00000-of-00010.parquet/test": 16902,
+    "open-r1__OpenR1-Math-220k__data__train-00000-of-00010.parquet/validation": 52380,
+    "bespokelabs__Bespoke-Stratos-17k__data__train-00000-of-00001.parquet/train": 11869729,
+    "bespokelabs__Bespoke-Stratos-17k__data__train-00000-of-00001.parquet/test": 68721,
+    "bespokelabs__Bespoke-Stratos-17k__data__train-00000-of-00001.parquet/validation": 70604,
+    "open-r1__OpenR1-Math-220k__extended__train-00001-of-00010.parquet/train": 6335342,
+    "open-r1__OpenR1-Math-220k__extended__train-00001-of-00010.parquet/validation": 11937,
+    "open-r1__OpenR1-Math-220k__extended__train-00001-of-00010.parquet/test": 30075,
+    "NovaSky-AI__Sky-T1_data_17k__Sky-T1_data_17k.parquet/train": 8556312,
+    "NovaSky-AI__Sky-T1_data_17k__Sky-T1_data_17k.parquet/validation": 34735,
+    "NovaSky-AI__Sky-T1_data_17k__Sky-T1_data_17k.parquet/test": 52564,
+    "open-r1__Mixture-of-Thoughts__science__train-00001-of-00004.parquet/train": 15032116,
+    "open-r1__Mixture-of-Thoughts__science__train-00001-of-00004.parquet/validation": 143556,
+    "open-r1__Mixture-of-Thoughts__science__train-00001-of-00004.parquet/test": 56547,
+    "a-m-team__AM-DeepSeek-R1-Distilled-1.4M__am_0.9M.parquet/train": 373274101,
+    "a-m-team__AM-DeepSeek-R1-Distilled-1.4M__am_0.9M.parquet/validation": 1914871,
+    "a-m-team__AM-DeepSeek-R1-Distilled-1.4M__am_0.9M.parquet/test": 1846280,
+    "open-r1__Mixture-of-Thoughts__code__train-00002-of-00008.parquet/train": 18105561,
+    "open-r1__Mixture-of-Thoughts__code__train-00002-of-00008.parquet/test": 168116,
+    "open-r1__Mixture-of-Thoughts__code__train-00002-of-00008.parquet/validation": 122042
+  },
+  "kinds": {
+    "long_think": 459936,
+    "chat": 5885,
+    "short_cot": 210947
+  },
+  "splits": {
+    "train": {
+      "real_tokens": 1529699784,
+      "stored_tokens": 1529700353,
+      "padding_tokens": 569,
+      "blocks": 1493848,
+      "files": [
+        {
+          "name": "train-0000.bin",
+          "tokens": 256000000,
+          "bytes": 512000000,
+          "sha256": "2015811a0364a360b1e87d410c9761988ceb619a7c4bc09cf610068204cd43e7"
+        },
+        {
+          "name": "train-0001.bin",
+          "tokens": 256000000,
+          "bytes": 512000000,
+          "sha256": "a3f6b47d936994469bdb468cb579f28acc295d5301d8b7fe82e281a2d0f21221"
+        },
+        {
+          "name": "train-0002.bin",
+          "tokens": 256000000,
+          "bytes": 512000000,
+          "sha256": "2054e82224fc0c4a2b16aeb22e15f797459a3c92dc0477739b28bf070a49a55b"
+        },
+        {
+          "name": "train-0003.bin",
+          "tokens": 256000000,
+          "bytes": 512000000,
+          "sha256": "0cc365f30d802ca91a4c5f76aec0ba7c3bba24863d206da1070d081da060e305"
+        },
+        {
+          "name": "train-0004.bin",
+          "tokens": 256000000,
+          "bytes": 512000000,
+          "sha256": "55dfeebccfff2cb349d57f5b536273b5a5f37c823b051085d429933c294d940d"
+        },
+        {
+          "name": "train-0005.bin",
+          "tokens": 249700353,
+          "bytes": 499400706,
+          "sha256": "d5253d2544e0ecfc829ba281acfa0e0b81447912def703f91811024316da51f6"
+        }
+      ]
+    },
+    "validation": {
+      "real_tokens": 7572390,
+      "stored_tokens": 7572481,
+      "padding_tokens": 91,
+      "blocks": 7395,
+      "files": [
+        {
+          "name": "validation-0000.bin",
+          "tokens": 7572481,
+          "bytes": 15144962,
+          "sha256": "58eb5084adf3c4394021ade1483948d9a7e1232eb3e7897e229285a87e5b112f"
+        }
+      ]
+    },
+    "test": {
+      "real_tokens": 7881773,
+      "stored_tokens": 7882753,
+      "padding_tokens": 980,
+      "blocks": 7698,
+      "files": [
+        {
+          "name": "test-0000.bin",
+          "tokens": 7882753,
+          "bytes": 15765506,
+          "sha256": "b0c684f79daaa03e18a27b20e9b1d3e5cd834eb07a0f656329d7d561822095af"
+        }
+      ]
+    }
+  }
+}

provenance/observed_upstream_revisions_at_training.json

@@ -0,0 +1,130 @@
+[
+  {
+    "repo": "AI-MO/NuminaMath-CoT",
+    "url": "https://huggingface.co/datasets/AI-MO/NuminaMath-CoT",
+    "license": "apache-2.0",
+    "license_name": null,
+    "license_link": null,
+    "current_revision": "9d8d210c9f6a36c8f3cd84045668c9b7800ef517",
+    "historical_local_revision": null
+  },
+  {
+    "repo": "GAIR/LIMO",
+    "url": "https://huggingface.co/datasets/GAIR/LIMO",
+    "license": "apache-2.0",
+    "license_name": null,
+    "license_link": null,
+    "current_revision": "b60f4462da9d927930b9c9bd43399cf875564416",
+    "historical_local_revision": null
+  },
+  {
+    "repo": "NovaSky-AI/Sky-T1_data_17k",
+    "url": "https://huggingface.co/datasets/NovaSky-AI/Sky-T1_data_17k",
+    "license": "apache-2.0",
+    "license_name": null,
+    "license_link": null,
+    "current_revision": "3e260822dae5d833d9b040e34265d5f9a2b8a6a5",
+    "historical_local_revision": null
+  },
+  {
+    "repo": "a-m-team/AM-DeepSeek-R1-Distilled-1.4M",
+    "url": "https://huggingface.co/datasets/a-m-team/AM-DeepSeek-R1-Distilled-1.4M",
+    "license": "cc-by-nc-4.0",
+    "license_name": null,
+    "license_link": null,
+    "current_revision": "53531c06634904118a2dcd83961918c4d69d1cdf",
+    "historical_local_revision": null
+  },
+  {
+    "repo": "bespokelabs/Bespoke-Stratos-17k",
+    "url": "https://huggingface.co/datasets/bespokelabs/Bespoke-Stratos-17k",
+    "license": "apache-2.0",
+    "license_name": null,
+    "license_link": null,
+    "current_revision": "9e9adba943911a9fc44dffcb30aaa18dc96ae6df",
+    "historical_local_revision": null
+  },
+  {
+    "repo": "meta-math/MetaMathQA",
+    "url": "https://huggingface.co/datasets/meta-math/MetaMathQA",
+    "license": "mit",
+    "license_name": null,
+    "license_link": null,
+    "current_revision": "aa4f34d3d2d3231299b5b03d9b3e5a20da45aa18",
+    "historical_local_revision": null
+  },
+  {
+    "repo": "microsoft/orca-math-word-problems-200k",
+    "url": "https://huggingface.co/datasets/microsoft/orca-math-word-problems-200k",
+    "license": "mit",
+    "license_name": null,
+    "license_link": null,
+    "current_revision": "29255d1770cc4eac66e5e7fa378cba542c026350",
+    "historical_local_revision": null
+  },
+  {
+    "repo": "nvidia/Llama-Nemotron-Post-Training-Dataset",
+    "url": "https://huggingface.co/datasets/nvidia/Llama-Nemotron-Post-Training-Dataset",
+    "license": "cc-by-4.0",
+    "license_name": null,
+    "license_link": null,
+    "current_revision": "ab2a40d258a6a4d9d4c277d702aeea445081766c",
+    "historical_local_revision": null
+  },
+  {
+    "repo": "open-r1/Mixture-of-Thoughts",
+    "url": "https://huggingface.co/datasets/open-r1/Mixture-of-Thoughts",
+    "license": null,
+    "license_name": null,
+    "license_link": null,
+    "current_revision": "e55fa28006c0d0ec60fb3547520f775dd42d02cd",
+    "historical_local_revision": null
+  },
+  {
+    "repo": "open-r1/OpenR1-Math-220k",
+    "url": "https://huggingface.co/datasets/open-r1/OpenR1-Math-220k",
+    "license": "apache-2.0",
+    "license_name": null,
+    "license_link": null,
+    "current_revision": "e4e141ec9dea9f8326f4d347be56105859b2bd68",
+    "historical_local_revision": null
+  },
+  {
+    "repo": "open-thoughts/OpenThoughts-114k",
+    "url": "https://huggingface.co/datasets/open-thoughts/OpenThoughts-114k",
+    "license": "apache-2.0",
+    "license_name": null,
+    "license_link": null,
+    "current_revision": "bd093c3994fd54d2390985b66988ddf282a55eb6",
+    "historical_local_revision": null
+  },
+  {
+    "repo": "openai/gsm8k",
+    "url": "https://huggingface.co/datasets/openai/gsm8k",
+    "license": [
+      "mit"
+    ],
+    "license_name": null,
+    "license_link": null,
+    "current_revision": "740312add88f781978c0658806c59bc2815b9866",
+    "historical_local_revision": null
+  },
+  {
+    "repo": "simplescaling/s1K",
+    "url": "https://huggingface.co/datasets/simplescaling/s1K",
+    "license": "apache-2.0",
+    "license_name": null,
+    "license_link": null,
+    "current_revision": "278d72baaa2b887a7e76a70a0ae254a5a45536e4",
+    "historical_local_revision": null
+  },
+  {
+    "repo": "simplescaling/s1K-1.1",
+    "url": "https://huggingface.co/datasets/simplescaling/s1K-1.1",
+    "license": "mit",
+    "license_name": null,
+    "license_link": null,
+    "current_revision": "96c411f1fe4c49d20f0e2a1565f61e1a28b0b84d",
+    "historical_local_revision": null
+  }
+]

provenance/source_manifest.json

@@ -0,0 +1,686 @@
+{
+  "created_utc": "2026-10-16T19:47:52.064014+00:00",
+  "source_directory": "D:\\AI_DATA\\wiki_clean\\thinking\\english",
+  "source_revision_note": "Historical upstream revisions were not retained in these cleaned local files. Local SHA-256 values define this exact snapshot; current upstream revisions are not claimed as historical identities.",
+  "limits": {
+    "total_bytes_strictly_less_than": 20000000000,
+    "single_file_bytes_strictly_less_than": 7000000000
+  },
+  "file_count": 61,
+  "total_bytes": 11142811114,
+  "total_rows": 4188730,
+  "files": [
+    {
+      "name": "a-m-team__AM-DeepSeek-R1-Distilled-1.4M__am_0.5M.parquet",
+      "bytes": 2086874144,
+      "sha256": "f7705bc2662c3aa11bfef47ec38b49ce03ef1fa32a13f5a275a527cc5703a421",
+      "rows": 477802,
+      "row_groups": 96,
+      "source": "AM-1.4M",
+      "upstream_repo": "a-m-team/AM-DeepSeek-R1-Distilled-1.4M",
+      "source_revision": null,
+      "local_mtime_ns": 1792179517836408900
+    },
+    {
+      "name": "a-m-team__AM-DeepSeek-R1-Distilled-1.4M__am_0.9M.parquet",
+      "bytes": 2735935999,
+      "sha256": "53a1b649ac99423895aead66c0afd991000e8d7700b1304661fce63432afb310",
+      "rows": 816121,
+      "row_groups": 164,
+      "source": "AM-1.4M",
+      "upstream_repo": "a-m-team/AM-DeepSeek-R1-Distilled-1.4M",
+      "source_revision": null,
+      "local_mtime_ns": 1792179560690975200
+    },
+    {
+      "name": "AI-MO__NuminaMath-CoT__data__train-00000-of-00005.parquet",
+      "bytes": 86185406,
+      "sha256": "a1ee43e9d2df90d79ad7344df405ba39f7f3110152b165bcfab77904d134185e",
+      "rows": 165152,
+      "row_groups": 34,
+      "source": "NuminaMath-CoT",
+      "upstream_repo": "AI-MO/NuminaMath-CoT",
+      "source_revision": null,
+      "local_mtime_ns": 1792179226881037700
+    },
+    {
+      "name": "AI-MO__NuminaMath-CoT__data__train-00001-of-00005.parquet",
+      "bytes": 86246979,
+      "sha256": "15cb4666fe2e3d0dc72bab84841c3a844847ce966896c111cacc1565e26c8ebd",
+      "rows": 165106,
+      "row_groups": 34,
+      "source": "NuminaMath-CoT",
+      "upstream_repo": "AI-MO/NuminaMath-CoT",
+      "source_revision": null,
+      "local_mtime_ns": 1792179226796229900
+    },
+    {
+      "name": "AI-MO__NuminaMath-CoT__data__train-00002-of-00005.parquet",
+      "bytes": 86229816,
+      "sha256": "c6ab04bb589dd8951b97517b10942bc9f063c507a51b1acd3db2b308dcd9208e",
+      "rows": 165099,
+      "row_groups": 34,
+      "source": "NuminaMath-CoT",
+      "upstream_repo": "AI-MO/NuminaMath-CoT",
+      "source_revision": null,
+      "local_mtime_ns": 1792179226681556800
+    },
+    {
+      "name": "AI-MO__NuminaMath-CoT__data__train-00003-of-00005.parquet",
+      "bytes": 86265193,
+      "sha256": "646fdb5f92ab485be3b46552ff06ed269ecab205bbca9ab33b7c0e680de5abf6",
+      "rows": 165098,
+      "row_groups": 34,
+      "source": "NuminaMath-CoT",
+      "upstream_repo": "AI-MO/NuminaMath-CoT",
+      "source_revision": null,
+      "local_mtime_ns": 1792179226750898000
+    },
+    {
+      "name": "AI-MO__NuminaMath-CoT__data__train-00004-of-00005.parquet",
+      "bytes": 86326134,
+      "sha256": "11661222639b501e78bd4a38b1eb2dbe5c86f1b70c635e24d1bd3be5a8513b54",
+      "rows": 165190,
+      "row_groups": 34,
+      "source": "NuminaMath-CoT",
+      "upstream_repo": "AI-MO/NuminaMath-CoT",
+      "source_revision": null,
+      "local_mtime_ns": 1792179226914385500
+    },
+    {
+      "name": "bespokelabs__Bespoke-Stratos-17k__data__train-00000-of-00001.parquet",
+      "bytes": 85955414,
+      "sha256": "05a1b4c47144bbcab0523e6c323a57099785683691ec828688150f66b186962a",
+      "rows": 16579,
+      "row_groups": 4,
+      "source": "Bespoke-Stratos-17k",
+      "upstream_repo": "bespokelabs/Bespoke-Stratos-17k",
+      "source_revision": null,
+      "local_mtime_ns": 1792179226025323500
+    },
+    {
+      "name": "GAIR__LIMO__limo.parquet",
+      "bytes": 4606559,
+      "sha256": "b653af3463cf62590d430b298479766975ee7c630d6c179e4967f7644e142ba1",
+      "rows": 754,
+      "row_groups": 1,
+      "source": "LIMO",
+      "upstream_repo": "GAIR/LIMO",
+      "source_revision": null,
+      "local_mtime_ns": 1792179296895342200
+    },
+    {
+      "name": "meta-math__MetaMathQA__MetaMathQA-395K.parquet",
+      "bytes": 96708013,
+      "sha256": "9ae5412e91f0cded00db934e9cf5c68874159209e226be82413200f3d85264bb",
+      "rows": 394927,
+      "row_groups": 79,
+      "source": "MetaMathQA",
+      "upstream_repo": "meta-math/MetaMathQA",
+      "source_revision": null,
+      "local_mtime_ns": 1792179310870038900
+    },
+    {
+      "name": "microsoft__orca-math-word-problems-200k__data__train-00000-of-00001.parquet",
+      "bytes": 56980871,
+      "sha256": "356d2c37588bd96964938d56fa59a939fe5e3b1e9adb60499991e4ecdb4171fc",
+      "rows": 199991,
+      "row_groups": 40,
+      "source": "orca-math-200k",
+      "upstream_repo": "microsoft/orca-math-word-problems-200k",
+      "source_revision": null,
+      "local_mtime_ns": 1792179241506812500
+    },
+    {
+      "name": "NovaSky-AI__Sky-T1_data_17k__Sky-T1_data_17k.parquet",
+      "bytes": 61195180,
+      "sha256": "dc7fcdd8949568d6c91b1d9b1b9eb6987b1cd0735fcb48a65b8ec399ddb0583f",
+      "rows": 15973,
+      "row_groups": 4,
+      "source": "Sky-T1_data_17k",
+      "upstream_repo": "NovaSky-AI/Sky-T1_data_17k",
+      "source_revision": null,
+      "local_mtime_ns": 1792179241321241500
+    },
+    {
+      "name": "nvidia__Llama-Nemotron-Post-Training-Dataset__SFT__chat__chat.parquet",
+      "bytes": 44238839,
+      "sha256": "9f2d8bb887a1669cfcaf865582a09b8b5e0b44ef387afb865afb215beae3f2a4",
+      "rows": 23393,
+      "row_groups": 5,
+      "source": "Nemotron",
+      "upstream_repo": "nvidia/Llama-Nemotron-Post-Training-Dataset",
+      "source_revision": null,
+      "local_mtime_ns": 1792179303051307700
+    },
+    {
+      "name": "nvidia__Llama-Nemotron-Post-Training-Dataset__SFT__safety__safety.parquet",
+      "bytes": 7167022,
+      "sha256": "84ad3790526eda3437143b1e51fb54ba2f5700040121f829538186d37f5a22c4",
+      "rows": 31412,
+      "row_groups": 7,
+      "source": "Nemotron",
+      "upstream_repo": "nvidia/Llama-Nemotron-Post-Training-Dataset",
+      "source_revision": null,
+      "local_mtime_ns": 1792179300507782500
+    },
+    {
+      "name": "nvidia__Llama-Nemotron-Post-Training-Dataset__SFT__science__science.parquet",
+      "bytes": 1862528028,
+      "sha256": "b9b2e317a2ce29705dd347bc818aea5265922f11b70988b4412e75ac4d750bef",
+      "rows": 704108,
+      "row_groups": 141,
+      "source": "Nemotron",
+      "upstream_repo": "nvidia/Llama-Nemotron-Post-Training-Dataset",
+      "source_revision": null,
+      "local_mtime_ns": 1792179469100922000
+    },
+    {
+      "name": "open-r1__Mixture-of-Thoughts__code__train-00000-of-00008.parquet",
+      "bytes": 130484469,
+      "sha256": "cf59c161afd013a5496225b656726ab4a5632f12ba2262f94c30113ce3142593",
+      "rows": 10091,
+      "row_groups": 3,
+      "source": "Mixture-of-Thoughts",
+      "upstream_repo": "open-r1/Mixture-of-Thoughts",
+      "source_revision": null,
+      "local_mtime_ns": 1792179251085966400
+    },
+    {
+      "name": "open-r1__Mixture-of-Thoughts__code__train-00001-of-00008.parquet",
+      "bytes": 129342913,
+      "sha256": "275023de0ecb5cc6f6baa10f208425de07b11e0ff3a6b5b75e7e89dd236faa47",
+      "rows": 10091,
+      "row_groups": 3,
+      "source": "Mixture-of-Thoughts",
+      "upstream_repo": "open-r1/Mixture-of-Thoughts",
+      "source_revision": null,
+      "local_mtime_ns": 1792179250936618400
+    },
+    {
+      "name": "open-r1__Mixture-of-Thoughts__code__train-00002-of-00008.parquet",
+      "bytes": 129941998,
+      "sha256": "55338acef313d01209e3278bb74153cdb0be865efec47d29c416de8e61515160",
+      "rows": 10076,
+      "row_groups": 3,
+      "source": "Mixture-of-Thoughts",
+      "upstream_repo": "open-r1/Mixture-of-Thoughts",
+      "source_revision": null,
+      "local_mtime_ns": 1792179250970499900
+    },
+    {
+      "name": "open-r1__Mixture-of-Thoughts__code__train-00003-of-00008.parquet",
+      "bytes": 130524546,
+      "sha256": "a3897c6214ac11d60f09ea1196334873299cb7ad0c7fc9800073faa324f86eb1",
+      "rows": 10079,
+      "row_groups": 3,
+      "source": "Mixture-of-Thoughts",
+      "upstream_repo": "open-r1/Mixture-of-Thoughts",
+      "source_revision": null,
+      "local_mtime_ns": 1792179255741680900
+    },
+    {
+      "name": "open-r1__Mixture-of-Thoughts__code__train-00004-of-00008.parquet",
+      "bytes": 130039578,
+      "sha256": "8021dbdcee95ea7043d432bcb7d27b6d8522de243e864e4c5dc93041a48000ff",
+      "rows": 10089,
+      "row_groups": 3,
+      "source": "Mixture-of-Thoughts",
+      "upstream_repo": "open-r1/Mixture-of-Thoughts",
+      "source_revision": null,
+      "local_mtime_ns": 1792179256157279000
+    },
+    {
+      "name": "open-r1__Mixture-of-Thoughts__code__train-00005-of-00008.parquet",
+      "bytes": 129659503,
+      "sha256": "956a9f30712ab7f9378c27edc810b7411e49310897e9aeec73b885540d7a5ab1",
+      "rows": 10051,
+      "row_groups": 3,
+      "source": "Mixture-of-Thoughts",
+      "upstream_repo": "open-r1/Mixture-of-Thoughts",
+      "source_revision": null,
+      "local_mtime_ns": 1792179255734478400
+    },
+    {
+      "name": "open-r1__Mixture-of-Thoughts__code__train-00006-of-00008.parquet",
+      "bytes": 131547090,
+      "sha256": "4388b8c0681fa1ee54191cfade9b953f5cf649dec8ddedacd3cf2f3fc1c0328c",
+      "rows": 10094,
+      "row_groups": 3,
+      "source": "Mixture-of-Thoughts",
+      "upstream_repo": "open-r1/Mixture-of-Thoughts",
+      "source_revision": null,
+      "local_mtime_ns": 1792179265368315000
+    },
+    {
+      "name": "open-r1__Mixture-of-Thoughts__code__train-00007-of-00008.parquet",
+      "bytes": 130266781,
+      "sha256": "ebc243dd9c5cd550c2a3bd3d9d00bec83b2358083d5f68ec4329e2d3ae4bdcfa",
+      "rows": 10087,
+      "row_groups": 3,
+      "source": "Mixture-of-Thoughts",
+      "upstream_repo": "open-r1/Mixture-of-Thoughts",
+      "source_revision": null,
+      "local_mtime_ns": 1792179265738333300
+    },
+    {
+      "name": "open-r1__Mixture-of-Thoughts__math__train-00000-of-00004.parquet",
+      "bytes": 121498456,
+      "sha256": "f6054d47efc95a244c98ee735d771c41e7b10074b354a5e85b678d914e5194fc",
+      "rows": 22802,
+      "row_groups": 5,
+      "source": "Mixture-of-Thoughts",
+      "upstream_repo": "open-r1/Mixture-of-Thoughts",
+      "source_revision": null,
+      "local_mtime_ns": 1792179262122759600
+    },
+    {
+      "name": "open-r1__Mixture-of-Thoughts__math__train-00001-of-00004.parquet",
+      "bytes": 122068555,
+      "sha256": "37a1fd8534a1cf2009505772f8e15f2ef72cb4f5452eaa32088cae6021fe7e25",
+      "rows": 22795,
+      "row_groups": 5,
+      "source": "Mixture-of-Thoughts",
+      "upstream_repo": "open-r1/Mixture-of-Thoughts",
+      "source_revision": null,
+      "local_mtime_ns": 1792179267334320900
+    },
+    {
+      "name": "open-r1__Mixture-of-Thoughts__math__train-00002-of-00004.parquet",
+      "bytes": 121743291,
+      "sha256": "1fd3a90977a28e46ac2a88577b77ec81e79d70d19d80178403dade2c2653c1b3",
+      "rows": 22742,
+      "row_groups": 5,
+      "source": "Mixture-of-Thoughts",
+      "upstream_repo": "open-r1/Mixture-of-Thoughts",
+      "source_revision": null,
+      "local_mtime_ns": 1792179267230438600
+    },
+    {
+      "name": "open-r1__Mixture-of-Thoughts__math__train-00003-of-00004.parquet",
+      "bytes": 121469741,
+      "sha256": "a2e4d72438dcd2a6994f582720ce3b62b0a3e07f1671cced80b7f2f6e497ec7f",
+      "rows": 22755,
+      "row_groups": 5,
+      "source": "Mixture-of-Thoughts",
+      "upstream_repo": "open-r1/Mixture-of-Thoughts",
+      "source_revision": null,
+      "local_mtime_ns": 1792179267414247700
+    },
+    {
+      "name": "open-r1__Mixture-of-Thoughts__science__train-00000-of-00004.parquet",
+      "bytes": 101082008,
+      "sha256": "4a5830ab79ed9e0edc28a9e99bc9d425e70b531fe200720a01589b71a72f0de0",
+      "rows": 42688,
+      "row_groups": 9,
+      "source": "Mixture-of-Thoughts",
+      "upstream_repo": "open-r1/Mixture-of-Thoughts",
+      "source_revision": null,
+      "local_mtime_ns": 1792179272541580900
+    },
+    {
+      "name": "open-r1__Mixture-of-Thoughts__science__train-00001-of-00004.parquet",
+      "bytes": 109398315,
+      "sha256": "7e2c4754a8064acd210a5fdbd7880198a6b6eb286a1f4921cf40993f6e7d9ea7",
+      "rows": 42667,
+      "row_groups": 9,
+      "source": "Mixture-of-Thoughts",
+      "upstream_repo": "open-r1/Mixture-of-Thoughts",
+      "source_revision": null,
+      "local_mtime_ns": 1792179277398023400
+    },
+    {
+      "name": "open-r1__Mixture-of-Thoughts__science__train-00002-of-00004.parquet",
+      "bytes": 102389477,
+      "sha256": "69447d9d719fd9803fa619a8d4d495033a80f553c3cdffa4f218d8e013c8d09b",
+      "rows": 42650,
+      "row_groups": 9,
+      "source": "Mixture-of-Thoughts",
+      "upstream_repo": "open-r1/Mixture-of-Thoughts",
+      "source_revision": null,
+      "local_mtime_ns": 1792179276509509400
+    },
+    {
+      "name": "open-r1__Mixture-of-Thoughts__science__train-00003-of-00004.parquet",
+      "bytes": 131262222,
+      "sha256": "f3f3aab7e174dbc9c46e1286043a2bc68328c62b35a50c3bc2e94e4cbebbf2f7",
+      "rows": 42391,
+      "row_groups": 9,
+      "source": "Mixture-of-Thoughts",
+      "upstream_repo": "open-r1/Mixture-of-Thoughts",
+      "source_revision": null,
+      "local_mtime_ns": 1792179281800712000
+    },
+    {
+      "name": "open-r1__OpenR1-Math-220k__data__train-00000-of-00010.parquet",
+      "bytes": 48459312,
+      "sha256": "5d313263c3b3038b5e3e95b3ee3e9c03d75922d67333b99e700c7e537bb5972d",
+      "rows": 9125,
+      "row_groups": 2,
+      "source": "OpenR1-Math-220k",
+      "upstream_repo": "open-r1/OpenR1-Math-220k",
+      "source_revision": null,
+      "local_mtime_ns": 1792179273185211800
+    },
+    {
+      "name": "open-r1__OpenR1-Math-220k__data__train-00001-of-00010.parquet",
+      "bytes": 48640554,
+      "sha256": "5a0df64143b3a40d4b371018737b396fb0d73ddc36ff63bffe745154bd11cede",
+      "rows": 9117,
+      "row_groups": 2,
+      "source": "OpenR1-Math-220k",
+      "upstream_repo": "open-r1/OpenR1-Math-220k",
+      "source_revision": null,
+      "local_mtime_ns": 1792179273557931600
+    },
+    {
+      "name": "open-r1__OpenR1-Math-220k__data__train-00002-of-00010.parquet",
+      "bytes": 48599671,
+      "sha256": "4424c783a0cee834207ab72733dc463a259a62949617ad8fd0a7c706696df5b9",
+      "rows": 9111,
+      "row_groups": 2,
+      "source": "OpenR1-Math-220k",
+      "upstream_repo": "open-r1/OpenR1-Math-220k",
+      "source_revision": null,
+      "local_mtime_ns": 1792179278732147900
+    },
+    {
+      "name": "open-r1__OpenR1-Math-220k__data__train-00003-of-00010.parquet",
+      "bytes": 49260457,
+      "sha256": "bfad0ae409e4e31be3fb917b3d71da347fb0215c8b4eaf71b9e9679d09e8d323",
+      "rows": 9135,
+      "row_groups": 2,
+      "source": "OpenR1-Math-220k",
+      "upstream_repo": "open-r1/OpenR1-Math-220k",
+      "source_revision": null,
+      "local_mtime_ns": 1792179279574156000
+    },
+    {
+      "name": "open-r1__OpenR1-Math-220k__data__train-00004-of-00010.parquet",
+      "bytes": 48643956,
+      "sha256": "826c15fa875442ec747298bf977aa521d50604212d30b478b48b7b8f012af8aa",
+      "rows": 9110,
+      "row_groups": 2,
+      "source": "OpenR1-Math-220k",
+      "upstream_repo": "open-r1/OpenR1-Math-220k",
+      "source_revision": null,
+      "local_mtime_ns": 1792179280029681400
+    },
+    {
+      "name": "open-r1__OpenR1-Math-220k__data__train-00005-of-00010.parquet",
+      "bytes": 48256536,
+      "sha256": "8c96915ea8a09e9802f89e961c4b83afa052dee5f0844d38f465ef58f6a2fc2b",
+      "rows": 9078,
+      "row_groups": 2,
+      "source": "OpenR1-Math-220k",
+      "upstream_repo": "open-r1/OpenR1-Math-220k",
+      "source_revision": null,
+      "local_mtime_ns": 1792179283425246100
+    },
+    {
+      "name": "open-r1__OpenR1-Math-220k__data__train-00006-of-00010.parquet",
+      "bytes": 48666886,
+      "sha256": "d401dd8c51010c5c559dfd29c5495a8fc38f63c030692dd1176cf95f83803e9e",
+      "rows": 9095,
+      "row_groups": 2,
+      "source": "OpenR1-Math-220k",
+      "upstream_repo": "open-r1/OpenR1-Math-220k",
+      "source_revision": null,
+      "local_mtime_ns": 1792179283876727300
+    },
+    {
+      "name": "open-r1__OpenR1-Math-220k__data__train-00007-of-00010.parquet",
+      "bytes": 49205331,
+      "sha256": "b420bb61f316f2d2212a6efb9c3731cf7ca335297a71708f50e8af0289af5628",
+      "rows": 9123,
+      "row_groups": 2,
+      "source": "OpenR1-Math-220k",
+      "upstream_repo": "open-r1/OpenR1-Math-220k",
+      "source_revision": null,
+      "local_mtime_ns": 1792179285807266400
+    },
+    {
+      "name": "open-r1__OpenR1-Math-220k__data__train-00008-of-00010.parquet",
+      "bytes": 48483061,
+      "sha256": "d61e1b210255a1a27a7eb86078b94865251a51e630df9f2af066265de33cd23a",
+      "rows": 9096,
+      "row_groups": 2,
+      "source": "OpenR1-Math-220k",
+      "upstream_repo": "open-r1/OpenR1-Math-220k",
+      "source_revision": null,
+      "local_mtime_ns": 1792179286894682000
+    },
+    {
+      "name": "open-r1__OpenR1-Math-220k__data__train-00009-of-00010.parquet",
+      "bytes": 48600348,
+      "sha256": "ee65e9cbbee24953135a2e3c3a750774178c4c1110c58b565ceca5a1e252d5bd",
+      "rows": 9104,
+      "row_groups": 2,
+      "source": "OpenR1-Math-220k",
+      "upstream_repo": "open-r1/OpenR1-Math-220k",
+      "source_revision": null,
+      "local_mtime_ns": 1792179287800430900
+    },
+    {
+      "name": "open-r1__OpenR1-Math-220k__extended__train-00000-of-00010.parquet",
+      "bytes": 45624311,
+      "sha256": "29b116f942d64cf478c2c3a17092b567b1b0bee20acb0b748d546605673672dc",
+      "rows": 11966,
+      "row_groups": 3,
+      "source": "OpenR1-Math-220k",
+      "upstream_repo": "open-r1/OpenR1-Math-220k",
+      "source_revision": null,
+      "local_mtime_ns": 1792179289731997700
+    },
+    {
+      "name": "open-r1__OpenR1-Math-220k__extended__train-00001-of-00010.parquet",
+      "bytes": 44778143,
+      "sha256": "8e65d14a26241a348cd6f83e47ae170d4ad1735b5af868cabeadd4d648af8bf1",
+      "rows": 11891,
+      "row_groups": 3,
+      "source": "OpenR1-Math-220k",
+      "upstream_repo": "open-r1/OpenR1-Math-220k",
+      "source_revision": null,
+      "local_mtime_ns": 1792179292418517500
+    },
+    {
+      "name": "open-r1__OpenR1-Math-220k__extended__train-00002-of-00010.parquet",
+      "bytes": 45586923,
+      "sha256": "9e94c01c26dfe6cab045ac42751911d6d68d334e2ef24c4a506c052df6f066fd",
+      "rows": 11904,
+      "row_groups": 3,
+      "source": "OpenR1-Math-220k",
+      "upstream_repo": "open-r1/OpenR1-Math-220k",
+      "source_revision": null,
+      "local_mtime_ns": 1792179293025308000
+    },
+    {
+      "name": "open-r1__OpenR1-Math-220k__extended__train-00003-of-00010.parquet",
+      "bytes": 45098279,
+      "sha256": "4142bb4841ada267ad49627627066d5f87baf4099877572c2ee45289bfe735f2",
+      "rows": 11890,
+      "row_groups": 3,
+      "source": "OpenR1-Math-220k",
+      "upstream_repo": "open-r1/OpenR1-Math-220k",
+      "source_revision": null,
+      "local_mtime_ns": 1792179294520971000
+    },
+    {
+      "name": "open-r1__OpenR1-Math-220k__extended__train-00004-of-00010.parquet",
+      "bytes": 44827704,
+      "sha256": "d914fb20d45072157e16a2069e0552aea2581813b2f959cc4f56858195baf43b",
+      "rows": 11868,
+      "row_groups": 3,
+      "source": "OpenR1-Math-220k",
+      "upstream_repo": "open-r1/OpenR1-Math-220k",
+      "source_revision": null,
+      "local_mtime_ns": 1792179295627115600
+    },
+    {
+      "name": "open-r1__OpenR1-Math-220k__extended__train-00005-of-00010.parquet",
+      "bytes": 45120218,
+      "sha256": "ce6ce7934bebc56ba45a672c692cc78e374b8199c47053aa9f5374e0b1fec528",
+      "rows": 11926,
+      "row_groups": 3,
+      "source": "OpenR1-Math-220k",
+      "upstream_repo": "open-r1/OpenR1-Math-220k",
+      "source_revision": null,
+      "local_mtime_ns": 1792179296211397400
+    },
+    {
+      "name": "open-r1__OpenR1-Math-220k__extended__train-00006-of-00010.parquet",
+      "bytes": 45711196,
+      "sha256": "e10f3d4a67eddabd780e88ebbe6b17918aad55d69c2741bc496590ba1f5bd381",
+      "rows": 11922,
+      "row_groups": 3,
+      "source": "OpenR1-Math-220k",
+      "upstream_repo": "open-r1/OpenR1-Math-220k",
+      "source_revision": null,
+      "local_mtime_ns": 1792179298228976400
+    },
+    {
+      "name": "open-r1__OpenR1-Math-220k__extended__train-00007-of-00010.parquet",
+      "bytes": 45098823,
+      "sha256": "637d74c76e53333420d98ac3840dfbadcdfc788fe42c36860cd002f948b7f8f2",
+      "rows": 11923,
+      "row_groups": 3,
+      "source": "OpenR1-Math-220k",
+      "upstream_repo": "open-r1/OpenR1-Math-220k",
+      "source_revision": null,
+      "local_mtime_ns": 1792179300743713800
+    },
+    {
+      "name": "open-r1__OpenR1-Math-220k__extended__train-00008-of-00010.parquet",
+      "bytes": 45330457,
+      "sha256": "bb4f83e657d0201f63ca4e553902f518dbed10799940f1b310e33144be7c938b",
+      "rows": 11930,
+      "row_groups": 3,
+      "source": "OpenR1-Math-220k",
+      "upstream_repo": "open-r1/OpenR1-Math-220k",
+      "source_revision": null,
+      "local_mtime_ns": 1792179301352191900
+    },
+    {
+      "name": "open-r1__OpenR1-Math-220k__extended__train-00009-of-00010.parquet",
+      "bytes": 45235138,
+      "sha256": "2786bdeb8e1d961116f22a312f8e2dc0bdc4656981703612ced71910ce7a0c03",
+      "rows": 11870,
+      "row_groups": 3,
+      "source": "OpenR1-Math-220k",
+      "upstream_repo": "open-r1/OpenR1-Math-220k",
+      "source_revision": null,
+      "local_mtime_ns": 1792179303004954300
+    },
+    {
+      "name": "open-thoughts__OpenThoughts-114k__data__train-00000-of-00006.parquet",
+      "bytes": 168454614,
+      "sha256": "00c3125d640ab41134f685d0b7b5dcaab96de474f2529d3203e6c823e8d0700e",
+      "rows": 18767,
+      "row_groups": 4,
+      "source": "OpenThoughts-114k",
+      "upstream_repo": "open-thoughts/OpenThoughts-114k",
+      "source_revision": null,
+      "local_mtime_ns": 1792179241299786900
+    },
+    {
+      "name": "open-thoughts__OpenThoughts-114k__data__train-00001-of-00006.parquet",
+      "bytes": 121306899,
+      "sha256": "bccba810638fbf40119ca45560177660b58c364080b7ca24060cc379e7ce08a0",
+      "rows": 18825,
+      "row_groups": 4,
+      "source": "OpenThoughts-114k",
+      "upstream_repo": "open-thoughts/OpenThoughts-114k",
+      "source_revision": null,
+      "local_mtime_ns": 1792179237036803300
+    },
+    {
+      "name": "open-thoughts__OpenThoughts-114k__data__train-00002-of-00006.parquet",
+      "bytes": 120655869,
+      "sha256": "6b37ac8d10edc7f6d50a62cc320823a1a4f845840aa4e12e22d0bc590ff26c40",
+      "rows": 18872,
+      "row_groups": 4,
+      "source": "OpenThoughts-114k",
+      "upstream_repo": "open-thoughts/OpenThoughts-114k",
+      "source_revision": null,
+      "local_mtime_ns": 1792179236817163400
+    },
+    {
+      "name": "open-thoughts__OpenThoughts-114k__data__train-00003-of-00006.parquet",
+      "bytes": 121024024,
+      "sha256": "dc634769caf9895eadf7cf5c0968350041a92b45bc82e6c4f343843420b8b115",
+      "rows": 18853,
+      "row_groups": 4,
+      "source": "OpenThoughts-114k",
+      "upstream_repo": "open-thoughts/OpenThoughts-114k",
+      "source_revision": null,
+      "local_mtime_ns": 1792179237001078200
+    },
+    {
+      "name": "open-thoughts__OpenThoughts-114k__data__train-00004-of-00006.parquet",
+      "bytes": 107188827,
+      "sha256": "7f30ed9879bb072ced357da6b8b864f54dcc65d5ca14bee74fc68ca1365472ca",
+      "rows": 18862,
+      "row_groups": 4,
+      "source": "OpenThoughts-114k",
+      "upstream_repo": "open-thoughts/OpenThoughts-114k",
+      "source_revision": null,
+      "local_mtime_ns": 1792179235912275200
+    },
+    {
+      "name": "open-thoughts__OpenThoughts-114k__data__train-00005-of-00006.parquet",
+      "bytes": 106094368,
+      "sha256": "b29c9b29a735eeddcd5e55ad994288f0e2dc2d8b808706ff56caadd8c06cb362",
+      "rows": 18890,
+      "row_groups": 4,
+      "source": "OpenThoughts-114k",
+      "upstream_repo": "open-thoughts/OpenThoughts-114k",
+      "source_revision": null,
+      "local_mtime_ns": 1792179235500524100
+    },
+    {
+      "name": "openai__gsm8k__main__train-00000-of-00001.parquet",
+      "bytes": 1508440,
+      "sha256": "da01129a1927329991bb621b90ba7c234a995ac5a32b2433a6392951a4fcbc5e",
+      "rows": 7472,
+      "row_groups": 2,
+      "source": "gsm8k",
+      "upstream_repo": "openai/gsm8k",
+      "source_revision": null,
+      "local_mtime_ns": 1792179295925372200
+    },
+    {
+      "name": "openai__gsm8k__socratic__train-00000-of-00001.parquet",
+      "bytes": 1758743,
+      "sha256": "37b453fe22a8e960d875a559a41cb0902cea26ecfbf0453c649e902580001128",
+      "rows": 7472,
+      "row_groups": 2,
+      "source": "gsm8k",
+      "upstream_repo": "openai/gsm8k",
+      "source_revision": null,
+      "local_mtime_ns": 1792179296252780300
+    },
+    {
+      "name": "simplescaling__s1K-1.1__data__train-00000-of-00001.parquet",
+      "bytes": 5553728,
+      "sha256": "fca09bb59158b4b336d79142564d40a66f26d8e7dca991564de95442fb742224",
+      "rows": 687,
+      "row_groups": 1,
+      "source": "s1K-1.1",
+      "upstream_repo": "simplescaling/s1K-1.1",
+      "source_revision": null,
+      "local_mtime_ns": 1792179296724801800
+    },
+    {
+      "name": "simplescaling__s1K__data__train-00000-of-00001.parquet",
+      "bytes": 3875758,
+      "sha256": "39c984fffabf0cb8384cd21fcac7c39ed1994fdc53c2b69225a63d10373f3ece",
+      "rows": 993,
+      "row_groups": 1,
+      "source": "s1K",
+      "upstream_repo": "simplescaling/s1K",
+      "source_revision": null,
+      "local_mtime_ns": 1792179296522904300
+    }
+  ],
+  "excluded": []
+}

provenance/upstream_review_2026-10-18.json

@@ -0,0 +1,136 @@
+{
+  "reviewed_utc": "2026-10-18T03:45:33.503401+00:00",
+  "sources": [
+    {
+      "repo": "AI-MO/NuminaMath-CoT",
+      "url": "https://huggingface.co/datasets/AI-MO/NuminaMath-CoT",
+      "observed_revision": "9d8d210c9f6a36c8f3cd84045668c9b7800ef517",
+      "observed_license_metadata": "apache-2.0",
+      "historical_training_revision": null,
+      "role": "training"
+    },
+    {
+      "repo": "GAIR/LIMO",
+      "url": "https://huggingface.co/datasets/GAIR/LIMO",
+      "observed_revision": "b60f4462da9d927930b9c9bd43399cf875564416",
+      "observed_license_metadata": "apache-2.0",
+      "historical_training_revision": null,
+      "role": "training"
+    },
+    {
+      "repo": "NovaSky-AI/Sky-T1_data_17k",
+      "url": "https://huggingface.co/datasets/NovaSky-AI/Sky-T1_data_17k",
+      "observed_revision": "3e260822dae5d833d9b040e34265d5f9a2b8a6a5",
+      "observed_license_metadata": "apache-2.0",
+      "historical_training_revision": null,
+      "role": "training"
+    },
+    {
+      "repo": "a-m-team/AM-DeepSeek-R1-Distilled-1.4M",
+      "url": "https://huggingface.co/datasets/a-m-team/AM-DeepSeek-R1-Distilled-1.4M",
+      "observed_revision": "53531c06634904118a2dcd83961918c4d69d1cdf",
+      "observed_license_metadata": "cc-by-nc-4.0",
+      "historical_training_revision": null,
+      "role": "training"
+    },
+    {
+      "repo": "bespokelabs/Bespoke-Stratos-17k",
+      "url": "https://huggingface.co/datasets/bespokelabs/Bespoke-Stratos-17k",
+      "observed_revision": "9e9adba943911a9fc44dffcb30aaa18dc96ae6df",
+      "observed_license_metadata": "apache-2.0",
+      "historical_training_revision": null,
+      "role": "training"
+    },
+    {
+      "repo": "meta-math/MetaMathQA",
+      "url": "https://huggingface.co/datasets/meta-math/MetaMathQA",
+      "observed_revision": "aa4f34d3d2d3231299b5b03d9b3e5a20da45aa18",
+      "observed_license_metadata": "mit",
+      "historical_training_revision": null,
+      "role": "training"
+    },
+    {
+      "repo": "microsoft/orca-math-word-problems-200k",
+      "url": "https://huggingface.co/datasets/microsoft/orca-math-word-problems-200k",
+      "observed_revision": "29255d1770cc4eac66e5e7fa378cba542c026350",
+      "observed_license_metadata": "mit",
+      "historical_training_revision": null,
+      "role": "training"
+    },
+    {
+      "repo": "nvidia/Llama-Nemotron-Post-Training-Dataset",
+      "url": "https://huggingface.co/datasets/nvidia/Llama-Nemotron-Post-Training-Dataset",
+      "observed_revision": "ab2a40d258a6a4d9d4c277d702aeea445081766c",
+      "observed_license_metadata": "cc-by-4.0",
+      "historical_training_revision": null,
+      "role": "training"
+    },
+    {
+      "repo": "open-r1/Mixture-of-Thoughts",
+      "url": "https://huggingface.co/datasets/open-r1/Mixture-of-Thoughts",
+      "observed_revision": "e55fa28006c0d0ec60fb3547520f775dd42d02cd",
+      "observed_license_metadata": null,
+      "historical_training_revision": null,
+      "role": "training"
+    },
+    {
+      "repo": "open-r1/OpenR1-Math-220k",
+      "url": "https://huggingface.co/datasets/open-r1/OpenR1-Math-220k",
+      "observed_revision": "e4e141ec9dea9f8326f4d347be56105859b2bd68",
+      "observed_license_metadata": "apache-2.0",
+      "historical_training_revision": null,
+      "role": "training"
+    },
+    {
+      "repo": "open-thoughts/OpenThoughts-114k",
+      "url": "https://huggingface.co/datasets/open-thoughts/OpenThoughts-114k",
+      "observed_revision": "bd093c3994fd54d2390985b66988ddf282a55eb6",
+      "observed_license_metadata": "apache-2.0",
+      "historical_training_revision": null,
+      "role": "training"
+    },
+    {
+      "repo": "openai/gsm8k",
+      "url": "https://huggingface.co/datasets/openai/gsm8k",
+      "observed_revision": "740312add88f781978c0658806c59bc2815b9866",
+      "observed_license_metadata": [
+        "mit"
+      ],
+      "historical_training_revision": null,
+      "role": "training"
+    },
+    {
+      "repo": "simplescaling/s1K",
+      "url": "https://huggingface.co/datasets/simplescaling/s1K",
+      "observed_revision": "278d72baaa2b887a7e76a70a0ae254a5a45536e4",
+      "observed_license_metadata": "apache-2.0",
+      "historical_training_revision": null,
+      "role": "training"
+    },
+    {
+      "repo": "simplescaling/s1K-1.1",
+      "url": "https://huggingface.co/datasets/simplescaling/s1K-1.1",
+      "observed_revision": "96c411f1fe4c49d20f0e2a1565f61e1a28b0b84d",
+      "observed_license_metadata": "mit",
+      "historical_training_revision": null,
+      "role": "training"
+    },
+    {
+      "repo": "Rowan/hellaswag",
+      "url": "https://huggingface.co/datasets/Rowan/hellaswag",
+      "observed_revision": "218ec52e09a7e7462a5400043bb9a69a41d06b76",
+      "observed_license_metadata": null,
+      "historical_training_revision": "218ec52e09a7e7462a5400043bb9a69a41d06b76",
+      "role": "evaluation"
+    },
+    {
+      "repo": "open-r1/codeforces-cots",
+      "url": "https://huggingface.co/datasets/open-r1/codeforces-cots",
+      "observed_revision": "39ac85c150806230473c70ad72c31f6232fe3f41",
+      "observed_license_metadata": "cc-by-4.0",
+      "historical_training_revision": null,
+      "role": "component reference"
+    }
+  ],
+  "note": "Current observations are not the unknown historical revisions used to make local training files. Local hashes identify the actual training snapshot."
+}

release_files.json

@@ -0,0 +1,186 @@
+{
+  "files": {
+    "chat_template.jinja": {
+      "bytes": 193,
+      "sha256": "f950548988e50713a9975a11be0eaf1d29c4fe4be0ba22a57daeebbc27a7adb9"
+    },
+    "code/checkpoint_cloud.py": {
+      "bytes": 5284,
+      "sha256": "5699bd95426b574a44b49b939fd486fbb99f22e40be8fe70aea26272033860b2"
+    },
+    "code/common.py": {
+      "bytes": 2573,
+      "sha256": "1f849d002422505b3c5a529339beb2cdeafaebb3d7fca2eb8c8f1caa1fd673f1"
+    },
+    "code/inventory.py": {
+      "bytes": 2249,
+      "sha256": "8087293e92b19d9980453b545322d9c611f1d6a98944fd7326f205a87147015f"
+    },
+    "code/prepare.py": {
+      "bytes": 12594,
+      "sha256": "2157b8253106b06a377705c1c2e44c0122f14708bd4f41e766773c13f05c4ecf"
+    },
+    "code/requirements.txt": {
+      "bytes": 130,
+      "sha256": "88f019ca571ea919815d9463f2f709b7d42c9bbaa1e0ab23b90b054a3019b3f2"
+    },
+    "code/stage_tokens.py": {
+      "bytes": 2141,
+      "sha256": "40f26f9335612a3a60e8d7c23f66decb349a17cf5c9b6ef26c3302ba52c02b53"
+    },
+    "code/train.py": {
+      "bytes": 22213,
+      "sha256": "32da4946c75a99ac4fdc268e11db7c2ea1b5c2036dd02aa409d16de36ca35ea9"
+    },
+    "code/verify_original_workspace_hellaswag.py": {
+      "bytes": 7636,
+      "sha256": "6f9e32523277ed89e5b876fe5123e0012fca9ab43faf58ec865f68782558c52e"
+    },
+    "config.json": {
+      "bytes": 1060,
+      "sha256": "212efca58e24c38d1df22c8aa61eb6749b5e09040f3afc7166a7a91d6ddabace"
+    },
+    "evaluation/hellaswag/environment.txt": {
+      "bytes": 18374,
+      "sha256": "985d80012c8ffc6a8eb64f247f06112838b44788ab73b054a871ee1dbc2af760"
+    },
+    "evaluation/hellaswag/hellaswag_worker.py": {
+      "bytes": 8555,
+      "sha256": "597f59452a387041f6e0b457df5d79a104a3864edf502f963096d7aeee939138"
+    },
+    "evaluation/hellaswag/rank-0-audit.json": {
+      "bytes": 138,
+      "sha256": "cbde996d6e71f0e9b67692ba069428482bc130b3e73681a97244359475599ba8"
+    },
+    "evaluation/hellaswag/rank-1-audit.json": {
+      "bytes": 138,
+      "sha256": "b7c0a5cf66d64d3bf56d5bf99f23653d0808e286b837235e487dce8f94e63c00"
+    },
+    "evaluation/hellaswag/results.json": {
+      "bytes": 9185,
+      "sha256": "f87fa8a086b366164f2508ae8eec32128043dd15bbb132617c87ced82c81846d"
+    },
+    "evaluation/hellaswag/summary.json": {
+      "bytes": 2006,
+      "sha256": "197021e4284a188dce7a2015a4d9a424796ea67acf1309ee9810a8f0211b9081"
+    },
+    "evaluation/hellaswag/verification.json": {
+      "bytes": 2523,
+      "sha256": "842a391d972c3bfded5b068f82ee1d65b3077accbbc115e1faa14aa37410e40c"
+    },
+    "generation_config.json": {
+      "bytes": 154,
+      "sha256": "fdec449c49d8e35489371a2562aab44148a24fd37bfea45535a75205b132ca4b"
+    },
+    "LICENSE": {
+      "bytes": 2062,
+      "sha256": "d761e44f1c55b9e7c423dc89b28d0653691a7719db3989eaf42432db80f7da39"
+    },
+    "licenses/Apache-2.0.txt": {
+      "bytes": 11358,
+      "sha256": "cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30"
+    },
+    "model.safetensors": {
+      "bytes": 288017256,
+      "sha256": "0f7bdaaae88077174d371c650759b71b83fce6e917638703b72466d59a2b3c9e"
+    },
+    "NOTICE": {
+      "bytes": 150,
+      "sha256": "be23a0032f0a1908e8ea9857ae0deb223ff561aac41360381521256436a7a001"
+    },
+    "provenance/corpus_manifest.json": {
+      "bytes": 19049,
+      "sha256": "f32e5f3dab44177534cf7457176f83327a02a087c3db2367d771ef64adb2a380"
+    },
+    "provenance/observed_upstream_revisions_at_training.json": {
+      "bytes": 4450,
+      "sha256": "ed3adb3b9468da41fdcd5e1060ec113a00885c7d3193a1567b5aeee6367f897c"
+    },
+    "provenance/source_manifest.json": {
+      "bytes": 26264,
+      "sha256": "8457a299e8602fb874b51ffac7c5a93460ba93e26d5f616b40991d52d33a0923"
+    },
+    "provenance/upstream_review_2026-10-18.json": {
+      "bytes": 5493,
+      "sha256": "1bee9e1791e33dbbf35241330fa56c4e93a171d8c89d41ea627ae4d79c27db38"
+    },
+    "README.md": {
+      "bytes": 8352,
+      "sha256": "c66618af9b0e4fadd6d2befb74b62b2d2f14f0689cce46118df677eb53940090"
+    },
+    "release_identity.json": {
+      "bytes": 676,
+      "sha256": "3a1349b27da5d75c0a590b9b94aec34c4a11101a922f269197f5467a87093d6d"
+    },
+    "special_tokens_map.json": {
+      "bytes": 520,
+      "sha256": "7c1d1537a230168ef0f2294f7e9bf2324c65d625f110d0c55fe8bb195bb47de7"
+    },
+    "THIRD_PARTY.md": {
+      "bytes": 4674,
+      "sha256": "60035dfc763b7b5e454983ddbcd003574a7dbdc3160e9e5f35e7a72e75d78910"
+    },
+    "tokenizer.json": {
+      "bytes": 2321688,
+      "sha256": "27cb97f527f7eee80c7de32e04ba9cc32f8b8c2117d8664690443d53e228433b"
+    },
+    "tokenizer_config.json": {
+      "bytes": 1454,
+      "sha256": "370884fd87a82f9ed30dd8a5f31a403e15d75a5090cee14f1033ce6b39d6b47c"
+    },
+    "training/hardware.json": {
+      "bytes": 171,
+      "sha256": "db204f448f9d3781d88e3410918f595b4f81eb2efdd84c981dd7888db81f9cd7"
+    },
+    "training/initial_validation.json": {
+      "bytes": 104,
+      "sha256": "4abfc9b4b35c35aa0df26f9db54dd078b036dd7f33c5064db9305415d87e45b2"
+    },
+    "training/run_status.json": {
+      "bytes": 1757,
+      "sha256": "1c54d624506ea183872307296268638f041723c573006149536e952737d55512"
+    },
+    "training/sample_generations.json": {
+      "bytes": 1053,
+      "sha256": "05afb3539487113fec6a44fcf515439ea5037870de3a8f4bfc46e33fc68a71fe"
+    },
+    "training/session-v5/metrics.jsonl": {
+      "bytes": 294289,
+      "sha256": "9cd6fe336d7886b9054ba41b137f010db1522625f73f28441fa167ef1cade5b8"
+    },
+    "training/session-v5/run_status.json": {
+      "bytes": 369,
+      "sha256": "0aea31b12f4ab57c2a519f7e814a428c96135b0ea4940fa999490725831d3155"
+    },
+    "training/session-v6/metrics.jsonl": {
+      "bytes": 323455,
+      "sha256": "505a5cab22bb5305faa0b6e1b2f9da4f5981ddc93ec219c0fdc34f47db83503a"
+    },
+    "training/session-v6/run_status.json": {
+      "bytes": 1668,
+      "sha256": "03e365cfb915b30a403837ae58f9c75762f7bebe48ff5c2f59105777101a7fa5"
+    },
+    "training/session-v7/metrics.jsonl": {
+      "bytes": 4091,
+      "sha256": "7df246a3c00a08f594bd675b1f35a94ab8ac6951086348d63d453c35136ef36f"
+    },
+    "training/session-v7/run_status.json": {
+      "bytes": 1757,
+      "sha256": "1c54d624506ea183872307296268638f041723c573006149536e952737d55512"
+    },
+    "training/step-00022889.pt": {
+      "bytes": 864233185,
+      "sha256": "d3877e62f8b073f54163e40117feef22b2172587211f7fec22c56bdf830d2cce"
+    },
+    "training/training_settings.json": {
+      "bytes": 3239,
+      "sha256": "811498d4e2d909d7b18a6a45f8e544b7d26a579a2b25ae74cd30545d677df8d3"
+    },
+    "training/validation_latest.json": {
+      "bytes": 122,
+      "sha256": "86bc06d691a5de5fb8e06e3f509749b30798a3f1ab53d1ba32d7c54ce1617462"
+    }
+  },
+  "bytes": 1155385852,
+  "manifest_excludes_itself": true
+}

release_identity.json

@@ -0,0 +1,15 @@
+{
+  "name": "TinyThinkModel-72M-V1",
+  "model_repo": "sraivante/TinyThinkModel-72M-V1",
+  "dataset_repo": "sraivante/TinyThinkModel-72M-V1-Data",
+  "visibility": "private",
+  "version": "v1",
+  "dataset_revision": "9d8dd335c0c946ae84ff0919b09c1042da5315f7",
+  "model_step": 22889,
+  "training_tokens": 1500053504,
+  "model_sha256": "0f7bdaaae88077174d371c650759b71b83fce6e917638703b72466d59a2b3c9e",
+  "checkpoint_sha256": "d3877e62f8b073f54163e40117feef22b2172587211f7fec22c56bdf830d2cce",
+  "corpus_manifest_sha256": "f32e5f3dab44177534cf7457176f83327a02a087c3db2367d771ef64adb2a380",
+  "original_export_unchanged": true,
+  "public_release_approved": false
+}

special_tokens_map.json

@@ -0,0 +1,28 @@
+{
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<think>",
+    "</think>"
+  ],
+  "bos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|im_end|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|pad|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,68 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<|pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "<think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "5": {
+      "content": "</think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<think>",
+    "</think>"
+  ],
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "extra_special_tokens": {},
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 1024,
+  "pad_token": "<|pad|>",
+  "tokenizer_class": "PreTrainedTokenizerFast"
+}

training/hardware.json

@@ -0,0 +1,11 @@
+{
+  "gpu_names": [
+    "Tesla T4",
+    "Tesla T4"
+  ],
+  "world_size": 2,
+  "torch": "2.11.0+cu128",
+  "cuda": "12.8",
+  "parameters": 72000000,
+  "resumed_step": 22732
+}

training/initial_validation.json

@@ -0,0 +1,5 @@
+{
+  "loss": 1.4877843260765076,
+  "perplexity": 4.427275245445667,
+  "evaluated_target_tokens": 65536
+}

training/run_status.json

@@ -0,0 +1,53 @@
+{
+  "status": "COMPLETE",
+  "step": 22889,
+  "total_steps": 22889,
+  "tokens_seen": 1500053504,
+  "validation": {
+    "loss": 1.5027585998177528,
+    "perplexity": 4.494069325224148,
+    "evaluated_target_tokens": 262144
+  },
+  "heldout_test": {
+    "loss": 1.5159990563988686,
+    "perplexity": 4.5539685263096,
+    "evaluated_target_tokens": 262144
+  },
+  "ended_utc": "2026-10-18T02:35:55.174165+00:00",
+  "sample_generation_error": null,
+  "model_files": {
+    "tokenizer.json": {
+      "bytes": 2321688,
+      "sha256": "27cb97f527f7eee80c7de32e04ba9cc32f8b8c2117d8664690443d53e228433b"
+    },
+    "chat_template.jinja": {
+      "bytes": 193,
+      "sha256": "f950548988e50713a9975a11be0eaf1d29c4fe4be0ba22a57daeebbc27a7adb9"
+    },
+    "generation_config.json": {
+      "bytes": 154,
+      "sha256": "fdec449c49d8e35489371a2562aab44148a24fd37bfea45535a75205b132ca4b"
+    },
+    "config.json": {
+      "bytes": 1060,
+      "sha256": "212efca58e24c38d1df22c8aa61eb6749b5e09040f3afc7166a7a91d6ddabace"
+    },
+    "model.safetensors": {
+      "bytes": 288017256,
+      "sha256": "0f7bdaaae88077174d371c650759b71b83fce6e917638703b72466d59a2b3c9e"
+    },
+    "tokenizer_config.json": {
+      "bytes": 1454,
+      "sha256": "370884fd87a82f9ed30dd8a5f31a403e15d75a5090cee14f1033ce6b39d6b47c"
+    },
+    "README.md": {
+      "bytes": 2446,
+      "sha256": "74baafaae4c11d9dd26c0f9795e9f27b7df57a5bb5339accbbfb03ff013e6d93"
+    },
+    "special_tokens_map.json": {
+      "bytes": 520,
+      "sha256": "7c1d1537a230168ef0f2294f7e9bf2324c65d625f110d0c55fe8bb195bb47de7"
+    }
+  },
+  "limitations": "A scratch-trained 72M model with a 1024-token window; Qwen3 architecture does not imply Qwen3-0.6B reasoning ability. Task accuracy requires separate evaluation."
+}

training/sample_generations.json

@@ -0,0 +1,14 @@
+[
+  {
+    "prompt": "What is 12 plus 7?",
+    "response": "To add the numbers 12 and 7, you can add them together:\n\n12 + 7 = 19\n\nSo, the sum of the numbers 12 and 7 is $\\boxed{19}$ .<|im_end|>"
+  },
+  {
+    "prompt": "If a box has 3 rows of 4 apples, how many apples are there?",
+    "response": "If there are 3 rows of 4 apples, then there are a total of 3 * 4 = 12 apples.\nIf the box has 3 rows of 4 apples, then there are a total of 3 * 4 = 12 apples.\nTherefore, there are 12 apples in the box.\n#### 12\nThe answer is: 12<|im_end|>"
+  },
+  {
+    "prompt": "Write a Python function that returns the square of a number.",
+    "response": "<think>\nOkay, I need to write a Python function that takes a number and returns its square. Hmm, let's think about how to approach this.\n\nFirst, I should understand what the function is supposed to do. The function takes a number as input. So, for example, if the input is 5, the output should be 25. That makes sense.\n\nWait, but what if the number is zero? Oh right, zero squared is zero. So"
+  }
+]

training/session-v5/run_status.json

@@ -0,0 +1,13 @@
+{
+  "status": "RUNNING",
+  "step": 10850,
+  "total_steps": 22889,
+  "tokens_seen": 711065600,
+  "train_loss": 1.528669834136963,
+  "lr": 0.00035497581922166527,
+  "grad_norm": 0.37580370903015137,
+  "step_seconds": 3.4384259159996873,
+  "tokens_per_second": 18816.80632016959,
+  "gpu_memory_peak_gb": 10.644335616,
+  "updated_utc": "2026-10-17T07:00:15.582862+00:00"
+}

training/session-v6/run_status.json

@@ -0,0 +1,49 @@
+{
+  "status": "CHECKPOINTED_NEEDS_RESUME",
+  "step": 22732,
+  "total_steps": 22889,
+  "tokens_seen": 1489764352,
+  "validation": {
+    "loss": 1.5039765872061253,
+    "perplexity": 4.499546379799344,
+    "evaluated_target_tokens": 262144
+  },
+  "heldout_test": null,
+  "ended_utc": "2026-10-17T18:04:21.085022+00:00",
+  "sample_generation_error": null,
+  "model_files": {
+    "config.json": {
+      "bytes": 1060,
+      "sha256": "212efca58e24c38d1df22c8aa61eb6749b5e09040f3afc7166a7a91d6ddabace"
+    },
+    "chat_template.jinja": {
+      "bytes": 193,
+      "sha256": "f950548988e50713a9975a11be0eaf1d29c4fe4be0ba22a57daeebbc27a7adb9"
+    },
+    "tokenizer_config.json": {
+      "bytes": 1454,
+      "sha256": "370884fd87a82f9ed30dd8a5f31a403e15d75a5090cee14f1033ce6b39d6b47c"
+    },
+    "tokenizer.json": {
+      "bytes": 2321688,
+      "sha256": "27cb97f527f7eee80c7de32e04ba9cc32f8b8c2117d8664690443d53e228433b"
+    },
+    "README.md": {
+      "bytes": 2373,
+      "sha256": "0810ca4f9fb9f686df9f9ebdd027363582dc91eaa7783294a68b85dfb51d494c"
+    },
+    "model.safetensors": {
+      "bytes": 288017256,
+      "sha256": "75a165da792c2469eb15c5f3b7e92119c8167194d7fa2dea5050591334fae93e"
+    },
+    "generation_config.json": {
+      "bytes": 154,
+      "sha256": "fdec449c49d8e35489371a2562aab44148a24fd37bfea45535a75205b132ca4b"
+    },
+    "special_tokens_map.json": {
+      "bytes": 520,
+      "sha256": "7c1d1537a230168ef0f2294f7e9bf2324c65d625f110d0c55fe8bb195bb47de7"
+    }
+  },
+  "limitations": "A scratch-trained 72M model with a 1024-token window; Qwen3 architecture does not imply Qwen3-0.6B reasoning ability. Task accuracy requires separate evaluation."
+}

training/session-v7/metrics.jsonl

@@ -0,0 +1,15 @@
+{"step": 22740, "total_steps": 22889, "tokens_seen": 1490288640, "train_loss": 1.417860746383667, "lr": 6.005797727048432e-05, "grad_norm": 0.3760775029659271, "step_seconds": 3.0333797209999602, "tokens_per_second": 19462.975462665578, "gpu_memory_peak_gb": 10.64600832}
+{"step": 22750, "total_steps": 22889, "tokens_seen": 1490944000, "train_loss": 1.5366804599761963, "lr": 6.0050504877485626e-05, "grad_norm": 0.38432180881500244, "step_seconds": 3.174847647999968, "tokens_per_second": 20322.678995059436, "gpu_memory_peak_gb": 10.64600832}
+{"step": 22760, "total_steps": 22889, "tokens_seen": 1491599360, "train_loss": 1.5436077117919922, "lr": 6.004354775996215e-05, "grad_norm": 0.39511725306510925, "step_seconds": 3.351158197000018, "tokens_per_second": 18461.68501357058, "gpu_memory_peak_gb": 10.64600832}
+{"step": 22770, "total_steps": 22889, "tokens_seen": 1492254720, "train_loss": 1.4007065296173096, "lr": 6.00371059311936e-05, "grad_norm": 0.37356412410736084, "step_seconds": 3.2235344970000597, "tokens_per_second": 18850.293841915012, "gpu_memory_peak_gb": 10.64600832}
+{"step": 22780, "total_steps": 22889, "tokens_seen": 1492910080, "train_loss": 1.4809062480926514, "lr": 6.003117940347599e-05, "grad_norm": 0.37283048033714294, "step_seconds": 3.2179078230000187, "tokens_per_second": 19159.484160017095, "gpu_memory_peak_gb": 10.64600832}
+{"step": 22790, "total_steps": 22889, "tokens_seen": 1493565440, "train_loss": 1.4814839363098145, "lr": 6.002576818812182e-05, "grad_norm": 0.37576723098754883, "step_seconds": 3.2572511909999093, "tokens_per_second": 19336.0482469626, "gpu_memory_peak_gb": 10.64600832}
+{"step": 22800, "total_steps": 22889, "tokens_seen": 1494220800, "train_loss": 1.6184918880462646, "lr": 6.002087229545993e-05, "grad_norm": 0.3853263854980469, "step_seconds": 3.2472839629999726, "tokens_per_second": 19447.129263376064, "gpu_memory_peak_gb": 10.64600832}
+{"step": 22810, "total_steps": 22889, "tokens_seen": 1494876160, "train_loss": 1.4637168645858765, "lr": 6.0016491734835484e-05, "grad_norm": 0.38263821601867676, "step_seconds": 3.2260073190000185, "tokens_per_second": 19546.474681147753, "gpu_memory_peak_gb": 10.64600832}
+{"step": 22820, "total_steps": 22889, "tokens_seen": 1495531520, "train_loss": 1.5490052700042725, "lr": 6.001262651461004e-05, "grad_norm": 0.3839413821697235, "step_seconds": 3.239130419999924, "tokens_per_second": 19625.70504193735, "gpu_memory_peak_gb": 10.64600832}
+{"step": 22830, "total_steps": 22889, "tokens_seen": 1496186880, "train_loss": 1.4312634468078613, "lr": 6.000927664216153e-05, "grad_norm": 0.375712513923645, "step_seconds": 3.248629615999903, "tokens_per_second": 19683.34215651565, "gpu_memory_peak_gb": 10.64600832}
+{"step": 22840, "total_steps": 22889, "tokens_seen": 1496842240, "train_loss": 1.6019612550735474, "lr": 6.0006442123884074e-05, "grad_norm": 0.38849130272865295, "step_seconds": 3.245212473000038, "tokens_per_second": 19729.685663532204, "gpu_memory_peak_gb": 10.64600832}
+{"step": 22850, "total_steps": 22889, "tokens_seen": 1497497600, "train_loss": 1.519477367401123, "lr": 6.00041229651882e-05, "grad_norm": 0.38142576813697815, "step_seconds": 3.244884326000033, "tokens_per_second": 19769.090678051012, "gpu_memory_peak_gb": 10.64600832}
+{"step": 22860, "total_steps": 22889, "tokens_seen": 1498152960, "train_loss": 1.5155744552612305, "lr": 6.0002319170500677e-05, "grad_norm": 0.3878445029258728, "step_seconds": 3.2361329599999635, "tokens_per_second": 19802.659186977155, "gpu_memory_peak_gb": 10.64600832}
+{"step": 22870, "total_steps": 22889, "tokens_seen": 1498808320, "train_loss": 1.5018209218978882, "lr": 6.000103074326451e-05, "grad_norm": 0.39479777216911316, "step_seconds": 3.247613422000086, "tokens_per_second": 19830.083030614216, "gpu_memory_peak_gb": 10.64600832}
+{"step": 22880, "total_steps": 22889, "tokens_seen": 1499463680, "train_loss": 1.4517441987991333, "lr": 6.0000257685939096e-05, "grad_norm": 0.3645331561565399, "step_seconds": 3.239045901000054, "tokens_per_second": 19854.178908238107, "gpu_memory_peak_gb": 10.64600832}

training/session-v7/run_status.json

@@ -0,0 +1,53 @@
+{
+  "status": "COMPLETE",
+  "step": 22889,
+  "total_steps": 22889,
+  "tokens_seen": 1500053504,
+  "validation": {
+    "loss": 1.5027585998177528,
+    "perplexity": 4.494069325224148,
+    "evaluated_target_tokens": 262144
+  },
+  "heldout_test": {
+    "loss": 1.5159990563988686,
+    "perplexity": 4.5539685263096,
+    "evaluated_target_tokens": 262144
+  },
+  "ended_utc": "2026-10-18T02:35:55.174165+00:00",
+  "sample_generation_error": null,
+  "model_files": {
+    "tokenizer.json": {
+      "bytes": 2321688,
+      "sha256": "27cb97f527f7eee80c7de32e04ba9cc32f8b8c2117d8664690443d53e228433b"
+    },
+    "chat_template.jinja": {
+      "bytes": 193,
+      "sha256": "f950548988e50713a9975a11be0eaf1d29c4fe4be0ba22a57daeebbc27a7adb9"
+    },
+    "generation_config.json": {
+      "bytes": 154,
+      "sha256": "fdec449c49d8e35489371a2562aab44148a24fd37bfea45535a75205b132ca4b"
+    },
+    "config.json": {
+      "bytes": 1060,
+      "sha256": "212efca58e24c38d1df22c8aa61eb6749b5e09040f3afc7166a7a91d6ddabace"
+    },
+    "model.safetensors": {
+      "bytes": 288017256,
+      "sha256": "0f7bdaaae88077174d371c650759b71b83fce6e917638703b72466d59a2b3c9e"
+    },
+    "tokenizer_config.json": {
+      "bytes": 1454,
+      "sha256": "370884fd87a82f9ed30dd8a5f31a403e15d75a5090cee14f1033ce6b39d6b47c"
+    },
+    "README.md": {
+      "bytes": 2446,
+      "sha256": "74baafaae4c11d9dd26c0f9795e9f27b7df57a5bb5339accbbfb03ff013e6d93"
+    },
+    "special_tokens_map.json": {
+      "bytes": 520,
+      "sha256": "7c1d1537a230168ef0f2294f7e9bf2324c65d625f110d0c55fe8bb195bb47de7"
+    }
+  },
+  "limitations": "A scratch-trained 72M model with a 1024-token window; Qwen3 architecture does not imply Qwen3-0.6B reasoning ability. Task accuracy requires separate evaluation."
+}

training/step-00022889.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d3877e62f8b073f54163e40117feef22b2172587211f7fec22c56bdf830d2cce
+size 864233185

training/training_settings.json

@@ -0,0 +1,120 @@
+{
+  "parameters": 72000000,
+  "context": 1024,
+  "vocab": 33069,
+  "seed": 20261017,
+  "world_size": 2,
+  "batch_size": 8,
+  "grad_accum": 4,
+  "blocks_per_step": 64,
+  "tokens_per_step": 65536,
+  "target_tokens": 1500000000,
+  "total_steps": 22889,
+  "warmup_steps": 150,
+  "lr": 0.0006,
+  "min_lr": 6e-05,
+  "optimizer": "AdamW betas=(0.9,0.95) eps=1e-8 decay=0.1 matrices only",
+  "precision": "FP16 autocast, FP32 weights/optimizer, GradScaler",
+  "corpus_manifest_sha256": "f32e5f3dab44177534cf7457176f83327a02a087c3db2367d771ef64adb2a380",
+  "tokenizer_sha256": "27cb97f527f7eee80c7de32e04ba9cc32f8b8c2117d8664690443d53e228433b",
+  "model_config": {
+    "vocab_size": 33069,
+    "max_position_embeddings": 1024,
+    "hidden_size": 512,
+    "intermediate_size": 1536,
+    "num_hidden_layers": 14,
+    "num_attention_heads": 8,
+    "use_sliding_window": false,
+    "sliding_window": null,
+    "max_window_layers": 14,
+    "num_key_value_heads": 4,
+    "head_dim": 128,
+    "hidden_act": "silu",
+    "initializer_range": 0.02,
+    "rms_norm_eps": 1e-06,
+    "use_cache": false,
+    "rope_theta": 1000000.0,
+    "rope_scaling": null,
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "layer_types": [
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention"
+    ],
+    "return_dict": true,
+    "output_hidden_states": false,
+    "torchscript": false,
+    "dtype": null,
+    "pruned_heads": {},
+    "tie_word_embeddings": true,
+    "chunk_size_feed_forward": 0,
+    "is_encoder_decoder": false,
+    "is_decoder": false,
+    "cross_attention_hidden_size": null,
+    "add_cross_attention": false,
+    "tie_encoder_decoder": false,
+    "architectures": [
+      "Qwen3ForCausalLM"
+    ],
+    "finetuning_task": null,
+    "id2label": {
+      "0": "LABEL_0",
+      "1": "LABEL_1"
+    },
+    "label2id": {
+      "LABEL_0": 0,
+      "LABEL_1": 1
+    },
+    "task_specific_params": null,
+    "problem_type": null,
+    "tokenizer_class": null,
+    "prefix": null,
+    "bos_token_id": 1,
+    "pad_token_id": 0,
+    "eos_token_id": 2,
+    "sep_token_id": null,
+    "decoder_start_token_id": null,
+    "max_length": 20,
+    "min_length": 0,
+    "do_sample": false,
+    "early_stopping": false,
+    "num_beams": 1,
+    "num_beam_groups": 1,
+    "diversity_penalty": 0.0,
+    "temperature": 1.0,
+    "top_k": 50,
+    "top_p": 1.0,
+    "typical_p": 1.0,
+    "repetition_penalty": 1.0,
+    "length_penalty": 1.0,
+    "no_repeat_ngram_size": 0,
+    "encoder_no_repeat_ngram_size": 0,
+    "bad_words_ids": null,
+    "num_return_sequences": 1,
+    "output_scores": false,
+    "return_dict_in_generate": false,
+    "forced_bos_token_id": null,
+    "forced_eos_token_id": null,
+    "remove_invalid_values": false,
+    "exponential_decay_length_penalty": null,
+    "suppress_tokens": null,
+    "begin_suppress_tokens": null,
+    "_name_or_path": "",
+    "tf_legacy_loss": false,
+    "use_bfloat16": false,
+    "model_type": "qwen3",
+    "output_attentions": false
+  }
+}

training/validation_latest.json

@@ -0,0 +1,6 @@
+{
+  "step": 22889,
+  "loss": 1.5027585998177528,
+  "perplexity": 4.494069325224148,
+  "evaluated_target_tokens": 262144
+}