codemaivanngu commited on
Commit
cfd7f24
·
unverified ·
1 Parent(s): 0ca529a

docs(dev): document upstream provenance and B200 workflow

Browse files
Files changed (4) hide show
  1. .gitattributes +7 -0
  2. README.md +13 -3
  3. docs/RUNTIME_B200.md +23 -0
  4. docs/UPSTREAM.md +19 -0
.gitattributes ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ *.py text eol=lf
2
+ *.sh text eol=lf
3
+ *.md text eol=lf
4
+ *.json text eol=lf
5
+ *.toml text eol=lf
6
+ *.txt text eol=lf
7
+ .gitignore text eol=lf
README.md CHANGED
@@ -1,8 +1,18 @@
1
- # Cross-Tokenizer-OPD
 
 
 
 
 
 
 
 
 
 
2
 
3
  This repository provides the implementation and example training scripts for `byte_align_ctkd` and `topk_byte_align_ctkd`.
4
 
5
- ![Overview of cross-tokenizer alignment during training, the effect of mismatch supervision, and shared-vocabulary probability mass](assets/overview.png)
6
 
7
  ## Environment
8
 
@@ -32,7 +42,7 @@ Before running, update the paths in each script or mount the files at the expect
32
  - Training data: `/workspace/data/Train/math_code_20k/train.parquet`
33
  - Evaluation data: `/workspace/data/math500/test.jsonl`
34
 
35
- The data files are included under `data/`; model checkpoints must be provided separately. WandB logging is disabled by default (`--use_wandb False`). To enable it, set `--use_wandb True`, add `--wandb_project <project_name>`, and configure your WandB account.
36
 
37
  Run the byte-alignment example from the repository root:
38
 
 
1
+ # ctopd
2
+
3
+ Development repository for `sontungkieu`, imported from the authors' [Cross-Tokenizer-OPD code](https://anonymous.4open.science/r/Cross-Tokenizer-OPD/) for [arXiv:2610.08448v1](https://arxiv.org/abs/2610.08448v1). This is an independent working copy; the original authors retain authorship of the imported implementation.
4
+
5
+ The initial import preserves 84 source/configuration files. See [upstream provenance](docs/UPSTREAM.md) and the exact [file hashes](docs/upstream-source-manifest.json). Dataset and figure directories were excluded from this source snapshot. Model weights, datasets, runtime archives and experiment outputs belong outside Git.
6
+
7
+ Start editing directly in `kdflow/algorithms/`, `kdflow/token_alignment.py`, `kdflow/trainer/`, and `run/`. The two example launchers use **Qwen2.5-7B-Instruct -> Llama-3.2-3B-Instruct on 8 GPUs**; update paths and qualify the requested topology before training.
8
+
9
+ Use the existing [qualified B200 runtime on Hugging Face](https://huggingface.co/datasets/codemaivanngu/cross-tokenizer-opd-b200-runtime) with a source overlay rather than reinstalling dependencies after each code edit. See [B200 development instructions](docs/RUNTIME_B200.md).
10
+
11
+ ## Original project documentation
12
 
13
  This repository provides the implementation and example training scripts for `byte_align_ctkd` and `topk_byte_align_ctkd`.
14
 
15
+ The overview figure is available in the [original repository](https://anonymous.4open.science/r/Cross-Tokenizer-OPD/README.md); assets are not included in this source import.
16
 
17
  ## Environment
18
 
 
42
  - Training data: `/workspace/data/Train/math_code_20k/train.parquet`
43
  - Evaluation data: `/workspace/data/math500/test.jsonl`
44
 
45
+ The original repository provides data under `data/`; this source-only import excludes that directory. Prepare data and model checkpoints separately at the runtime host, then update the paths above. WandB logging is disabled by default (`--use_wandb False`). To enable it, set `--use_wandb True`, add `--wandb_project <project_name>`, and configure your WandB account.
46
 
47
  Run the byte-alignment example from the repository root:
48
 
docs/RUNTIME_B200.md ADDED
@@ -0,0 +1,23 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Develop against the existing B200 runtime
2
+
3
+ Runtime mirror: https://huggingface.co/datasets/codemaivanngu/cross-tokenizer-opd-b200-runtime
4
+
5
+ - Artifact revision: `6058422a8319cbe03958bf5d52750293e4e0da65`.
6
+ - Downloader revision: `93a118a96df88fe7dd0852b50d5b62616b701aa8`.
7
+ - Archive: `ctopd-b200-cu129-v4/runtime.tar.gz`, 16,515,065,867 bytes.
8
+ - SHA-256: `084c321c0b7af1cd3a36d12a595c623567d4a37525aa0eecbc3933aa7f9b15c3`.
9
+
10
+ Download/install directly on the target host using the pinned HF instructions. The archive requires Linux x86_64, glibc >= 2.39, g++ for JIT, and a B200 driver compatible with CUDA 12.9. Allow about 35 GiB free for parallel download/assembly or 60 GiB including installation. Model weights and datasets are separate.
11
+
12
+ Once installed, point `RUNTIME` at the actual extracted prefix and run source from this checkout:
13
+
14
+ ```bash
15
+ cd /absolute/path/to/ctopd
16
+ RUNTIME=/absolute/path/to/installed/ctopd-b200
17
+ export PYTHONPATH="$PWD${PYTHONPATH:+:$PYTHONPATH}"
18
+ bash "$RUNTIME/python-b200.sh" -m kdflow.cli.train_kd_on_policy --help
19
+ ```
20
+
21
+ The CLI imports CUDA dependencies and should be checked inside the allocated GPU environment. It is not a CPU-only test. Code/configuration edits do not require rebuilding dependencies; requalify runtime changes that affect CUDA, Torch, native extensions or ABI compatibility.
22
+
23
+ Before training, update the example scripts' source/model/data/output paths and actual GPU allocation. Their current hard-coded source prefix is `/workspace/Cross-Tokenizer-OPD`; change it to this checkout so they load your edits. The original scripts assume one node with eight GPUs. No training job is launched by these instructions.
docs/UPSTREAM.md ADDED
@@ -0,0 +1,19 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Upstream provenance
2
+
3
+ - Paper: https://arxiv.org/abs/2610.08448v1
4
+ - Authors' source: https://anonymous.4open.science/r/Cross-Tokenizer-OPD/
5
+ - Retrieval timestamp: 2026-10-17T19:28:35.738288+00:00
6
+ - Imported files: 84 (78 Python files), checked against SHA-256 before import.
7
+ - Snapshot manifest hash: `93cef1dfa18bcd5fc66cf80347557382efcd7e8955c3f6b464435711e82bf49b`.
8
+ - The source service did not expose an upstream Git commit. The local import commit starts a new history and does not claim to preserve upstream Git history.
9
+ - `docs/upstream-source-manifest.json` records original byte sizes and hashes. It describes the initial import; later development commits may intentionally change those files.
10
+
11
+ The initial commit preserves the imported implementation. Subsequent setup changes update documentation and line-ending attributes only. Datasets and figures were excluded from the source snapshot, and no runtime archives, model weights, credentials, account metadata or experiment outputs are included.
12
+
13
+ ## License metadata
14
+
15
+ The original `pyproject.toml` declares MIT and refers to `LICENSE`, `THIRD_PARTY_NOTICES.md`, and `LICENSES/*.txt`. Those files were absent from the retrieved source listing. This working copy preserves that metadata without inventing missing license text or changing third-party ownership. Consult the original authors for the missing license/notice files before making independent licensing claims.
16
+
17
+ ## Validation boundary
18
+
19
+ Initial import verification checks source hashes, Python syntax and Bash syntax only. This is not an end-to-end training run. The separate runtime qualification covers six synthetic probes on one Modal B200; it does not validate production training, company-node compatibility or multi-GPU behavior.