docs(dev): document upstream provenance and B200 workflow
Browse files- .gitattributes +7 -0
- README.md +13 -3
- docs/RUNTIME_B200.md +23 -0
- docs/UPSTREAM.md +19 -0
.gitattributes
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
*.py text eol=lf
|
| 2 |
+
*.sh text eol=lf
|
| 3 |
+
*.md text eol=lf
|
| 4 |
+
*.json text eol=lf
|
| 5 |
+
*.toml text eol=lf
|
| 6 |
+
*.txt text eol=lf
|
| 7 |
+
.gitignore text eol=lf
|
README.md
CHANGED
|
@@ -1,8 +1,18 @@
|
|
| 1 |
-
#
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2 |
|
| 3 |
This repository provides the implementation and example training scripts for `byte_align_ctkd` and `topk_byte_align_ctkd`.
|
| 4 |
|
| 5 |
-
|
| 6 |
|
| 7 |
## Environment
|
| 8 |
|
|
@@ -32,7 +42,7 @@ Before running, update the paths in each script or mount the files at the expect
|
|
| 32 |
- Training data: `/workspace/data/Train/math_code_20k/train.parquet`
|
| 33 |
- Evaluation data: `/workspace/data/math500/test.jsonl`
|
| 34 |
|
| 35 |
-
The
|
| 36 |
|
| 37 |
Run the byte-alignment example from the repository root:
|
| 38 |
|
|
|
|
| 1 |
+
# ctopd
|
| 2 |
+
|
| 3 |
+
Development repository for `sontungkieu`, imported from the authors' [Cross-Tokenizer-OPD code](https://anonymous.4open.science/r/Cross-Tokenizer-OPD/) for [arXiv:2610.08448v1](https://arxiv.org/abs/2610.08448v1). This is an independent working copy; the original authors retain authorship of the imported implementation.
|
| 4 |
+
|
| 5 |
+
The initial import preserves 84 source/configuration files. See [upstream provenance](docs/UPSTREAM.md) and the exact [file hashes](docs/upstream-source-manifest.json). Dataset and figure directories were excluded from this source snapshot. Model weights, datasets, runtime archives and experiment outputs belong outside Git.
|
| 6 |
+
|
| 7 |
+
Start editing directly in `kdflow/algorithms/`, `kdflow/token_alignment.py`, `kdflow/trainer/`, and `run/`. The two example launchers use **Qwen2.5-7B-Instruct -> Llama-3.2-3B-Instruct on 8 GPUs**; update paths and qualify the requested topology before training.
|
| 8 |
+
|
| 9 |
+
Use the existing [qualified B200 runtime on Hugging Face](https://huggingface.co/datasets/codemaivanngu/cross-tokenizer-opd-b200-runtime) with a source overlay rather than reinstalling dependencies after each code edit. See [B200 development instructions](docs/RUNTIME_B200.md).
|
| 10 |
+
|
| 11 |
+
## Original project documentation
|
| 12 |
|
| 13 |
This repository provides the implementation and example training scripts for `byte_align_ctkd` and `topk_byte_align_ctkd`.
|
| 14 |
|
| 15 |
+
The overview figure is available in the [original repository](https://anonymous.4open.science/r/Cross-Tokenizer-OPD/README.md); assets are not included in this source import.
|
| 16 |
|
| 17 |
## Environment
|
| 18 |
|
|
|
|
| 42 |
- Training data: `/workspace/data/Train/math_code_20k/train.parquet`
|
| 43 |
- Evaluation data: `/workspace/data/math500/test.jsonl`
|
| 44 |
|
| 45 |
+
The original repository provides data under `data/`; this source-only import excludes that directory. Prepare data and model checkpoints separately at the runtime host, then update the paths above. WandB logging is disabled by default (`--use_wandb False`). To enable it, set `--use_wandb True`, add `--wandb_project <project_name>`, and configure your WandB account.
|
| 46 |
|
| 47 |
Run the byte-alignment example from the repository root:
|
| 48 |
|
docs/RUNTIME_B200.md
ADDED
|
@@ -0,0 +1,23 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Develop against the existing B200 runtime
|
| 2 |
+
|
| 3 |
+
Runtime mirror: https://huggingface.co/datasets/codemaivanngu/cross-tokenizer-opd-b200-runtime
|
| 4 |
+
|
| 5 |
+
- Artifact revision: `6058422a8319cbe03958bf5d52750293e4e0da65`.
|
| 6 |
+
- Downloader revision: `93a118a96df88fe7dd0852b50d5b62616b701aa8`.
|
| 7 |
+
- Archive: `ctopd-b200-cu129-v4/runtime.tar.gz`, 16,515,065,867 bytes.
|
| 8 |
+
- SHA-256: `084c321c0b7af1cd3a36d12a595c623567d4a37525aa0eecbc3933aa7f9b15c3`.
|
| 9 |
+
|
| 10 |
+
Download/install directly on the target host using the pinned HF instructions. The archive requires Linux x86_64, glibc >= 2.39, g++ for JIT, and a B200 driver compatible with CUDA 12.9. Allow about 35 GiB free for parallel download/assembly or 60 GiB including installation. Model weights and datasets are separate.
|
| 11 |
+
|
| 12 |
+
Once installed, point `RUNTIME` at the actual extracted prefix and run source from this checkout:
|
| 13 |
+
|
| 14 |
+
```bash
|
| 15 |
+
cd /absolute/path/to/ctopd
|
| 16 |
+
RUNTIME=/absolute/path/to/installed/ctopd-b200
|
| 17 |
+
export PYTHONPATH="$PWD${PYTHONPATH:+:$PYTHONPATH}"
|
| 18 |
+
bash "$RUNTIME/python-b200.sh" -m kdflow.cli.train_kd_on_policy --help
|
| 19 |
+
```
|
| 20 |
+
|
| 21 |
+
The CLI imports CUDA dependencies and should be checked inside the allocated GPU environment. It is not a CPU-only test. Code/configuration edits do not require rebuilding dependencies; requalify runtime changes that affect CUDA, Torch, native extensions or ABI compatibility.
|
| 22 |
+
|
| 23 |
+
Before training, update the example scripts' source/model/data/output paths and actual GPU allocation. Their current hard-coded source prefix is `/workspace/Cross-Tokenizer-OPD`; change it to this checkout so they load your edits. The original scripts assume one node with eight GPUs. No training job is launched by these instructions.
|
docs/UPSTREAM.md
ADDED
|
@@ -0,0 +1,19 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Upstream provenance
|
| 2 |
+
|
| 3 |
+
- Paper: https://arxiv.org/abs/2610.08448v1
|
| 4 |
+
- Authors' source: https://anonymous.4open.science/r/Cross-Tokenizer-OPD/
|
| 5 |
+
- Retrieval timestamp: 2026-10-17T19:28:35.738288+00:00
|
| 6 |
+
- Imported files: 84 (78 Python files), checked against SHA-256 before import.
|
| 7 |
+
- Snapshot manifest hash: `93cef1dfa18bcd5fc66cf80347557382efcd7e8955c3f6b464435711e82bf49b`.
|
| 8 |
+
- The source service did not expose an upstream Git commit. The local import commit starts a new history and does not claim to preserve upstream Git history.
|
| 9 |
+
- `docs/upstream-source-manifest.json` records original byte sizes and hashes. It describes the initial import; later development commits may intentionally change those files.
|
| 10 |
+
|
| 11 |
+
The initial commit preserves the imported implementation. Subsequent setup changes update documentation and line-ending attributes only. Datasets and figures were excluded from the source snapshot, and no runtime archives, model weights, credentials, account metadata or experiment outputs are included.
|
| 12 |
+
|
| 13 |
+
## License metadata
|
| 14 |
+
|
| 15 |
+
The original `pyproject.toml` declares MIT and refers to `LICENSE`, `THIRD_PARTY_NOTICES.md`, and `LICENSES/*.txt`. Those files were absent from the retrieved source listing. This working copy preserves that metadata without inventing missing license text or changing third-party ownership. Consult the original authors for the missing license/notice files before making independent licensing claims.
|
| 16 |
+
|
| 17 |
+
## Validation boundary
|
| 18 |
+
|
| 19 |
+
Initial import verification checks source hashes, Python syntax and Bash syntax only. This is not an end-to-end training run. The separate runtime qualification covers six synthetic probes on one Modal B200; it does not validate production training, company-node compatibility or multi-GPU behavior.
|