Instructions to use Shuibai12138/LLaDA-8B-MDLM-LoRA-OpenCodeInstruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use Shuibai12138/LLaDA-8B-MDLM-LoRA-OpenCodeInstruct with PEFT:
Task type is invalid.
- Notebooks
- Google Colab
- Kaggle
Model card: point to v1.0.3-corrective-training (pinned base-model revision)
Browse files
README.md
CHANGED
|
@@ -37,7 +37,7 @@ Code: [zhangshuibai/CDLM](https://github.com/zhangshuibai/CDLM).
|
|
| 37 |
| Precision | bf16 base weights, fp32 LoRA parameters and optimiser states, activation checkpointing |
|
| 38 |
| Seed | 42 |
|
| 39 |
| Hardware | 4 x A100-PCIE-40GB, 50.8 min |
|
| 40 |
-
| Code | `bash training/llada8b_lora/run_train_opencodeinstruct.sh mdlm` at commit `96906ed`; tag `v1.0.
|
| 41 |
|
| 42 |
The trainer's `DataLoader` runs two workers over an unsharded `IterableDataset`, so each chunk is used twice in a
|
| 43 |
row with independent noise; 2,000 steps see 12,000 distinct chunks. This is how every LLaDA-8B adapter of the
|
|
@@ -94,7 +94,7 @@ To reproduce the numbers above, use the code repository ([zhangshuibai/CDLM](htt
|
|
| 94 |
(`<owner>/<name>[/<subfolder>][@<revision>]`). The commands below pin the revision that holds these weights.
|
| 95 |
|
| 96 |
```bash
|
| 97 |
-
git clone --branch v1.0.
|
| 98 |
pip install -r training/llada8b_lora/requirements.txt
|
| 99 |
bash training/llada8b_lora/fetch_crb_inputs.sh # the 24 CRB input files of this experiment
|
| 100 |
# localisation (confidence gap and Top-K hit rate)
|
|
|
|
| 37 |
| Precision | bf16 base weights, fp32 LoRA parameters and optimiser states, activation checkpointing |
|
| 38 |
| Seed | 42 |
|
| 39 |
| Hardware | 4 x A100-PCIE-40GB, 50.8 min |
|
| 40 |
+
| Code | `bash training/llada8b_lora/run_train_opencodeinstruct.sh mdlm` at commit `96906ed`; tag `v1.0.3-corrective-training` has the same trainer (plus the pinned base-model revision, the one this run loaded) and `run_train.sh`, and its launcher passes them the same data and arguments (later commits added comments, a shards-per-rank check, rank and position columns in `data_shards.tsv` and stricter link handling) |
|
| 41 |
|
| 42 |
The trainer's `DataLoader` runs two workers over an unsharded `IterableDataset`, so each chunk is used twice in a
|
| 43 |
row with independent noise; 2,000 steps see 12,000 distinct chunks. This is how every LLaDA-8B adapter of the
|
|
|
|
| 94 |
(`<owner>/<name>[/<subfolder>][@<revision>]`). The commands below pin the revision that holds these weights.
|
| 95 |
|
| 96 |
```bash
|
| 97 |
+
git clone --branch v1.0.3-corrective-training https://github.com/zhangshuibai/CDLM && cd CDLM
|
| 98 |
pip install -r training/llada8b_lora/requirements.txt
|
| 99 |
bash training/llada8b_lora/fetch_crb_inputs.sh # the 24 CRB input files of this experiment
|
| 100 |
# localisation (confidence gap and Top-K hit rate)
|