File size: 5,337 Bytes
7f87a1d 736c211 79b4d87 736c211 7f87a1d 494b24c 93f322e fb3fd2d b689df8 74e919a b689df8 fb3fd2d b689df8 f2c565a b689df8 f2c565a b7f5465 f2c565a b7f5465 f2c565a 7f87a1d 5c84315 494b24c | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 | # Rust DMHY Template Apply
Multi-core Rust implementation of the DMHY template recipe apply stage.
Build template recipes from the repository root:
```powershell
cargo run --release --manifest-path tools\rust_dmhy_template_apply\Cargo.toml -- `
--cluster `
--input datasets\AnimeName\dmhy_list.jsonl `
--summary-output reports\dmhy_template_clusters.full_top5000.summary.json `
--samples-output reports\dmhy_template_clusters.full_top5000.samples.jsonl `
--clusters-output reports\dmhy_template_clusters.full_top5000.jsonl `
--recipes-output reports\dmhy_template_recipes.full_top5000.seed.jsonl `
--review-output reports\dmhy_template_review.full_top5000.jsonl `
--top 5000 `
--recipe-top 5000 `
--review-top 5000 `
--min-count 2 `
--recipe-min-count 25 `
--threads 24
```
Apply template recipes from the repository root:
```powershell
cargo run --release --manifest-path tools\rust_dmhy_template_apply\Cargo.toml -- `
--input datasets\AnimeName\dmhy_list.jsonl `
--recipes reports\dmhy_template_recipes.full_top5000.seed.jsonl `
--output reports\dmhy_weak.template_generated.top5000.rust.jsonl `
--manifest-output reports\dmhy_weak.template_generated.top5000.rust.manifest.json
```
Audit low-frequency recipe output from the repository root:
```powershell
cargo run --release --manifest-path tools\rust_dmhy_template_apply\Cargo.toml -- `
--audit-low-frequency `
--input datasets\AnimeName\dmhy_list.jsonl `
--recipes reports\dmhy_template_recipes.full_top5000.seed.jsonl `
--audit-output reports\dmhy_low_frequency_audit.rust.jsonl `
--audit-max-count 50 `
--threads 24
```
Verify the generated training output has no low-frequency blocking warnings:
```powershell
cargo run --release --manifest-path tools\rust_dmhy_template_apply\Cargo.toml -- `
--verify-generated-output `
--input reports\dmhy_weak.template_generated.top5000.rust.jsonl `
--recipes reports\dmhy_template_recipes.full_top5000.seed.jsonl `
--audit-max-count 50
```
Review ambiguous title-boundary decisions with a local LM Studio model or with
Pi RPC mode. This is an optional final-pass reviewer: it writes a decision cache
JSONL and does not modify recipes or training data directly.
```powershell
cargo run --release --manifest-path tools\rust_dmhy_template_apply\Cargo.toml -- `
--review-title-boundaries-lmstudio `
--recipes reports\dmhy_template_recipes.full_top5000.seed.jsonl `
--title-boundary-decisions-output reports\dmhy_title_boundary_lmstudio_decisions.jsonl `
--lmstudio-base-url http://127.0.0.1:1234/v1 `
--lmstudio-model qwen `
--title-boundary-min-similarity 0.25 `
--title-boundary-max-similarity 0.85 `
--limit 50
```
```powershell
cargo run --release --manifest-path tools\rust_dmhy_template_apply\Cargo.toml -- `
--review-title-boundaries-pi-rpc `
--recipes reports\dmhy_template_recipes.full_top5000.seed.jsonl `
--title-boundary-decisions-output reports\dmhy_title_boundary_pi_rpc_decisions.jsonl `
--pi-rpc-model openai/gpt-5.5 `
--title-boundary-min-similarity 0.25 `
--title-boundary-max-similarity 0.85 `
--threads 4 `
--limit 50
```
For stratified audits, select heuristic strata explicitly:
```powershell
# High-similarity adjacent-title merges. The old high-similarity keep heuristic
# was removed after Pi RPC review found it over-preserved title continuations.
cargo run --release --manifest-path tools\rust_dmhy_template_apply\Cargo.toml -- `
--review-title-boundaries-pi-rpc `
--recipes reports\dmhy_template_recipes.full_top5000.seed.jsonl `
--title-boundary-decisions-output reports\dmhy_title_boundary_pi_rpc_adjacent_highsim.jsonl `
--title-boundary-heuristic-contains merge_adjacent_title_continuation `
--title-boundary-min-similarity 0.85 `
--title-boundary-max-similarity 1.0 `
--threads 4 `
--limit 20
# Structural keep candidates normally excluded from ambiguous merge review.
cargo run --release --manifest-path tools\rust_dmhy_template_apply\Cargo.toml -- `
--review-title-boundaries-pi-rpc `
--recipes reports\dmhy_template_recipes.full_top5000.seed.jsonl `
--title-boundary-decisions-output reports\dmhy_title_boundary_pi_rpc_structural_keep.jsonl `
--title-boundary-heuristic-contains keep_structural_boundary `
--title-boundary-min-similarity 0.0 `
--title-boundary-max-similarity 1.0 `
--threads 4 `
--limit 20
```
Optional controls:
```powershell
--threads 24
--limit 50000
--limit-templates 1000
--min-count 10
--confidence high
--expand sample --sample-per-template 100
--keep-encoding-noise
```
The output record schema is `filename`, `tokens`, `labels`, `template_id`, and
`template`, plus optional `source_filename`, `path_trimmed`, and
`dropped_title_candidate_positions`. Clustered recipe rows also include
`title_spans` and `title_boundary_decisions` metadata so downstream synthetic
augmentation can distinguish one logical title span from repeated/path title
slots.
For low-frequency templates (`count <= --audit-max-count`, default `50`), apply
uses a conservative gate: records with `no_title`, `multiple_title_spans`,
`path_retained`, or `hash_labeled` audit warnings are skipped from the training
JSONL and left in the audit/review files. This keeps common templates stable
while preventing rare ambiguous path/title cases from polluting the generated
dataset.
|