Simplify title-boundary title span heuristic

AGENTS.md

@@ -134,16 +134,18 @@ cargo run --release --manifest-path tools\rust_dmhy_template_apply\Cargo.toml --
 ```
 
 For reviewer calibration, run stratified samples before full review. Use
-`--title-boundary-heuristic-contains` to inspect high-similar duplicate/alias
-keeps, structural keeps, and path-boundary keeps that are normally skipped by
-the ambiguous merge review:
+`--title-boundary-heuristic-contains` to inspect high-similarity adjacent-title
+merges, structural keeps, and path-boundary keeps that are normally skipped by
+the ambiguous merge review. The old high-similarity duplicate/alias keep
+heuristic was removed after Pi RPC review found it over-preserved title
+continuations:
 
 ```powershell
 cargo run --release --manifest-path tools\rust_dmhy_template_apply\Cargo.toml -- `
   --review-title-boundaries-pi-rpc `
-  --recipes reports\dmhy_template_recipes.full_top5000.title_boundary_trial.seed.jsonl `
-  --title-boundary-decisions-output reports\dmhy_title_boundary_pi_rpc_highsim_keep_sample.jsonl `
-  --title-boundary-heuristic-contains keep_similar_duplicate_or_alias `
+  --recipes reports\dmhy_template_recipes.full_top5000.seed.jsonl `
+  --title-boundary-decisions-output reports\dmhy_title_boundary_pi_rpc_adjacent_highsim_sample.jsonl `
+  --title-boundary-heuristic-contains merge_adjacent_title_continuation `
   --title-boundary-min-similarity 0.85 `
   --title-boundary-max-similarity 1.0 `
   --threads 4 `

docs/title_boundary_heuristic_experiments.md

@@ -0,0 +1,177 @@
+# Title-Boundary Heuristic Experiments
+
+This note records deterministic Rust heuristic changes and Pi RPC reviewer evidence
+for DMHY template `title_spans` generation. The reviewer output is evidence only;
+recipes and training JSONL are changed only by Rust code.
+
+## Baseline
+
+Input recipe evidence:
+
+- `reports/dmhy_template_recipes.full_top5000.title_boundary_trial.seed.jsonl`
+- total title-boundary decisions: `5631`
+
+Decision distribution:
+
+| decision | count |
+| --- | ---: |
+| `merge_dissimilar_title_continuation` | 4661 |
+| `keep_similar_duplicate_or_alias` | 463 |
+| `keep_structural_boundary` | 502 |
+| `keep_path_boundary` | 5 |
+
+Pi RPC stratified sample, `--threads 4`:
+
+| stratum | rows | model result | note |
+| --- | ---: | --- | --- |
+| ambiguous merge, similarity `0.25..0.85` | 20 | 20 merge | Good. Reviewer cited concrete title-word pairings. |
+| high-sim `keep_similar_duplicate_or_alias`, similarity `0.85..1.0` | 20 | 19 merge, 1 parse-error | Bad baseline heuristic. Examples such as `Asobi ni Iku yo!`, `Code Geass ~Akito the Exiled~`, `Yagate Kimi ni Naru`, and `Big Order` were wrongly kept separate by the heuristic. |
+| structural keep | 20 | 18 keep, 1 merge, 1 skip | Mostly good. Covers category+title, multilingual aliases, and distinct fields. |
+| path keep | 5 | 5 keep | Good. Covers path duplicate/category boundaries. |
+
+Conclusion: `keep_similar_duplicate_or_alias` was too broad because it used max
+literal similarity. Short connector repeats (`no`, `wa`, `ni`, `the`, `of`) and
+repeated title words inflated similarity to `1.0` even for ordinary contiguous
+title phrases.
+
+## Experiment 1: strict substantive duplicate keep
+
+Change tried:
+
+- Keep adjacent `TITLE SEP TITLE` only when top literals include a substantive
+  duplicate/near-duplicate pair.
+- Ignore short connectors and literals with normalized length `< 6`.
+
+Generated recipe output:
+
+- `reports/dmhy_template_recipes.full_top5000.strictdup.seed.jsonl`
+
+Decision distribution:
+
+| decision | count |
+| --- | ---: |
+| `merge_dissimilar_title_continuation` | 4661 |
+| `merge_adjacent_title_continuation` | 444 |
+| `keep_similar_duplicate_or_alias` | 19 |
+| `keep_structural_boundary` | 502 |
+| `keep_path_boundary` | 5 |
+
+Pi RPC review of all 19 remaining `keep_similar_duplicate_or_alias` decisions:
+
+| rows | model result |
+| ---: | --- |
+| 19 | 19 merge |
+
+Examples the reviewer judged as merge:
+
+- `Air In Summer`, `Assault Lily BOUQUET`
+- `Aoki Hagane no Arpeggio Ars Nova`
+- `Shoujo Kageki Revue Starlight`
+- `Motto! Ojamajo Doremi`
+- `Fate stay night Heaven's Feel`
+- `Arafou Otoko no Isekai Tsuuhan Seikatsu`
+- `Watashi, Nouryoku wa Heikinchi de tte Itta yo ne!`
+
+Conclusion: strict duplicate evidence still produced false keeps. The repeated
+literal was often a real repeated word inside one title phrase or a top-literal
+artifact, not a duplicate/alias boundary.
+
+## Experiment 2: merge all adjacent separator-only title slots
+
+Final change:
+
+- `TITLE PATH TITLE` remains `keep_path_boundary`.
+- Non-separator structural boundaries remain `keep_structural_boundary`.
+- Every separator-only adjacent title boundary, `TITLE SEP TITLE`, now merges.
+- Low-similarity merges keep the old decision name
+  `merge_dissimilar_title_continuation`.
+- High-similarity merges use `merge_adjacent_title_continuation`.
+- `keep_similar_duplicate_or_alias` is no longer emitted by the deterministic
+  heuristic.
+
+Generated recipe output:
+
+- `reports/dmhy_template_recipes.full_top5000.mergeall.seed.jsonl`
+
+Decision distribution:
+
+| decision | count |
+| --- | ---: |
+| `merge_dissimilar_title_continuation` | 4661 |
+| `merge_adjacent_title_continuation` | 463 |
+| `keep_structural_boundary` | 502 |
+| `keep_path_boundary` | 5 |
+
+Pi RPC sample of high-similarity `merge_adjacent_title_continuation` decisions:
+
+| rows | model result |
+| ---: | --- |
+| 30 | 29 merge, 1 parse-error |
+
+The reviewer cited concrete title continuations such as:
+
+- `Asobi ni Iku yo!`
+- `Code Geass ~Akito the Exiled~`
+- `Kami nomi zo Shiru Sekai`
+- `Yagate Kimi ni Naru`
+- `Big Order`
+- `Gate Jieitai Kanochi nite Kaku Tatakaeri`
+- `JoJo's Bizarre Adventure - Stardust Crusaders`
+
+Conclusion: merge-all adjacent separator-only title slots is the best of the
+tried heuristics. It matches Pi RPC evidence and simplifies downstream synthetic
+title-span replacement.
+
+## Synthetic generation smoke
+
+Generated schema v2 synthetic augmentation with merge-all recipes:
+
+- `data/schema_v2_synthetic_aug.mergeall.jsonl`
+- `data/schema_v2_synthetic_aug.mergeall.manifest.json`
+
+Validator passed. Manifest summary:
+
+| metric | value |
+| --- | ---: |
+| `generated_rows` | 49998 |
+| `numeric_title_rows` | 30000 |
+| `path_rows` | 19998 |
+| `path_series_rows` | 15000 |
+| `path_movie_rows` | 1666 |
+| `path_special_rows` | 1666 |
+| `path_confuser_rows` | 1666 |
+| `dropped_media_kind_mismatch` | 19611 |
+
+## Commands
+
+Pi RPC high-sim adjacent merge sample:
+
+```powershell
+cargo run --release --manifest-path tools\rust_dmhy_template_apply\Cargo.toml -- `
+  --review-title-boundaries-pi-rpc `
+  --recipes reports\dmhy_template_recipes.full_top5000.mergeall.seed.jsonl `
+  --title-boundary-decisions-output reports\dmhy_title_boundary_pi_rpc_mergeall.adjacent_highsim.jsonl `
+  --title-boundary-heuristic-contains merge_adjacent_title_continuation `
+  --title-boundary-min-similarity 0.85 `
+  --title-boundary-max-similarity 1.0 `
+  --threads 4 `
+  --limit 30
+```
+
+Synthetic smoke:
+
+```powershell
+cargo run --release --manifest-path tools\schema_v2_synthetic_augment\Cargo.toml --bin schema_v2_synthetic_augment -- `
+  --recipes reports\dmhy_template_recipes.full_top5000.mergeall.seed.jsonl `
+  --label-schema-file label_schema.json `
+  --numeric-title-seeds data\synthetic_numeric_titles.txt `
+  --path-prefix-seeds data\synthetic_path_prefixes.txt `
+  --limit-templates 3000 `
+  --max-rows 50000 `
+  --output data\schema_v2_synthetic_aug.mergeall.jsonl `
+  --manifest-output data\schema_v2_synthetic_aug.mergeall.manifest.json
+
+cargo run --release --manifest-path tools\schema_v2_synthetic_augment\Cargo.toml --bin validate_synthetic_aug_jsonl -- `
+  --input data\schema_v2_synthetic_aug.mergeall.jsonl `
+  --manifest data\schema_v2_synthetic_aug.mergeall.manifest.json
+```

tools/rust_dmhy_template_apply/README.md

@@ -84,12 +84,13 @@ cargo run --release --manifest-path tools\rust_dmhy_template_apply\Cargo.toml --
 For stratified audits, select heuristic strata explicitly:
 
 ```powershell
-# High-similarity duplicate/alias keep candidates.
+# High-similarity adjacent-title merges. The old high-similarity keep heuristic
+# was removed after Pi RPC review found it over-preserved title continuations.
 cargo run --release --manifest-path tools\rust_dmhy_template_apply\Cargo.toml -- `
   --review-title-boundaries-pi-rpc `
   --recipes reports\dmhy_template_recipes.full_top5000.seed.jsonl `
-  --title-boundary-decisions-output reports\dmhy_title_boundary_pi_rpc_highsim_keep.jsonl `
-  --title-boundary-heuristic-contains keep_similar_duplicate_or_alias `
+  --title-boundary-decisions-output reports\dmhy_title_boundary_pi_rpc_adjacent_highsim.jsonl `
+  --title-boundary-heuristic-contains merge_adjacent_title_continuation `
   --title-boundary-min-similarity 0.85 `
   --title-boundary-max-similarity 1.0 `
   --threads 4 `

tools/rust_dmhy_template_apply/src/main.rs

@@ -824,7 +824,7 @@ fn compute_title_span_metadata(
     let mut spans: Vec<Vec<usize>> = Vec::new();
     let mut decisions: Vec<Value> = Vec::new();
     let mut current_span: Vec<usize> = Vec::new();
-    let similarity_keep_threshold = 0.72;
+    let low_similarity_threshold = 0.72;
 
     for title_index in title_indices {
         if current_span.is_empty() {
@@ -850,12 +850,12 @@ fn compute_title_span_metadata(
             }));
             spans.push(current_span);
             current_span = vec![title_index];
-        } else if only_separators_between && similarity < similarity_keep_threshold {
+        } else if only_separators_between {
             decisions.push(json!({
                 "left_role_index": previous,
                 "right_role_index": title_index,
                 "similarity": similarity,
-                "decision": "merge_dissimilar_title_continuation",
+                "decision": if similarity < low_similarity_threshold { "merge_dissimilar_title_continuation" } else { "merge_adjacent_title_continuation" },
                 "source": "heuristic",
             }));
             current_span.push(title_index);
@@ -5325,7 +5325,7 @@ mod tests {
     }
 
     #[test]
-    fn title_span_metadata_keeps_similar_duplicates_separate() {
+    fn title_span_metadata_merges_similar_adjacent_titles() {
         let key = "TEXT SEP TEXT SEP EPISODE";
         let roles = suggested_roles(key);
         let mut cluster = Cluster::default();
@@ -5333,8 +5333,27 @@ mod tests {
         cluster.position_literals[0].insert("Frieren".to_string(), 10);
         cluster.position_literals[2].insert("Frieren".to_string(), 10);
         let (spans, decisions) = compute_title_span_metadata(key, &roles, &cluster);
-        assert_eq!(spans.len(), 2, "{spans:?} {decisions:?}");
-        assert_eq!(decisions[0]["decision"], "keep_similar_duplicate_or_alias");
+        assert_eq!(spans.len(), 1, "{spans:?} {decisions:?}");
+        assert_eq!(
+            decisions[0]["decision"],
+            "merge_adjacent_title_continuation"
+        );
+    }
+
+    #[test]
+    fn title_span_metadata_merges_high_similarity_short_connectors() {
+        let key = "TEXT SEP TEXT SEP EPISODE";
+        let roles = suggested_roles(key);
+        let mut cluster = Cluster::default();
+        cluster.position_literals = vec![HashMap::new(); key.split_whitespace().count()];
+        cluster.position_literals[0].insert("no".to_string(), 10);
+        cluster.position_literals[2].insert("no".to_string(), 10);
+        let (spans, decisions) = compute_title_span_metadata(key, &roles, &cluster);
+        assert_eq!(spans.len(), 1, "{spans:?} {decisions:?}");
+        assert_eq!(
+            decisions[0]["decision"],
+            "merge_adjacent_title_continuation"
+        );
     }
 
     #[test]