Loading entities...
  30255 entities loaded
  313 have agronomy:hasTaxonomicRank
  Aho-Corasick: 25780 patterns
Streaming 50,000 samples from /data1/users/usr1-iairo/.cache/huggingface/hub/datasets--AnmolNimmala0--agri-slm-india-v1/snapshots/f48459bc1b9c423101145a368d4969f9ac7aaa2e/data [train]...
  reading 124 local parquet shards
  5,000 samples, 5,000 written, 2225 unique entities (25.6s)
  10,000 samples, 10,000 written, 2696 unique entities (25.8s)
  15,000 samples, 15,000 written, 3003 unique entities (26.0s)
  20,000 samples, 20,000 written, 3297 unique entities (26.2s)
  25,000 samples, 25,000 written, 3506 unique entities (26.4s)
  30,000 samples, 30,000 written, 3698 unique entities (26.6s)
  35,000 samples, 35,000 written, 3852 unique entities (26.8s)
  40,000 samples, 40,000 written, 3968 unique entities (26.9s)
  45,000 samples, 45,000 written, 4088 unique entities (27.2s)
  50,000 samples, 50,000 written, 4218 unique entities (27.4s)
  Done: 50,000 lines, 86,671 entity occurrences, 4218 unique variants in 27.4s
  Corpus saved to /data1/users/usr1-iairo/agri-slm/tokenizer_output/hf_corpus.txt (78.4 MB)
  Top 5 by corpus frequency:
      1610  'dry matter'
      1491  'oryza sativa'
      1292  'rabi season'
      1141  'triticum aestivum'
      1129  'kharif season'

=== Scanning corpus for base tokenizer frequencies ===
  50,000 lines, 36315 unique tokens (1.7s)
  100,000 lines, 45913 unique tokens (7.1s)
  150,000 lines, 50120 unique tokens (12.7s)
  200,000 lines, 53369 unique tokens (18.1s)
  Done: 201,434 lines, 53673 unique tokens in 18.3s

=== Loading base tokenizer from /data1/users/usr1-iairo/agri-slm/tokenizer.json ===
  Target base vocab: 30,300
  Intermediate expansion: 30300 -> 32427 tokens (0.0s)
  Merges: 32,245 / 151,387 (0.0s)
  Final base vocab: 32,450
  Saved base tokenizer to /data1/users/usr1-iairo/agri-slm/tokenizer_output/tokenizer_pre_injection.json

=== Coverage check + frequency-per-slot injection (budget=8000) ===
  Groups: 4930, Naturally covered (1 form, single-token): 0
  Rejected by filter (attested but filtered): 87
  Zero attested frequency: 479
  Injected: 8000 case-forms (8000 slots used, 0 remaining)
  Injected entities by source: agrovoc=2596, kg=811
  Top 5 by freq-per-slot:
    wfps=  4473.0  fps=  1491.0  cost=2  freq=  2982  [kg] 'oryza sativa' (crops.rice)
    wfps=  3876.0  fps=  1292.0  cost=2  freq=  2584  [kg] 'rabi season' (crops.season.rabi)
    wfps=  3423.0  fps=  1141.0  cost=2  freq=  2282  [kg] 'triticum aestivum' (crops.wheat)
    wfps=  3387.0  fps=  1129.0  cost=2  freq=  2258  [kg] 'kharif season' (crops.season.kharif)
    wfps=  3186.0  fps=  1062.0  cost=2  freq=  2124  [kg] 'zea mays' (crops.maize)
  Final vocab size: 39987
  Injected with leading space (' entity'), so mid-sentence occurrences cost one token rather than two.
  Still fragmented after injection: 0

  Warm-start mapping: /data1/users/usr1-iairo/agri-slm/tokenizer_output/warmstart_init.json (7537 entries)

  Final tokenizer: /data1/users/usr1-iairo/agri-slm/tokenizer_output/tokenizer_final.json

=== Smoke Test ===
  ✓ 'wheat'                             →  2 tokens  [single-word common]  OK (any tokens)
  ✗ 'Triticum aestivum'                 →  6 tokens  [binomial]  FAIL (got 6, expected 1)
  ✗ 'Puccinia graminis'                 →  6 tokens  [pathogen binomial]  FAIL (got 6, expected 1)
  ✓ 'glyphosate'                        →  2 tokens  [single-word chemical]  OK (any tokens)
  ✓ 'xylem'                             →  3 tokens  [anatomy single]  OK (any tokens)
  ✗ 'nitrogen fixation'                 →  3 tokens  [multi-word common]  FAIL (got 3, expected 1)
  ✗ 'agriculture in india'              →  5 tokens  [entity: agriculture]  FAIL (got 5, expected 1)
  ✗ 'apiculture (beekeeping)'           →  6 tokens  [entity: apiculture]  FAIL (got 6, expected 1)
  ✗ 'madhumakhi palan'                  →  6 tokens  [entity: apiculture]  FAIL (got 6, expected 1)
  ✗ 'apis dorsata'                      →  3 tokens  [entity: apiculture]  FAIL (got 3, expected 1)
  ✗ 'apis mellifera'                    →  4 tokens  [entity: apiculture]  FAIL (got 4, expected 1)

  8 FAILURES

=== Final Stats ===
  Vocab size: 39987
  Naturally covered: 0
  Injected case-forms: 8000
  Injection slots used: 8000 / 8000
  Total groups ranked: 4930

Done! Output in /data1/users/usr1-iairo/agri-slm/tokenizer_output/