| Loading entities... |
| 30255 entities loaded |
| 313 have agronomy:hasTaxonomicRank |
| Aho-Corasick: 25780 patterns |
| Streaming 50,000 samples from /data1/users/usr1-iairo/.cache/huggingface/hub/datasets--AnmolNimmala0--agri-slm-india-v1/snapshots/f48459bc1b9c423101145a368d4969f9ac7aaa2e/data [train]... |
| reading 124 local parquet shards |
| 5,000 samples, 5,000 written, 2225 unique entities (25.6s) |
| 10,000 samples, 10,000 written, 2696 unique entities (25.8s) |
| 15,000 samples, 15,000 written, 3003 unique entities (26.0s) |
| 20,000 samples, 20,000 written, 3297 unique entities (26.2s) |
| 25,000 samples, 25,000 written, 3506 unique entities (26.4s) |
| 30,000 samples, 30,000 written, 3698 unique entities (26.6s) |
| 35,000 samples, 35,000 written, 3852 unique entities (26.8s) |
| 40,000 samples, 40,000 written, 3968 unique entities (26.9s) |
| 45,000 samples, 45,000 written, 4088 unique entities (27.2s) |
| 50,000 samples, 50,000 written, 4218 unique entities (27.4s) |
| Done: 50,000 lines, 86,671 entity occurrences, 4218 unique variants in 27.4s |
| Corpus saved to /data1/users/usr1-iairo/agri-slm/tokenizer_output/hf_corpus.txt (78.4 MB) |
| Top 5 by corpus frequency: |
| 1610 'dry matter' |
| 1491 'oryza sativa' |
| 1292 'rabi season' |
| 1141 'triticum aestivum' |
| 1129 'kharif season' |
|
|
| === Scanning corpus for base tokenizer frequencies === |
| 50,000 lines, 36315 unique tokens (1.7s) |
| 100,000 lines, 45913 unique tokens (7.1s) |
| 150,000 lines, 50120 unique tokens (12.7s) |
| 200,000 lines, 53369 unique tokens (18.1s) |
| Done: 201,434 lines, 53673 unique tokens in 18.3s |
|
|
| === Loading base tokenizer from /data1/users/usr1-iairo/agri-slm/tokenizer.json === |
| Target base vocab: 30,300 |
| Intermediate expansion: 30300 -> 32427 tokens (0.0s) |
| Merges: 32,245 / 151,387 (0.0s) |
| Final base vocab: 32,450 |
| Saved base tokenizer to /data1/users/usr1-iairo/agri-slm/tokenizer_output/tokenizer_pre_injection.json |
|
|
| === Coverage check + frequency-per-slot injection (budget=8000) === |
| Groups: 4930, Naturally covered (1 form, single-token): 0 |
| Rejected by filter (attested but filtered): 87 |
| Zero attested frequency: 479 |
| Injected: 8000 case-forms (8000 slots used, 0 remaining) |
| Injected entities by source: agrovoc=2596, kg=811 |
| Top 5 by freq-per-slot: |
| wfps= 4473.0 fps= 1491.0 cost=2 freq= 2982 [kg] 'oryza sativa' (crops.rice) |
| wfps= 3876.0 fps= 1292.0 cost=2 freq= 2584 [kg] 'rabi season' (crops.season.rabi) |
| wfps= 3423.0 fps= 1141.0 cost=2 freq= 2282 [kg] 'triticum aestivum' (crops.wheat) |
| wfps= 3387.0 fps= 1129.0 cost=2 freq= 2258 [kg] 'kharif season' (crops.season.kharif) |
| wfps= 3186.0 fps= 1062.0 cost=2 freq= 2124 [kg] 'zea mays' (crops.maize) |
| Final vocab size: 39987 |
| Injected with leading space (' entity'), so mid-sentence occurrences cost one token rather than two. |
| Still fragmented after injection: 0 |
|
|
| Warm-start mapping: /data1/users/usr1-iairo/agri-slm/tokenizer_output/warmstart_init.json (7537 entries) |
|
|
| Final tokenizer: /data1/users/usr1-iairo/agri-slm/tokenizer_output/tokenizer_final.json |
|
|
| === Smoke Test === |
| ✓ 'wheat' → 2 tokens [single-word common] OK (any tokens) |
| ✗ 'Triticum aestivum' → 6 tokens [binomial] FAIL (got 6, expected 1) |
| ✗ 'Puccinia graminis' → 6 tokens [pathogen binomial] FAIL (got 6, expected 1) |
| ✓ 'glyphosate' → 2 tokens [single-word chemical] OK (any tokens) |
| ✓ 'xylem' → 3 tokens [anatomy single] OK (any tokens) |
| ✗ 'nitrogen fixation' → 3 tokens [multi-word common] FAIL (got 3, expected 1) |
| ✗ 'agriculture in india' → 5 tokens [entity: agriculture] FAIL (got 5, expected 1) |
| ✗ 'apiculture (beekeeping)' → 6 tokens [entity: apiculture] FAIL (got 6, expected 1) |
| ✗ 'madhumakhi palan' → 6 tokens [entity: apiculture] FAIL (got 6, expected 1) |
| ✗ 'apis dorsata' → 3 tokens [entity: apiculture] FAIL (got 3, expected 1) |
| ✗ 'apis mellifera' → 4 tokens [entity: apiculture] FAIL (got 4, expected 1) |
|
|
| 8 FAILURES |
|
|
| === Final Stats === |
| Vocab size: 39987 |
| Naturally covered: 0 |
| Injected case-forms: 8000 |
| Injection slots used: 8000 / 8000 |
| Total groups ranked: 4930 |
|
|
| Done! Output in /data1/users/usr1-iairo/agri-slm/tokenizer_output/ |
|
|
