--- language: - multilingual - ar - zh - ru - ja - ko - he - fa - hi - el - ka - am - hy - ur - bn - ta - te - th tags: - toponym-matching - cross-script - phonetic-embeddings - geospatial - named-entity - information-retrieval - teacher-student - knowledge-distillation license: cc-by-4.0 datasets: - geonames - wikidata - getty-tgn metrics: - recall@k - mrr pipeline_tag: feature-extraction --- # Symphonym v8 — Universal Phonetic Embeddings for Cross-Script Toponym Matching [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.22767194.svg)](https://doi.org/10.5281/zenodo.22767194) Symphonym maps toponyms (place names) from **36 writing systems** into a unified **128-dimensional phonetic embedding space**, enabling direct cross-script similarity comparison without runtime phonetic conversion or language identification. > *"London" / "Лондон" / "伦敦" / "لندن" → [0.12, -0.34, …] (all nearby)* **v8 supersedes [v7](https://huggingface.co/docuracy/symphonym-v7)**, which remains available unchanged. v8 is a retrain from scratch, not a fine-tune: the vocabulary, the script coverage and the IPA lineage all differ, so **v8 weights and v7 vocabularies are not interchangeable** (see *Migrating from v7*). ## Intended Use - **Cross-script toponym matching** in geographic databases and gazetteers - **Phonetic search** — retrieve results for a place name entered in any script - **Historical record linkage** — match pre-standardisation spelling variants - **Multilingual named entity linking** in NLP pipelines - **Digital humanities** — reconciling place references across archival sources The model operates on **phonetic similarity**, not semantic or orthographic similarity. It is designed as a **candidate retrieval** component within a larger reconciliation pipeline, where candidates are subsequently filtered by geographic proximity and other constraints. 🛑 **A similarity score measures the NAME and nothing else.** It carries no geographic term and cannot separate two places that share a name: Newcastle in Australia scores exactly as well as the one in England. Anything auto-accepting a match on this score needs a second, non-name signal. ## Quick Start ```python from inference import SymphonymModel model = SymphonymModel() # loads weights from this directory sim = model.similarity("London", "en", "Лондон", "ru") print(f"London / Лондон: {sim:.3f}") embeddings = model.batch_embed([ ("London", "en"), ("Лондон", "ru"), ("伦敦", "zh"), ("لندن", "ar"), ("ლონდონი", "ka"), ]) ``` ### With HuggingFace `huggingface_hub` ```python from huggingface_hub import snapshot_download model_dir = snapshot_download("docuracy/symphonym-v8") from inference import SymphonymModel model = SymphonymModel(model_dir=model_dir) ``` ## Model Architecture Symphonym uses a **Teacher–Student knowledge distillation** framework. ### Teacher (PhoneticEncoder) — training only - Input: IPA transcriptions via Epitran (+ 102 extensions), Phonikud, CharsiuG2P - Representation: PanPhon192 — 24-dim articulatory feature vectors, 8-bin positional pooling → 192-dim fixed-length input - Architecture: BiLSTM → Self-Attention → Attention Pooling → 128-dim projection ### Student (UniversalEncoder) — deployed model - Input: raw Unicode characters + script ID + language ID + length bucket - Vocabulary: **114,845 characters**, **37 script ids** (36 named scripts plus an `OTHER` catch-all), **2,438 language codes** - Architecture: Character/Script/Language/Length embeddings → Input projection → BiLSTM → Self-Attention (residual) → Attention Pooling → 128-dim projection → L2 normalisation - Parameters: ~8.3M The **length bucket embedding** (16 buckets, 8-dim) conditions every character representation on sequence length, mitigating spurious matches between short toponyms and long compound strings. ### Three-Phase Training Curriculum | Phase | Objective | Epochs | Notes | |-------|-----------|--------|-------| | 1 | Teacher: triplet margin loss on PanPhon192 features | 50 | val_loss 0.0056 | | 2 | Student–Teacher distillation: α·MSE + (1−α)·cosine | 50 | α=0.5, Student–Teacher cosine 0.942 | | 3 | Hard negative fine-tuning (triplet, margin=0.3) | 30 | val_loss 0.02122 | ⚠ **Do not select a checkpoint by training loss.** Four v8 candidates were trained; the one with the best validation loss at *all three* phases was the worst of the four on every downstream measure. Phase loss orders candidates within a fixed batch size and does not compare across recipes. Selection criteria and the four candidate scorecards are in `models/PROVENANCE.md`. ## Evaluation ### MEHDIE Hebrew–Arabic Historical Benchmark (Sagi et al., 2025) Independent evaluation on medieval Hebrew and Arabic geographical sources — **not in training data**. Unweighted mean across the five testsets (137 queries), which is the benchmark's own convention; the string baselines were re-run in the same pass rather than quoted, and reproduce their published values to within 0.1. | Method | R@1 | R@5 | R@10 | MRR | |--------|-----|-----|------|-----| | PanPhon192 (ablation) | 41.1% | 48.2% | 52.3% | 45.0% | | Levenshtein + AnyAscii | 81.5% | 97.6% | **99.4%** | 88.5% | | Jaro-Winkler + AnyAscii | 78.5% | 96.3% | 97.8% | 86.3% | | Symphonym v7 | 85.2% | 97.0% | 97.6% | 90.8% | | **Symphonym v8** | **89.3%** | 97.0% | 98.2% | **92.8%** | **R@1 +4.1 points over v7, MRR +2.0.** Note the *shape*: R@5 is unchanged and R@10 moves +0.6, while R@1 and MRR move substantially. **v8 is not finding answers v7 could not find — it is ranking the right answer first more often.** ⚠ **Levenshtein still wins R@10** (99.4 vs 98.2), as it did for v7. ⚠ **This is a small benchmark.** +4.1 points on R@1 is about five or six queries changing rank. The comparison is paired (same queries, both models), which is stronger than independent samples, but it will not carry more weight than that. ### Cross-Script Pair Validation (11,723 pairs, 179 script combinations) Systematically sampled from training data (up to 10 pairs per script-pair bin); this tests embedding retrieval quality over the full 73.5M-toponym index, not generalisation to unseen sources. 11,535 pairs scored; 188 had an endpoint absent from the current index. | Metric | v7 | v8 | |--------|----|----| | Pass rate (≥0.75 cosine) | 90.7% | **90.8%** | | Cyrillic–Latin (n=1,306) | 0.923 | **0.937** | | Arabic–Latin (n=786) | 0.898 | **0.916** | | Devanagari–Telugu | 0.976 | 0.975 | | Hiragana–Katakana | 0.981 | **0.993** | **Strongest pairs at scale:** Gujarati–Latin 0.957 (n=228), Georgian–Latin 0.947 (n=210), Armenian–Latin 0.943 (n=236), Devanagari–Latin 0.943 (n=681). ## v8 Changes ### 1. The Chinese training signal was contaminated in v7 — this is a correction v7's card attributed low CJK–Hiragana similarity to "a genuine phonological mismatch, not a model deficiency". **That explanation was wrong.** v7's IPA for Chinese Han toponyms was produced under a defective language tag, so the model learned Chinese characters with **Japanese readings** attached. The similarity structure that resulted looked like a phonological fact and was an artefact of the G2P dispatch. v8 fixes the tag and retrains. CJK–Hiragana mean similarity **falls from 0.437 to 0.320** — and that fall is the *correct* direction: Chinese Han and Japanese Hiragana genuinely do not sound alike, and v7's higher score was the contamination. ### 2. Script coverage nearly doubled 20 script ids → 37 (36 named + `OTHER`); 113,280 → 114,845 characters; 1,944 → 2,438 language codes. ⚠ **The vocabularies are not additive.** v8 is not v7 plus new entries: ids are reassigned, and a v8 checkpoint paired with a v7 vocabulary **does not raise an error — it produces plausible garbage.** See *Migrating from v7*. ### 3. Ranking improved; reach did not Measured over one fixed 1,053,229-name haystack with 4,843 queries, v7 against v8: | | v7 | v8 | |---|---|---| | recall@1 | 0.0593 | 0.0690 | | recall@10 | 0.2893 | 0.3176 | | recall@200 | 0.4792 | 0.4908 | | MRR | 0.1383 | 0.1557 | Every metric improved, but **recall@200 rose only +0.0116 while MRR rose +0.0173**. Three independent measurements — this one, MEHDIE, and order-sensitivity bands — agree that v8 orders better rather than reaching further. ## Limitations - **Phonetic similarity only**: no geographic coordinates, semantics or entity types. Phonetically similar but geographically unrelated names (Austria/Australia: 0.883) score highly. - 🛑 **Discrimination fell slightly.** Separating true pairs from false ones, **AUC 0.9324 (v7) → 0.9270 (v8)**. v8 retrieves and ranks better while separating marginally worse. If your use is thresholding rather than ranking, measure before assuming v8 is an upgrade for it. - **More than half the correct answers are beyond rank 200.** At recall@200 = 0.4908 on the corpus above, no amount of downstream re-ranking can reach them; that is a property of the embedding space, not of the scoring layer. - **Training bias**: sources over-represent populated places with official names in high-resource languages. Under-represented scripts and mundane places are weaker. - **Tonal languages**: PanPhon encodes segmental articulatory features but not tone. - **IPA coverage is partial**: 54% of training-namespace toponyms received an IPA transcription; the rest contribute through character-level distillation only. ## Migrating from v7 🛑 **Weights and vocabularies must be replaced together.** Pairing v8 weights with v7 vocabularies raises no error and silently produces meaningless embeddings. ``` char_vocab.json 16a4d41e868fecc16e1b701363e1bda8 114,845 chars lang_vocab.json 17ce7fe69a836ddd32c9577cdd60f389 2,438 langs script_vocab.json 18fd9410e1c543429ec8327a29ee3225 37 script ids ``` **Embeddings are not comparable across versions.** A v7 query vector scored against a v8 index returns confident nonsense rather than an error. Re-embed the corpus, and if a client generates query vectors, upgrade the client in the same change. ## Training Data Trained on 73.5 million unique toponyms from: | Source | License | |--------|---------| | GeoNames | CC BY 4.0 | | Wikidata | CC0 | | Getty TGN | ODC-By 1.0 | ## Repository Contents ``` model.safetensors Student (UniversalEncoder) weights config.json Architecture hyperparameters + measured index stats inference.py Self-contained inference module requirements.txt Dependencies PROVENANCE.md Checkpoint lineage, candidate selection, vocab md5s symphonym-v8.onnx ONNX export (batch 1) for browser/edge inference symphonym-v8.onnx.provenance.json vocab/ char_vocab.json 114,845-character vocabulary lang_vocab.json 2,438 language codes script_vocab.json 37 script ids evaluation/ MEHDIE, cross-script and candidate-arm reports training_stats/ IPA coverage, phase{1,2,3} metrics epitran_extensions/ 102 custom CSV G2P files ``` `config.json` carries `total_toponyms`, `embedding_coverage`, `as_of` and `measured_from`, each **measured from the live index at publish time** rather than remembered — an earlier release shipped a hard-coded 66,924,548 at 100% coverage against a live index of 72,703,777. ## Citation ```bibtex @misc{symphonym2026, author = {Gadd, Stephen}, title = {Symphonym: Universal Phonetic Embeddings for Cross-Script Toponym Matching}, year = {2026}, eprint = {2601.06932}, archivePrefix = {arXiv}, primaryClass = {cs.CL}, url = {https://arxiv.org/abs/2601.06932}, doi = {10.48550/arXiv.2601.06932} } @dataset{symphonym_v8_zenodo, author = {Gadd, Stephen}, title = {Symphonym v8 — Universal Phonetic Embeddings for Cross-Script Toponym Matching}, year = {2026}, doi = {10.5281/zenodo.22767194}, url = {https://doi.org/10.5281/zenodo.22767194} } ``` ⚠ **The arXiv preprint describes v7**, including the two claims corrected above (the Chinese contamination, and the weight carried by letter order). A revision is in preparation. Cite the Zenodo DOI for v8 specifically.