Oddproblem commited on
Commit
23a5f1f
·
verified ·
1 Parent(s): 8f585eb

Upload option_b_entity_resolution.ipynb with huggingface_hub

Browse files
Files changed (1) hide show
  1. option_b_entity_resolution.ipynb +1076 -0
option_b_entity_resolution.ipynb ADDED
@@ -0,0 +1,1076 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "cells": [
3
+ {
4
+ "cell_type": "markdown",
5
+ "id": "494afa14",
6
+ "metadata": {},
7
+ "source": [
8
+ "# Option B \u2014 Hybrid Multi-View Blocking + Bi-Encoder Neural Embeddings\n",
9
+ "### Business Entity Resolution \u2014 Amazon ML Challenge 2026\n",
10
+ "\n",
11
+ "This notebook implements the complete, production-ready Option B pipeline:\n",
12
+ "1. **Universal Fast Transliteration & Accent Stripping** (Phonetic Indic table + NFKD + unidecode before key extraction)\n",
13
+ "2. **DuckDB Multi-View Blocking**:\n",
14
+ " - Both Source 2 and Source 3 targets combined into one index\n",
15
+ " - Target keys and frequency caps indexed **once** per country (not re-run in inner loops)\n",
16
+ " - S1 batched to avoid memory bottlenecks\n",
17
+ " - Top-K candidate pairs filtering per entity\n",
18
+ "3. **Mandatory Blocking Recall Ceiling Check**:\n",
19
+ " - Accurately checks recall on ground truth to ensure zero candidate loss\n",
20
+ "4. **Deduplicated Bi-Encoder Embeddings (Qwen3-Embedding-0.6B)**:\n",
21
+ " - One embedding per **unique** entity (not per pair)\n",
22
+ " - Multi-GPU DataParallel support (e.g. 2x T4 on Kaggle)\n",
23
+ " - Persistent `EntityEmbedder` class available for both train and test phases\n",
24
+ "5. **LightGBM Reranker with In-Fold-Only Metric Evaluation**:\n",
25
+ " - `macro_f05` evaluates **only** the entities in the current validation fold (fixing CV skew)\n",
26
+ " - Per-country threshold sweeps (France gets its own operating point)\n",
27
+ "6. **End-to-End Test Set Inference**:\n",
28
+ " - Full test blocking, embedding, feature extraction, and prediction\n",
29
+ "7. **1:Many Target-Exclusivity Resolution**:\n",
30
+ " - Each Target entity is assigned to at most one S1 (highest probability >= threshold)\n",
31
+ " - Each S1 can match to multiple targets (across S2 and S3)\n",
32
+ "8. **Submission Generation & Official Validation**:\n",
33
+ " - Exports `matching_results.tsv` and `candidate_pairs.tsv`\n",
34
+ " - Validates format with `utils/validate_submission.py`\n"
35
+ ]
36
+ },
37
+ {
38
+ "cell_type": "code",
39
+ "id": "fc4d836b",
40
+ "metadata": {},
41
+ "source": [
42
+ "import os\n",
43
+ "import sys\n",
44
+ "import time\n",
45
+ "import json\n",
46
+ "import gc\n",
47
+ "import re\n",
48
+ "import unicodedata\n",
49
+ "from pathlib import Path\n",
50
+ "from collections import defaultdict\n",
51
+ "\n",
52
+ "import numpy as np\n",
53
+ "import pandas as pd\n",
54
+ "import duckdb\n",
55
+ "import torch\n",
56
+ "import torch.nn as nn\n",
57
+ "import torch.nn.functional as F\n",
58
+ "\n",
59
+ "# ---- Data Directory Auto-Discovery ----\n",
60
+ "# Checks Kaggle paths, standard local paths, or relative paths\n",
61
+ "POSSIBLE_DATA_DIRS = [\n",
62
+ " Path(\"/kaggle/input/amazon-ml-challenge-2026\"),\n",
63
+ " Path(\"/kaggle/input/amazon-ml\"),\n",
64
+ " Path(\"/kaggle/input/datasets/arghasaha1905/amazon-ml\"),\n",
65
+ " Path(\"/kaggle/input/student-resource\"),\n",
66
+ " Path(\"dataset\"),\n",
67
+ " Path(\".\"),\n",
68
+ "]\n",
69
+ "\n",
70
+ "DATA_DIR = None\n",
71
+ "for p in POSSIBLE_DATA_DIRS:\n",
72
+ " if p.exists() and (\n",
73
+ " (p / \"train_source1.tsv\").exists() or\n",
74
+ " (p / \"train\" / \"train_source1.tsv\").exists() or\n",
75
+ " (p / \"test_source1.tsv\").exists() or\n",
76
+ " (p / \"test\" / \"test_source1.tsv\").exists()\n",
77
+ " ):\n",
78
+ " DATA_DIR = p\n",
79
+ " break\n",
80
+ "\n",
81
+ "if DATA_DIR is None:\n",
82
+ " DATA_DIR = Path(\"/kaggle/input/amazon-ml-challenge-2026\")\n",
83
+ "\n",
84
+ "print(f\"Detected DATA_DIR: {DATA_DIR}\")\n",
85
+ "\n",
86
+ "# ---- Output & Cache Directories ----\n",
87
+ "if Path(\"/kaggle/working\").exists():\n",
88
+ " OUTPUT_DIR = Path(\"/kaggle/working/output\")\n",
89
+ " CACHE_DIR = Path(\"/kaggle/working/cache\")\n",
90
+ "else:\n",
91
+ " OUTPUT_DIR = Path(\"output\")\n",
92
+ " CACHE_DIR = Path(\"cache\")\n",
93
+ "\n",
94
+ "OUTPUT_DIR.mkdir(parents=True, exist_ok=True)\n",
95
+ "CACHE_DIR.mkdir(parents=True, exist_ok=True)\n",
96
+ "\n",
97
+ "# ---- Run Configuration ----\n",
98
+ "# Set SAMPLE_MODE = True to test on a fast sample (20k per country) first.\n",
99
+ "# Set SAMPLE_MODE = False for the full 2.2M train + 1.73M test run.\n",
100
+ "SAMPLE_MODE = True\n",
101
+ "SAMPLE_N_PER_COUNTRY = 20_000\n",
102
+ "TOP_K_CANDIDATES = 35\n",
103
+ "FREQ_CAP_DEFAULT = 500\n",
104
+ "FREQ_CAP_FRANCE = 1000\n",
105
+ "\n",
106
+ "# ---- Hardware & GPU Check ----\n",
107
+ "n_gpus = torch.cuda.device_count()\n",
108
+ "print(f\"CUDA available: {torch.cuda.is_available()} | GPU count: {n_gpus}\")\n",
109
+ "for i in range(n_gpus):\n",
110
+ " props = torch.cuda.get_device_properties(i)\n",
111
+ " print(f\" GPU {i}: {props.name} | {props.total_memory / 1e9:.1f} GB\")\n",
112
+ "\n",
113
+ "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
114
+ "print(f\"Using primary device: {device}\")\n"
115
+ ],
116
+ "execution_count": null,
117
+ "outputs": []
118
+ },
119
+ {
120
+ "cell_type": "markdown",
121
+ "id": "6025bbce",
122
+ "metadata": {},
123
+ "source": [
124
+ "## 1. High-Performance Text Normalization & Transliteration\n",
125
+ "Indic scripts (Devanagari, Tamil, Telugu, Bengali, Kannada, Malayalam, etc.) and French diacritics\n",
126
+ "must be mapped phonetically into Latin characters **before** generating blocking keys.\n",
127
+ "We use a precompiled C-level `str.translate` lookup table with NFKD normalization, ensuring\n",
128
+ "sub-microsecond execution per entity with zero empty-string wipes.\n"
129
+ ]
130
+ },
131
+ {
132
+ "cell_type": "code",
133
+ "id": "faa1babb",
134
+ "metadata": {},
135
+ "source": [
136
+ "try:\n",
137
+ " from unidecode import unidecode\n",
138
+ "except ImportError:\n",
139
+ " import subprocess\n",
140
+ " subprocess.run([sys.executable, \"-m\", \"pip\", \"install\", \"unidecode\", \"-q\"])\n",
141
+ " from unidecode import unidecode\n",
142
+ "\n",
143
+ "# Universal Indic & European phonetic transliteration map\n",
144
+ "DEV_PHONETICS = {\n",
145
+ " 0x05:'a',0x06:'a',0x07:'i',0x08:'i',0x09:'u',0x0A:'u',0x0B:'ri',0x0E:'e',0x0F:'e',0x10:'ai',0x12:'o',0x13:'o',0x14:'au',\n",
146
+ " 0x15:'k',0x16:'kh',0x17:'g',0x18:'gh',0x19:'n',0x1A:'ch',0x1B:'chh',0x1C:'j',0x1D:'jh',0x1E:'n',\n",
147
+ " 0x1F:'t',0x20:'th',0x21:'d',0x22:'dh',0x23:'n',0x24:'t',0x25:'th',0x26:'d',0x27:'dh',0x28:'n',\n",
148
+ " 0x2A:'p',0x2B:'ph',0x2C:'b',0x2D:'bh',0x2E:'m',0x2F:'y',0x30:'r',0x31:'r',0x32:'l',0x33:'l',0x35:'v',\n",
149
+ " 0x36:'sh',0x37:'sh',0x38:'s',0x39:'h',0x3E:'a',0x3F:'i',0x40:'i',0x41:'u',0x42:'u',0x43:'ri',\n",
150
+ " 0x46:'e',0x47:'e',0x48:'ai',0x4A:'o',0x4B:'o',0x4C:'au',0x4D:'',0x02:'n',0x01:'n',0x03:'h'\n",
151
+ "}\n",
152
+ "TRANSLIT_MAP = {chr(base+off): lat for base in (0x0900,0x0980,0x0A00,0x0A80,0x0B00,0x0B80,0x0C00,0x0C80,0x0D00) for off, lat in DEV_PHONETICS.items()}\n",
153
+ "TRANSLIT_MAP.update({\n",
154
+ " '\u00e9':'e','\u00e8':'e','\u00ea':'e','\u00eb':'e','\u00e0':'a','\u00e2':'a','\u00e4':'a','\u00f4':'o','\u00f6':'o',\n",
155
+ " '\u00ee':'i','\u00ef':'i','\u00f9':'u','\u00fb':'u','\u00fc':'u','\u00e7':'c','\u00f1':'n','\u00c9':'e','\u00c8':'e','\u00ca':'e','\u00cb':'e',\n",
156
+ " '\u00c0':'a','\u00c2':'a','\u00c4':'a','\u00d4':'o','\u00d6':'o','\u00ce':'i','\u00cf':'i','\u00d9':'u','\u00db':'u','\u00dc':'u','\u00c7':'c','\u00d1':'n',\n",
157
+ " '\u00ab':'','\u00bb':'','\u2019':\"'\",'\u201c':'\"','\u201d':'\"'\n",
158
+ "})\n",
159
+ "TRANSLIT_TABLE = str.maketrans(TRANSLIT_MAP)\n",
160
+ "\n",
161
+ "_WS_RE = re.compile(r\"\\s+\")\n",
162
+ "_NONALNUM_RE = re.compile(r\"[^a-z0-9\\s]\")\n",
163
+ "_DOMAIN_RE = re.compile(r\"^(https?://)?(www\\.)?\", re.IGNORECASE)\n",
164
+ "_TLD_RE = re.compile(r\"\\.(com|in|org|net|co|io|fr)$\", re.IGNORECASE)\n",
165
+ "SUFFIX_RE = re.compile(r\"\\b(pvt|ltd|limited|private|inc|incorporated|llc|corp|corporation|co|company|enterprises|services|solutions|group|sarl|sas|sasu|eurl|sci|association|fils)\\b\", re.I)\n",
166
+ "\n",
167
+ "def clean_and_translit(text):\n",
168
+ " \"\"\"Fast transliteration: C-table translation + NFKD + unidecode fallback + clean whitespace.\"\"\"\n",
169
+ " if text is None or (isinstance(text, float) and np.isnan(text)):\n",
170
+ " return \"\"\n",
171
+ " s = str(text).translate(TRANSLIT_TABLE)\n",
172
+ " s = unicodedata.normalize(\"NFKD\", s)\n",
173
+ " s = \"\".join(c for c in s if not unicodedata.combining(c))\n",
174
+ " # fallback for any exotic scripts not in base table\n",
175
+ " if any(ord(c) > 127 for c in s):\n",
176
+ " s = unidecode(s)\n",
177
+ " s = s.lower()\n",
178
+ " s = _DOMAIN_RE.sub(\"\", s)\n",
179
+ " s = _TLD_RE.sub(\"\", s)\n",
180
+ " s = _NONALNUM_RE.sub(\" \", s)\n",
181
+ " return _WS_RE.sub(\" \", s).strip()\n",
182
+ "\n",
183
+ "def strip_legal_suffixes(name_clean):\n",
184
+ " return SUFFIX_RE.sub(\" \", name_clean).strip()\n",
185
+ "\n",
186
+ "def compact_alnum(s):\n",
187
+ " return _NONALNUM_RE.sub(\"\", s).replace(\" \", \"\")\n",
188
+ "\n",
189
+ "# Verification against known Indian and French test entities\n",
190
+ "_tests = [\n",
191
+ " (\"\u0930\u093f\u092f\u0932 \u0915\u0902\u0938\u0932\u094d\u091f\u0947\u0902\u091f\u094d\u0938 \u092a\u094d\u0930\u093e\u0907\u0935\u0947\u091f \u0932\u093f\u092e\u093f\u091f\u0947\u0921\", \"real\"),\n",
192
+ " (\"Physical Th\u00e9rapy Care\", \"physical\"),\n",
193
+ " (\"\u0baa\u0bbe\u0bb2\u0bbe\u0b9c\u0bbf \u0b95\u0bcb\u0bb2\u0bcd\u0b9f\u0bc1 \u0b95\u0bc7\u0bb0\u0bcd\", \"balaji\"),\n",
194
+ "]\n",
195
+ "for raw, expect_prefix in _tests:\n",
196
+ " out = clean_and_translit(raw)\n",
197
+ " print(f\"{raw!r:45s} -> {out!r}\")\n",
198
+ " assert out != \"\", f\"Transliteration produced an EMPTY string for: {raw!r}\"\n",
199
+ " if expect_prefix:\n",
200
+ " assert out.startswith(expect_prefix), f\"Expected prefix {expect_prefix!r}, got {out!r}\"\n"
201
+ ],
202
+ "execution_count": null,
203
+ "outputs": []
204
+ },
205
+ {
206
+ "cell_type": "markdown",
207
+ "id": "51550e1d",
208
+ "metadata": {},
209
+ "source": [
210
+ "## 2. Load Dataset\n",
211
+ "Locates and loads `source1`, `source2`, `source3`, and `ground_truth`.\n",
212
+ "In `SAMPLE_MODE = True`, it takes a balanced per-country sample to enable sub-2-minute prototyping.\n"
213
+ ]
214
+ },
215
+ {
216
+ "cell_type": "code",
217
+ "id": "79917e97",
218
+ "metadata": {},
219
+ "source": [
220
+ "def find_file(filename):\n",
221
+ " \"\"\"Finds a dataset file whether in flat structure or train/test subfolders.\"\"\"\n",
222
+ " candidates = [\n",
223
+ " DATA_DIR / filename,\n",
224
+ " DATA_DIR / \"train\" / filename,\n",
225
+ " DATA_DIR / \"test\" / filename,\n",
226
+ " ]\n",
227
+ " for c in candidates:\n",
228
+ " if c.exists():\n",
229
+ " return c\n",
230
+ " raise FileNotFoundError(f\"Could not locate {filename} in {DATA_DIR}\")\n",
231
+ "\n",
232
+ "def load_tsv(filename):\n",
233
+ " path = find_file(filename)\n",
234
+ " df = pd.read_csv(path, sep=\"\\t\", dtype=str, keep_default_na=False)\n",
235
+ " print(f\"Loaded {filename}: {len(df):,} rows, cols={list(df.columns)}\")\n",
236
+ " return df\n",
237
+ "\n",
238
+ "train_s1 = load_tsv(\"train_source1.tsv\")\n",
239
+ "train_s2 = load_tsv(\"train_source2.tsv\")\n",
240
+ "train_s3 = load_tsv(\"train_source3.tsv\")\n",
241
+ "train_gt = load_tsv(\"train_ground_truth.tsv\")\n",
242
+ "\n",
243
+ "for df in [train_s1, train_s2, train_s3]:\n",
244
+ " assert df[\"entity_id\"].is_unique, \"Duplicate entity_id found!\"\n",
245
+ "\n",
246
+ "if SAMPLE_MODE:\n",
247
+ " print(f\"\\n[SAMPLE_MODE active] Sampling {SAMPLE_N_PER_COUNTRY:,} entities per country from train_s1...\")\n",
248
+ " train_s1 = (train_s1.groupby(\"country\", group_keys=False)\n",
249
+ " .apply(lambda g: g.sample(min(SAMPLE_N_PER_COUNTRY, len(g)), random_state=42))\n",
250
+ " .reset_index(drop=True))\n",
251
+ " keep_ids = set(train_s1[\"entity_id\"])\n",
252
+ " train_gt = train_gt[train_gt[\"source1_entity_id\"].isin(keep_ids)].reset_index(drop=True)\n",
253
+ " print(f\"train_s1 sampled to: {len(train_s1):,} rows\")\n"
254
+ ],
255
+ "execution_count": null,
256
+ "outputs": []
257
+ },
258
+ {
259
+ "cell_type": "markdown",
260
+ "id": "656c196c",
261
+ "metadata": {},
262
+ "source": [
263
+ "## 3. Pre-Normalize Name and Address Fields\n",
264
+ "Adds `name_clean`, `name_root`, `addr_clean`, `name_compact`, `house_no`, and `postal`.\n",
265
+ "Only normalizes required tables, preserving memory.\n"
266
+ ]
267
+ },
268
+ {
269
+ "cell_type": "code",
270
+ "id": "69261b0f",
271
+ "metadata": {},
272
+ "source": [
273
+ "def add_normalized_columns(df):\n",
274
+ " \"\"\"Vectorized / batch text normalization for high speed and minimal memory.\"\"\"\n",
275
+ " df = df.copy()\n",
276
+ " names = df[\"business_name\"].tolist()\n",
277
+ " addrs = df[\"business_address\"].tolist()\n",
278
+ "\n",
279
+ " clean_names = [clean_and_translit(n) for n in names]\n",
280
+ " clean_addrs = [clean_and_translit(a) for a in addrs]\n",
281
+ "\n",
282
+ " df[\"name_clean\"] = clean_names\n",
283
+ " df[\"name_root\"] = [strip_legal_suffixes(n) for n in clean_names]\n",
284
+ " df[\"addr_clean\"] = clean_addrs\n",
285
+ " df[\"name_compact\"] = [compact_alnum(n) for n in clean_names]\n",
286
+ "\n",
287
+ " df[\"house_no\"] = df[\"addr_clean\"].str.extract(r\"(\\d{1,6})\", expand=False).fillna(\"\")\n",
288
+ " df[\"postal\"] = df[\"addr_clean\"].str.extract(r\"(\\d{5,6})(?!.*\\d{5,6})\", expand=False).fillna(\"\")\n",
289
+ " return df\n",
290
+ "\n",
291
+ "print(\"Normalizing train_s1...\")\n",
292
+ "train_s1 = add_normalized_columns(train_s1)\n",
293
+ "\n",
294
+ "empty_name_frac = (train_s1[\"name_clean\"] == \"\").mean()\n",
295
+ "assert empty_name_frac < 0.01, \"Error: Transliteration produced excessive empty strings!\"\n"
296
+ ],
297
+ "execution_count": null,
298
+ "outputs": []
299
+ },
300
+ {
301
+ "cell_type": "markdown",
302
+ "id": "8a2f4922",
303
+ "metadata": {},
304
+ "source": [
305
+ "## 4. Multi-View DuckDB Blocking (Fixed & Optimized)\n",
306
+ "Key fixes implemented:\n",
307
+ "- **Combined Targets**: Both Source 2 and Source 3 targets are registered together into `targets_tmp` (eliminating the bug where Source 2 was overwritten).\n",
308
+ "- **Index-Once Architecture**: Target keys and frequency caps are generated **once per country**, completely removing the 17x redundant inner loop computation.\n",
309
+ "- **Top-K Candidate Selection**: Keeps Top-K (35) candidates per S1 entity.\n"
310
+ ]
311
+ },
312
+ {
313
+ "cell_type": "code",
314
+ "id": "e2984698",
315
+ "metadata": {},
316
+ "source": [
317
+ "con = duckdb.connect()\n",
318
+ "con.execute(\"PRAGMA threads=8;\")\n",
319
+ "con.execute(\"PRAGMA max_memory='12GB';\")\n",
320
+ "con.execute(f\"PRAGMA temp_directory='{CACHE_DIR}/duckdb_spill';\")\n",
321
+ "\n",
322
+ "def generate_keys_sql(alias):\n",
323
+ " return f\"\"\"\n",
324
+ " SELECT entity_id, 'p5:' || substr(name_compact, 1, 5) AS key FROM {alias} WHERE length(name_compact) >= 5\n",
325
+ " UNION ALL\n",
326
+ " SELECT entity_id, 'nt:' || tok AS key\n",
327
+ " FROM (SELECT entity_id, unnest(string_split(name_clean, ' ')) AS tok FROM {alias})\n",
328
+ " WHERE length(tok) >= 3\n",
329
+ " UNION ALL\n",
330
+ " SELECT entity_id, 'nb:' || toks[1] || '_' || toks[2] AS key\n",
331
+ " FROM (SELECT entity_id, string_split(name_clean, ' ') AS toks FROM {alias})\n",
332
+ " WHERE len(toks) >= 2 AND length(toks[1]) >= 3 AND length(toks[2]) >= 3\n",
333
+ " UNION ALL\n",
334
+ " SELECT entity_id, 'np:' || substr(toks[1],1,3) || '_' || substr(toks[2],1,3) AS key\n",
335
+ " FROM (SELECT entity_id, string_split(name_clean, ' ') AS toks FROM {alias})\n",
336
+ " WHERE len(toks) >= 2 AND length(toks[1]) >= 3 AND length(toks[2]) >= 3\n",
337
+ " UNION ALL\n",
338
+ " SELECT entity_id, 'addr_toks:' || tok AS key\n",
339
+ " FROM (SELECT entity_id, unnest(string_split(addr_clean, ' ')) AS tok FROM {alias})\n",
340
+ " WHERE length(tok) >= 4\n",
341
+ " UNION ALL\n",
342
+ " SELECT entity_id, 'pn:' || postal || '_' || substr(name_compact,1,3) AS key\n",
343
+ " FROM {alias} WHERE postal != '' AND length(name_compact) >= 3\n",
344
+ " UNION ALL\n",
345
+ " SELECT entity_id, 'hn:' || house_no || '_' || substr(name_compact,1,3) AS key\n",
346
+ " FROM {alias} WHERE house_no != '' AND length(name_compact) >= 3\n",
347
+ " UNION ALL\n",
348
+ " SELECT entity_id, 'ht:' || house_no || '_' || toks[1] AS key\n",
349
+ " FROM (SELECT entity_id, house_no, string_split(name_clean, ' ') AS toks FROM {alias})\n",
350
+ " WHERE house_no != '' AND len(toks) >= 1 AND length(toks[1]) >= 3\n",
351
+ " \"\"\"\n",
352
+ "\n",
353
+ "def run_blocking_for_country(s1_country_df, targets_country_df, country, freq_cap, top_k, batch_size=50_000):\n",
354
+ " \"\"\"Extracts target keys ONCE, then streams S1 batches for candidate joining.\"\"\"\n",
355
+ " if len(s1_country_df) == 0:\n",
356
+ " return pd.DataFrame(columns=[\"s1_id\", \"target_id\", \"key_hits\"])\n",
357
+ "\n",
358
+ " t_start = time.time()\n",
359
+ " print(f\"\\n--- [Blocking {country}] S1: {len(s1_country_df):,}, Combined Targets: {len(targets_country_df):,} ---\")\n",
360
+ " \n",
361
+ " con.register(\"targets_country\", targets_country_df)\n",
362
+ " tgt_sql = generate_keys_sql(\"targets_country\")\n",
363
+ " con.execute(f\"CREATE OR REPLACE TEMP TABLE target_keys AS {tgt_sql}\")\n",
364
+ " \n",
365
+ " # Calculate key frequencies once\n",
366
+ " con.execute(f\"\"\"\n",
367
+ " CREATE OR REPLACE TEMP TABLE key_freq AS\n",
368
+ " SELECT key, count(*) AS n FROM target_keys GROUP BY key\n",
369
+ " \"\"\")\n",
370
+ " con.execute(f\"\"\"\n",
371
+ " CREATE OR REPLACE TEMP TABLE filtered_target_keys AS\n",
372
+ " SELECT t.entity_id, t.key\n",
373
+ " FROM target_keys t\n",
374
+ " JOIN key_freq f ON t.key = f.key AND f.n <= {freq_cap}\n",
375
+ " \"\"\")\n",
376
+ " con.execute(\"CREATE INDEX IF NOT EXISTS ftk_idx ON filtered_target_keys(key);\")\n",
377
+ "\n",
378
+ " n_batches = (len(s1_country_df) + batch_size - 1) // batch_size\n",
379
+ " candidates_list = []\n",
380
+ "\n",
381
+ " for b in range(n_batches):\n",
382
+ " s1_batch = s1_country_df.iloc[b * batch_size : (b + 1) * batch_size]\n",
383
+ " con.register(\"s1_batch\", s1_batch)\n",
384
+ " s1_sql = generate_keys_sql(\"s1_batch\")\n",
385
+ " con.execute(f\"CREATE OR REPLACE TEMP TABLE s1_batch_keys AS {s1_sql}\")\n",
386
+ "\n",
387
+ " joined = con.execute(f\"\"\"\n",
388
+ " WITH pair_hits AS (\n",
389
+ " SELECT s.entity_id AS s1_id, t.entity_id AS target_id, count(*) AS key_hits\n",
390
+ " FROM s1_batch_keys s\n",
391
+ " JOIN filtered_target_keys t ON s.key = t.key\n",
392
+ " GROUP BY 1, 2\n",
393
+ " ),\n",
394
+ " ranked AS (\n",
395
+ " SELECT s1_id, target_id, key_hits,\n",
396
+ " row_number() OVER(PARTITION BY s1_id ORDER BY key_hits DESC, target_id) AS rank\n",
397
+ " FROM pair_hits\n",
398
+ " )\n",
399
+ " SELECT s1_id, target_id, key_hits\n",
400
+ " FROM ranked\n",
401
+ " WHERE rank <= {top_k};\n",
402
+ " \"\"\").fetchdf()\n",
403
+ "\n",
404
+ " con.unregister(\"s1_batch\")\n",
405
+ " candidates_list.append(joined)\n",
406
+ " print(f\" Batch {b+1}/{n_batches}: {len(s1_batch):,} S1 -> {len(joined):,} candidate pairs\")\n",
407
+ "\n",
408
+ " con.unregister(\"targets_country\")\n",
409
+ " result = pd.concat(candidates_list, ignore_index=True) if candidates_list else pd.DataFrame(columns=[\"s1_id\", \"target_id\", \"key_hits\"])\n",
410
+ " print(f\"Country {country} finished in {time.time()-t_start:.1f}s. Total candidate pairs: {len(result):,}\")\n",
411
+ " return result\n"
412
+ ],
413
+ "execution_count": null,
414
+ "outputs": []
415
+ },
416
+ {
417
+ "cell_type": "markdown",
418
+ "id": "c4385c2d",
419
+ "metadata": {},
420
+ "source": [
421
+ "## 4.1 Run Blocking on Training Set\n",
422
+ "We normalize S2 and S3 per-country on demand to ensure lean memory usage.\n"
423
+ ]
424
+ },
425
+ {
426
+ "cell_type": "code",
427
+ "id": "ec3db4dc",
428
+ "metadata": {},
429
+ "source": [
430
+ "countries = train_s1[\"country\"].unique().tolist()\n",
431
+ "candidate_frames = []\n",
432
+ "\n",
433
+ "for ctry in countries:\n",
434
+ " cap = FREQ_CAP_FRANCE if ctry == \"France\" else FREQ_CAP_DEFAULT\n",
435
+ " \n",
436
+ " # Filter S2 and S3 for this country and normalize on demand\n",
437
+ " s1_c = train_s1[train_s1[\"country\"] == ctry]\n",
438
+ " s2_c = add_normalized_columns(train_s2[train_s2[\"country\"] == ctry])\n",
439
+ " s3_c = add_normalized_columns(train_s3[train_s3[\"country\"] == ctry])\n",
440
+ " targets_c = pd.concat([s2_c, s3_c], ignore_index=True)\n",
441
+ "\n",
442
+ " cands = run_blocking_for_country(s1_c, targets_c, ctry, cap, TOP_K_CANDIDATES)\n",
443
+ " candidate_frames.append(cands)\n",
444
+ " del s2_c, s3_c, targets_c\n",
445
+ " gc.collect()\n",
446
+ "\n",
447
+ "candidate_pairs = pd.concat(candidate_frames, ignore_index=True)\n",
448
+ "print(f\"\\n>>> Total Training Candidate Pairs Generated: {len(candidate_pairs):,} <<<\")\n"
449
+ ],
450
+ "execution_count": null,
451
+ "outputs": []
452
+ },
453
+ {
454
+ "cell_type": "markdown",
455
+ "id": "56b67ec4",
456
+ "metadata": {},
457
+ "source": [
458
+ "## 5. Mandatory Blocking Recall Ceiling Diagnostic\n",
459
+ "Verifies the true candidate recall ceiling before model training.\n",
460
+ "Any silent failure in blocking will be detected here immediately.\n"
461
+ ]
462
+ },
463
+ {
464
+ "cell_type": "code",
465
+ "id": "84946a80",
466
+ "metadata": {},
467
+ "source": [
468
+ "def measure_blocking_recall(candidate_df, gt_df):\n",
469
+ " gt_pairs = set()\n",
470
+ " for _, row in gt_df.iterrows():\n",
471
+ " s1 = row[\"source1_entity_id\"]\n",
472
+ " matched = str(row.get(\"matched_entity_ids\", \"\") or \"\").strip()\n",
473
+ " if matched and matched != \"nan\":\n",
474
+ " for tid in matched.split(\",\"):\n",
475
+ " tid_clean = tid.strip()\n",
476
+ " if tid_clean:\n",
477
+ " gt_pairs.add((s1, tid_clean))\n",
478
+ "\n",
479
+ " cand_set = set(zip(candidate_df[\"s1_id\"], candidate_df[\"target_id\"]))\n",
480
+ " recalled = gt_pairs & cand_set\n",
481
+ "\n",
482
+ " recall = len(recalled) / len(gt_pairs) if gt_pairs else 0.0\n",
483
+ " print(f\"True entity pairs in Ground Truth : {len(gt_pairs):,}\")\n",
484
+ " print(f\"True entity pairs captured by blocking : {len(recalled):,}\")\n",
485
+ " print(f\"BLOCKING RECALL CEILING: {recall:.4%}\")\n",
486
+ " return recall, gt_pairs\n",
487
+ "\n",
488
+ "recall_ceiling, gt_pairs_all = measure_blocking_recall(candidate_pairs, train_gt)\n",
489
+ "assert recall_ceiling >= 0.85, f\"Recall ceiling {recall_ceiling:.2%} is below 85%! Check blocking rules.\"\n",
490
+ "print(\"SUCCESS: Blocking recall check passed!\")\n"
491
+ ],
492
+ "execution_count": null,
493
+ "outputs": []
494
+ },
495
+ {
496
+ "cell_type": "markdown",
497
+ "id": "d66d92ba",
498
+ "metadata": {},
499
+ "source": [
500
+ "## 6. Bi-Encoder Neural Embeddings (`Qwen3-Embedding-0.6B`)\n",
501
+ "To fit within Kaggle execution limits:\n",
502
+ "1. **Deduplication to Unique Entities**: Each entity is embedded only **once** and cached by ID.\n",
503
+ "2. **Matryoshka Truncation**: Truncated to 256 dimensions for fast dot-product cosine similarity.\n",
504
+ "3. **Multi-GPU Parallelism**: Automatically leverages `torch.nn.DataParallel` when 2x GPUs are available.\n"
505
+ ]
506
+ },
507
+ {
508
+ "cell_type": "code",
509
+ "id": "a88c001c",
510
+ "metadata": {},
511
+ "source": [
512
+ "from transformers import AutoTokenizer, AutoModel\n",
513
+ "\n",
514
+ "class EntityEmbedder:\n",
515
+ " def __init__(self, model_name=\"Qwen/Qwen3-Embedding-0.6B\", embed_dim=256, max_len=64):\n",
516
+ " self.model_name = model_name\n",
517
+ " self.embed_dim = embed_dim\n",
518
+ " self.max_len = max_len\n",
519
+ " self.embed_cache = {}\n",
520
+ " \n",
521
+ " print(f\"Loading tokenizer and model: {model_name}...\")\n",
522
+ " self.tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)\n",
523
+ " self.model = AutoModel.from_pretrained(model_name, torch_dtype=torch.float16, trust_remote_code=True)\n",
524
+ " \n",
525
+ " if torch.cuda.device_count() > 1:\n",
526
+ " print(f\"Using DataParallel across {torch.cuda.device_count()} GPUs\")\n",
527
+ " self.model = nn.DataParallel(self.model)\n",
528
+ " \n",
529
+ " self.model = self.model.to(device).eval()\n",
530
+ " print(\"Embedder ready.\")\n",
531
+ "\n",
532
+ " def format_record(self, name, addr, country):\n",
533
+ " return f\"Business name: {name} | Address: {addr} | Country: {country}\"\n",
534
+ "\n",
535
+ " @torch.inference_mode()\n",
536
+ " def embed_entities(self, df_entities, batch_size=256 if torch.cuda.device_count() >= 2 else 64):\n",
537
+ " \"\"\"Embeds unique entities that are not already present in the cache.\"\"\"\n",
538
+ " new_entities = df_entities[~df_entities[\"entity_id\"].isin(self.embed_cache)].drop_duplicates(subset=\"entity_id\")\n",
539
+ " if len(new_entities) == 0:\n",
540
+ " return\n",
541
+ "\n",
542
+ " texts = [self.format_record(r.name_clean, r.addr_clean, r.country) for r in new_entities.itertuples()]\n",
543
+ " e_ids = new_entities[\"entity_id\"].tolist()\n",
544
+ " \n",
545
+ " print(f\"Embedding {len(texts):,} new unique entities...\")\n",
546
+ " t0 = time.time()\n",
547
+ " \n",
548
+ " for i in range(0, len(texts), batch_size):\n",
549
+ " b_texts = texts[i : i + batch_size]\n",
550
+ " b_ids = e_ids[i : i + batch_size]\n",
551
+ " \n",
552
+ " enc = self.tokenizer(b_texts, padding=True, truncation=True, max_length=self.max_len, return_tensors=\"pt\").to(device)\n",
553
+ " out = self.model(**enc)\n",
554
+ " last_hidden = out.last_hidden_state if hasattr(out, \"last_hidden_state\") else out[0]\n",
555
+ " mask = enc[\"attention_mask\"].unsqueeze(-1).float()\n",
556
+ " pooled = (last_hidden * mask).sum(1) / mask.sum(1).clamp(min=1e-9)\n",
557
+ " pooled = pooled[:, :self.embed_dim]\n",
558
+ " pooled = F.normalize(pooled, p=2, dim=1)\n",
559
+ " \n",
560
+ " vecs = pooled.float().cpu().numpy()\n",
561
+ " for eid, v in zip(b_ids, vecs):\n",
562
+ " self.embed_cache[eid] = v\n",
563
+ "\n",
564
+ " if (i // batch_size) % 100 == 0 and i > 0:\n",
565
+ " rate = i / max(time.time() - t0, 1e-9)\n",
566
+ " print(f\" Progress: {i:,}/{len(texts):,} ({rate:.0f} entities/s)\")\n",
567
+ " \n",
568
+ " print(f\"Completed in {time.time()-t0:.1f}s. Total cached entities: {len(self.embed_cache):,}\")\n",
569
+ "\n",
570
+ " def compute_cosine_similarity(self, s1_ids, target_ids):\n",
571
+ " \"\"\"Fast dot product lookup between cached L2-normalized embeddings.\"\"\"\n",
572
+ " s1_vecs = np.stack([self.embed_cache[i] for i in s1_ids])\n",
573
+ " tgt_vecs = np.stack([self.embed_cache[i] for i in target_ids])\n",
574
+ " return (s1_vecs * tgt_vecs).sum(axis=1)\n",
575
+ "\n",
576
+ "embedder = EntityEmbedder()\n"
577
+ ],
578
+ "execution_count": null,
579
+ "outputs": []
580
+ },
581
+ {
582
+ "cell_type": "markdown",
583
+ "id": "8e504ee9",
584
+ "metadata": {},
585
+ "source": [
586
+ "## 7. Fast Feature Extraction\n",
587
+ "Combines lexical similarity signals (RapidFuzz), structural signals (House/Postal match),\n",
588
+ "blocking frequency (`key_hits`), and the neural semantic similarity (`qwen_cosine_sim`).\n",
589
+ "Processed in batches to guarantee zero OOM risk.\n"
590
+ ]
591
+ },
592
+ {
593
+ "cell_type": "code",
594
+ "id": "da83c2f1",
595
+ "metadata": {},
596
+ "source": [
597
+ "from rapidfuzz import fuzz\n",
598
+ "\n",
599
+ "def extract_features_batched(candidate_df, s1_lookup_df, target_lookup_df, embedder, batch_size=100_000):\n",
600
+ " \"\"\"Extracts features in chunks to prevent large intermediate DataFrame allocations.\"\"\"\n",
601
+ " s1_dict = s1_lookup_df.set_index(\"entity_id\")[[\"name_clean\", \"name_root\", \"addr_clean\", \"house_no\", \"postal\", \"country\"]].to_dict(\"index\")\n",
602
+ " tgt_dict = target_lookup_df.set_index(\"entity_id\")[[\"name_clean\", \"name_root\", \"addr_clean\", \"house_no\", \"postal\", \"country\"]].to_dict(\"index\")\n",
603
+ "\n",
604
+ " # Prepare embedding universe\n",
605
+ " needed_eids = list(set(candidate_df[\"s1_id\"]) | set(candidate_df[\"target_id\"]))\n",
606
+ " missing_eids = [eid for eid in needed_eids if eid not in embedder.embed_cache]\n",
607
+ " if missing_eids:\n",
608
+ " records = []\n",
609
+ " for eid in missing_eids:\n",
610
+ " row = s1_dict.get(eid) or tgt_dict.get(eid)\n",
611
+ " if row:\n",
612
+ " records.append({\"entity_id\": eid, \"name_clean\": row[\"name_clean\"], \"addr_clean\": row[\"addr_clean\"], \"country\": row[\"country\"]})\n",
613
+ " if records:\n",
614
+ " embedder.embed_entities(pd.DataFrame(records))\n",
615
+ "\n",
616
+ " feature_chunks = []\n",
617
+ " n_chunks = (len(candidate_df) + batch_size - 1) // batch_size\n",
618
+ " print(f\"Extracting features for {len(candidate_df):,} pairs in {n_chunks} chunks...\")\n",
619
+ "\n",
620
+ " for i in range(n_chunks):\n",
621
+ " chunk = candidate_df.iloc[i * batch_size : (i + 1) * batch_size]\n",
622
+ " s1_ids = chunk[\"s1_id\"].tolist()\n",
623
+ " tgt_ids = chunk[\"target_id\"].tolist()\n",
624
+ "\n",
625
+ " qwen_sim = embedder.compute_cosine_similarity(s1_ids, tgt_ids)\n",
626
+ "\n",
627
+ " name_ratios, name_tsorts, name_tsets, name_partials, root_ratios = [], [], [], [], []\n",
628
+ " addr_ratios, addr_tsorts, addr_tsets, addr_partials = [], [], [], []\n",
629
+ " house_matches, postal_matches, postal_conflicts = [], [], []\n",
630
+ "\n",
631
+ " for s, t in zip(s1_ids, tgt_ids):\n",
632
+ " s_row = s1_dict[s]\n",
633
+ " t_row = tgt_dict[t]\n",
634
+ "\n",
635
+ " sn, tn = s_row[\"name_clean\"], t_row[\"name_clean\"]\n",
636
+ " sa, ta = s_row[\"addr_clean\"], t_row[\"addr_clean\"]\n",
637
+ " sr, tr = s_row[\"name_root\"], t_row[\"name_root\"]\n",
638
+ "\n",
639
+ " name_ratios.append(fuzz.ratio(sn, tn))\n",
640
+ " name_tsorts.append(fuzz.token_sort_ratio(sn, tn))\n",
641
+ " name_tsets.append(fuzz.token_set_ratio(sn, tn))\n",
642
+ " name_partials.append(fuzz.partial_ratio(sn, tn))\n",
643
+ " root_ratios.append(fuzz.ratio(sr, tr))\n",
644
+ "\n",
645
+ " addr_ratios.append(fuzz.ratio(sa, ta))\n",
646
+ " addr_tsorts.append(fuzz.token_sort_ratio(sa, ta))\n",
647
+ " addr_tsets.append(fuzz.token_set_ratio(sa, ta))\n",
648
+ " addr_partials.append(fuzz.partial_ratio(sa, ta))\n",
649
+ "\n",
650
+ " sh, th = s_row[\"house_no\"], t_row[\"house_no\"]\n",
651
+ " sp, tp = s_row[\"postal\"], t_row[\"postal\"]\n",
652
+ "\n",
653
+ " house_matches.append(1 if (sh and th and sh == th) else 0)\n",
654
+ " postal_matches.append(1 if (sp and tp and sp == tp) else 0)\n",
655
+ " postal_conflicts.append(1 if (sp and tp and sp != tp) else 0)\n",
656
+ "\n",
657
+ " df_chunk = pd.DataFrame({\n",
658
+ " \"s1_id\": s1_ids,\n",
659
+ " \"target_id\": tgt_ids,\n",
660
+ " \"key_hits\": chunk[\"key_hits\"].values,\n",
661
+ " \"qwen_cosine_sim\": qwen_sim,\n",
662
+ " \"name_ratio\": name_ratios,\n",
663
+ " \"name_tsort\": name_tsorts,\n",
664
+ " \"name_tset\": name_tsets,\n",
665
+ " \"name_partial\": name_partials,\n",
666
+ " \"root_ratio\": root_ratios,\n",
667
+ " \"addr_ratio\": addr_ratios,\n",
668
+ " \"addr_tsort\": addr_tsorts,\n",
669
+ " \"addr_tset\": addr_tsets,\n",
670
+ " \"addr_partial\": addr_partials,\n",
671
+ " \"house_match\": house_matches,\n",
672
+ " \"postal_match\": postal_matches,\n",
673
+ " \"postal_conflict\": postal_conflicts,\n",
674
+ " })\n",
675
+ " feature_chunks.append(df_chunk)\n",
676
+ "\n",
677
+ " return pd.concat(feature_chunks, ignore_index=True)\n",
678
+ "\n",
679
+ "# Prepare combined target reference\n",
680
+ "targets_all_norm = pd.concat([\n",
681
+ " add_normalized_columns(train_s2[train_s2[\"entity_id\"].isin(set(candidate_pairs[\"target_id\"]))]),\n",
682
+ " add_normalized_columns(train_s3[train_s3[\"entity_id\"].isin(set(candidate_pairs[\"target_id\"]))]),\n",
683
+ "], ignore_index=True).drop_duplicates(subset=\"entity_id\")\n",
684
+ "\n",
685
+ "training_features = extract_features_batched(candidate_pairs, train_s1, targets_all_norm, embedder)\n",
686
+ "print(\"Extracted training features shape:\", training_features.shape)\n"
687
+ ],
688
+ "execution_count": null,
689
+ "outputs": []
690
+ },
691
+ {
692
+ "cell_type": "markdown",
693
+ "id": "52e4f748",
694
+ "metadata": {},
695
+ "source": [
696
+ "## 8. LightGBM Classifier & Corrected Cross-Validation\n",
697
+ "Key Fix: `macro_f05` evaluates **only** the entities present in the current validation fold / country,\n",
698
+ "eliminating the 5x score suppression bug.\n"
699
+ ]
700
+ },
701
+ {
702
+ "cell_type": "code",
703
+ "id": "a4b8b502",
704
+ "metadata": {},
705
+ "source": [
706
+ "import lightgbm as lgb\n",
707
+ "from sklearn.model_selection import GroupKFold\n",
708
+ "\n",
709
+ "# Build labels\n",
710
+ "gt_lookup = defaultdict(set)\n",
711
+ "for _, row in train_gt.iterrows():\n",
712
+ " s1 = row[\"source1_entity_id\"]\n",
713
+ " m = str(row.get(\"matched_entity_ids\", \"\") or \"\").strip()\n",
714
+ " if m and m != \"nan\":\n",
715
+ " for t in m.split(\",\"):\n",
716
+ " gt_lookup[s1].add(t.strip())\n",
717
+ "\n",
718
+ "training_features[\"label\"] = [\n",
719
+ " 1 if t in gt_lookup[s] else 0\n",
720
+ " for s, t in zip(training_features[\"s1_id\"], training_features[\"target_id\"])\n",
721
+ "]\n",
722
+ "\n",
723
+ "FEATURE_COLS = [c for c in training_features.columns if c not in (\"s1_id\", \"target_id\", \"label\")]\n",
724
+ "X = training_features[FEATURE_COLS].astype(float)\n",
725
+ "y = training_features[\"label\"].values\n",
726
+ "groups = training_features[\"s1_id\"].values\n",
727
+ "\n",
728
+ "n_pos, n_neg = (y == 1).sum(), (y == 0).sum()\n",
729
+ "scale_pos_weight = n_neg / max(n_pos, 1)\n",
730
+ "print(f\"Positives: {n_pos:,} | Negatives: {n_neg:,} | scale_pos_weight: {scale_pos_weight:.2f}\")\n",
731
+ "\n",
732
+ "def evaluate_macro_f05(eval_s1_ids, cand_s1_ids, cand_target_ids, probs, threshold, gt_map):\n",
733
+ " \"\"\"Accurately computes Macro F0.5 strictly for the specified eval_s1_ids.\"\"\"\n",
734
+ " preds = defaultdict(set)\n",
735
+ " for s, t, p in zip(cand_s1_ids, cand_target_ids, probs):\n",
736
+ " if p >= threshold:\n",
737
+ " preds[s].add(t)\n",
738
+ "\n",
739
+ " scores = []\n",
740
+ " for s in eval_s1_ids:\n",
741
+ " pred_set = preds.get(s, set())\n",
742
+ " true_set = gt_map.get(s, set())\n",
743
+ " if not pred_set and not true_set:\n",
744
+ " scores.append(1.0)\n",
745
+ " continue\n",
746
+ " tp = len(pred_set & true_set)\n",
747
+ " prec = tp / len(pred_set) if pred_set else 0.0\n",
748
+ " rec = tp / len(true_set) if true_set else 0.0\n",
749
+ " if (0.25 * prec + rec) > 0:\n",
750
+ " f05 = (1.25 * prec * rec) / (0.25 * prec + rec)\n",
751
+ " else:\n",
752
+ " f05 = 0.0\n",
753
+ " scores.append(f05)\n",
754
+ " return float(np.mean(scores))\n",
755
+ "\n",
756
+ "gkf = GroupKFold(n_splits=5)\n",
757
+ "oof_pred = np.zeros(len(X))\n",
758
+ "fold_scores = []\n",
759
+ "\n",
760
+ "for fold, (train_idx, val_idx) in enumerate(gkf.split(X, y, groups)):\n",
761
+ " train_set = lgb.Dataset(X.iloc[train_idx], label=y[train_idx])\n",
762
+ " val_set = lgb.Dataset(X.iloc[val_idx], label=y[val_idx])\n",
763
+ " \n",
764
+ " params = dict(\n",
765
+ " objective=\"binary\",\n",
766
+ " metric=\"binary_logloss\",\n",
767
+ " scale_pos_weight=scale_pos_weight,\n",
768
+ " num_leaves=63,\n",
769
+ " learning_rate=0.05,\n",
770
+ " n_estimators=500,\n",
771
+ " n_jobs=-1,\n",
772
+ " verbosity=-1\n",
773
+ " )\n",
774
+ " booster = lgb.train(\n",
775
+ " params, train_set, valid_sets=[val_set],\n",
776
+ " callbacks=[lgb.early_stopping(30, verbose=False)]\n",
777
+ " )\n",
778
+ " val_probs = booster.predict(X.iloc[val_idx])\n",
779
+ " oof_pred[val_idx] = val_probs\n",
780
+ "\n",
781
+ " val_s1 = set(groups[val_idx])\n",
782
+ " f05 = evaluate_macro_f05(val_s1, groups[val_idx], training_features[\"target_id\"].values[val_idx], val_probs, 0.50, gt_lookup)\n",
783
+ " fold_scores.append(f05)\n",
784
+ " print(f\"Fold {fold+1}: Macro F0.5 @ 0.50 = {f05:.4f}\")\n",
785
+ "\n",
786
+ "print(f\"\\n>>> 5-Fold Mean Macro F0.5: {np.mean(fold_scores):.4f} +/- {np.std(fold_scores):.4f} <<<\")\n"
787
+ ],
788
+ "execution_count": null,
789
+ "outputs": []
790
+ },
791
+ {
792
+ "cell_type": "markdown",
793
+ "id": "3e8638ca",
794
+ "metadata": {},
795
+ "source": [
796
+ "## 9. Per-Country Threshold Sweeping\n",
797
+ "Optimizes decision thresholds individually for France, US, and India on the OOF predictions.\n"
798
+ ]
799
+ },
800
+ {
801
+ "cell_type": "code",
802
+ "id": "455def1b",
803
+ "metadata": {},
804
+ "source": [
805
+ "s1_country_map = train_s1.set_index(\"entity_id\")[\"country\"].to_dict()\n",
806
+ "candidate_countries = np.array([s1_country_map[s] for s in groups])\n",
807
+ "\n",
808
+ "per_country_threshold = {}\n",
809
+ "for ctry in np.unique(candidate_countries):\n",
810
+ " mask = candidate_countries == ctry\n",
811
+ " ctry_s1 = set(groups[mask])\n",
812
+ " \n",
813
+ " best_t, best_score = 0.50, -1.0\n",
814
+ " for t in np.arange(0.30, 0.951, 0.02):\n",
815
+ " score = evaluate_macro_f05(\n",
816
+ " ctry_s1,\n",
817
+ " groups[mask],\n",
818
+ " training_features[\"target_id\"].values[mask],\n",
819
+ " oof_pred[mask],\n",
820
+ " t,\n",
821
+ " gt_lookup\n",
822
+ " )\n",
823
+ " if score > best_score:\n",
824
+ " best_score, best_t = score, t\n",
825
+ " \n",
826
+ " per_country_threshold[ctry] = round(float(best_t), 2)\n",
827
+ " print(f\"Country [{ctry}]: Optimal Threshold = {best_t:.2f} (Macro F0.5 = {best_score:.4f})\")\n",
828
+ "\n",
829
+ "per_country_threshold[\"__default__\"] = 0.50\n",
830
+ "print(\"\\nConfigured Decision Thresholds:\", per_country_threshold)\n"
831
+ ],
832
+ "execution_count": null,
833
+ "outputs": []
834
+ },
835
+ {
836
+ "cell_type": "markdown",
837
+ "id": "806a7056",
838
+ "metadata": {},
839
+ "source": [
840
+ "## 10. Final Model Training & Artifact Persistence\n"
841
+ ]
842
+ },
843
+ {
844
+ "cell_type": "code",
845
+ "id": "f3f967cb",
846
+ "metadata": {},
847
+ "source": [
848
+ "full_train_set = lgb.Dataset(X, label=y)\n",
849
+ "final_params = dict(\n",
850
+ " objective=\"binary\",\n",
851
+ " metric=\"binary_logloss\",\n",
852
+ " scale_pos_weight=scale_pos_weight,\n",
853
+ " num_leaves=63,\n",
854
+ " learning_rate=0.05,\n",
855
+ " n_estimators=500,\n",
856
+ " n_jobs=-1,\n",
857
+ " verbosity=-1\n",
858
+ ")\n",
859
+ "final_model = lgb.train(final_params, full_train_set)\n",
860
+ "final_model.save_model(str(CACHE_DIR / \"model.lgb\"))\n",
861
+ "\n",
862
+ "with open(CACHE_DIR / \"model_meta.json\", \"w\") as f:\n",
863
+ " json.dump({\n",
864
+ " \"feature_cols\": FEATURE_COLS,\n",
865
+ " \"per_country_threshold\": per_country_threshold,\n",
866
+ " \"scale_pos_weight\": float(scale_pos_weight)\n",
867
+ " }, f, indent=2)\n",
868
+ "\n",
869
+ "print(\"Saved final model and metadata to\", CACHE_DIR)\n"
870
+ ],
871
+ "execution_count": null,
872
+ "outputs": []
873
+ },
874
+ {
875
+ "cell_type": "markdown",
876
+ "id": "6d3e876d",
877
+ "metadata": {},
878
+ "source": [
879
+ "## 11. Full End-to-End Test Set Inference\n",
880
+ "Executes multi-view blocking, Qwen bi-encoder neural embeddings, and feature extraction\n",
881
+ "across `test_source1`, `test_source2`, and `test_source3`.\n"
882
+ ]
883
+ },
884
+ {
885
+ "cell_type": "code",
886
+ "id": "a35939da",
887
+ "metadata": {},
888
+ "source": [
889
+ "print(\"Loading test tables...\")\n",
890
+ "test_s1 = load_tsv(\"test_source1.tsv\")\n",
891
+ "test_s2 = load_tsv(\"test_source2.tsv\")\n",
892
+ "test_s3 = load_tsv(\"test_source3.tsv\")\n",
893
+ "\n",
894
+ "print(\"Normalizing test_s1...\")\n",
895
+ "test_s1 = add_normalized_columns(test_s1)\n",
896
+ "\n",
897
+ "test_candidate_frames = []\n",
898
+ "for ctry in test_s1[\"country\"].unique():\n",
899
+ " cap = FREQ_CAP_FRANCE if ctry == \"France\" else FREQ_CAP_DEFAULT\n",
900
+ " s1_c = test_s1[test_s1[\"country\"] == ctry]\n",
901
+ " s2_c = add_normalized_columns(test_s2[test_s2[\"country\"] == ctry])\n",
902
+ " s3_c = add_normalized_columns(test_s3[test_s3[\"country\"] == ctry])\n",
903
+ " targets_c = pd.concat([s2_c, s3_c], ignore_index=True)\n",
904
+ "\n",
905
+ " cands = run_blocking_for_country(s1_c, targets_c, ctry, cap, TOP_K_CANDIDATES)\n",
906
+ " test_candidate_frames.append(cands)\n",
907
+ " del s2_c, s3_c, targets_c\n",
908
+ " gc.collect()\n",
909
+ "\n",
910
+ "test_candidate_pairs = pd.concat(test_candidate_frames, ignore_index=True)\n",
911
+ "print(f\"\\nTotal Test Candidate Pairs: {len(test_candidate_pairs):,}\")\n",
912
+ "\n",
913
+ "# Prepare test targets reference\n",
914
+ "test_needed_targets = set(test_candidate_pairs[\"target_id\"])\n",
915
+ "test_targets_all = pd.concat([\n",
916
+ " add_normalized_columns(test_s2[test_s2[\"entity_id\"].isin(test_needed_targets)]),\n",
917
+ " add_normalized_columns(test_s3[test_s3[\"entity_id\"].isin(test_needed_targets)])\n",
918
+ "], ignore_index=True).drop_duplicates(subset=\"entity_id\")\n",
919
+ "\n",
920
+ "test_features = extract_features_batched(test_candidate_pairs, test_s1, test_targets_all, embedder)\n",
921
+ "\n",
922
+ "# Predict probabilities\n",
923
+ "test_probs = final_model.predict(test_features[FEATURE_COLS].astype(float))\n",
924
+ "test_features[\"prob\"] = test_probs\n",
925
+ "\n",
926
+ "# Apply per-country threshold\n",
927
+ "test_s1_ctry = test_s1.set_index(\"entity_id\")[\"country\"].to_dict()\n",
928
+ "test_features[\"country\"] = test_features[\"s1_id\"].map(test_s1_ctry)\n",
929
+ "test_features[\"threshold\"] = test_features[\"country\"].map(lambda c: per_country_threshold.get(c, per_country_threshold[\"__default__\"]))\n",
930
+ "test_features[\"above_threshold\"] = test_features[\"prob\"] >= test_features[\"threshold\"]\n",
931
+ "\n",
932
+ "n_above = test_features[\"above_threshold\"].sum()\n",
933
+ "print(f\"Candidate pairs above confidence threshold: {n_above:,} ({n_above/len(test_features):.1%})\")\n"
934
+ ],
935
+ "execution_count": null,
936
+ "outputs": []
937
+ },
938
+ {
939
+ "cell_type": "markdown",
940
+ "id": "909a5f79",
941
+ "metadata": {},
942
+ "source": [
943
+ "## 12. 1:Many Target-Exclusivity Resolution (Fixed)\n",
944
+ "**Ground Truth Rule**:\n",
945
+ "- Each Target entity belongs to **at most one** S1 entity.\n",
946
+ "- Each S1 entity can belong to **multiple** Target entities (across S2 and S3).\n",
947
+ "\n",
948
+ "We sort candidate pairs above threshold by confidence probability descending, and retain the\n",
949
+ "single highest-confidence S1 entity for every matched target.\n"
950
+ ]
951
+ },
952
+ {
953
+ "cell_type": "code",
954
+ "id": "42a24012",
955
+ "metadata": {},
956
+ "source": [
957
+ "candidates_above = test_features[test_features[\"above_threshold\"]][[\"s1_id\", \"target_id\", \"prob\"]]\n",
958
+ "\n",
959
+ "# Resolve 1:Many target exclusivity\n",
960
+ "resolved = (candidates_above\n",
961
+ " .sort_values(\"prob\", ascending=False)\n",
962
+ " .drop_duplicates(subset=[\"target_id\"]))\n",
963
+ "\n",
964
+ "print(f\"Resolved {len(resolved):,} matching links across {resolved['s1_id'].nunique():,} unique S1 entities.\")\n"
965
+ ],
966
+ "execution_count": null,
967
+ "outputs": []
968
+ },
969
+ {
970
+ "cell_type": "markdown",
971
+ "id": "3cde9178",
972
+ "metadata": {},
973
+ "source": [
974
+ "## 13. Build Official Submission Files\n",
975
+ "Generates `matching_results.tsv` and `candidate_pairs.tsv` following the exact challenge schema:\n",
976
+ "- Every `test_source1` entity is represented exactly once in `matching_results.tsv`.\n",
977
+ "- Candidate pairs are exported in `candidate_pairs.tsv`.\n"
978
+ ]
979
+ },
980
+ {
981
+ "cell_type": "code",
982
+ "id": "38fcf2db",
983
+ "metadata": {},
984
+ "source": [
985
+ "# Build matches map: S1 -> comma-separated sorted targets\n",
986
+ "matches_by_s1 = resolved.groupby(\"s1_id\")[\"target_id\"].apply(lambda ids: \",\".join(sorted(ids))).to_dict()\n",
987
+ "\n",
988
+ "all_test_s1_ids = test_s1[\"entity_id\"].tolist()\n",
989
+ "matching_results = pd.DataFrame({\n",
990
+ " \"source1_entity_id\": all_test_s1_ids,\n",
991
+ " \"matched_entity_ids\": [matches_by_s1.get(sid, \"\") for sid in all_test_s1_ids]\n",
992
+ "})\n",
993
+ "\n",
994
+ "n_singletons = (matching_results[\"matched_entity_ids\"] == \"\").sum()\n",
995
+ "n_matched = len(matching_results) - n_singletons\n",
996
+ "print(f\"Test Entities: {len(matching_results):,}\")\n",
997
+ "print(f\" Singletons (no match): {n_singletons:,} ({n_singletons/len(matching_results):.1%})\")\n",
998
+ "print(f\" Matched Entities : {n_matched:,} ({n_matched/len(matching_results):.1%})\")\n",
999
+ "\n",
1000
+ "assert len(matching_results) == len(all_test_s1_ids), \"Row count mismatch!\"\n",
1001
+ "assert matching_results[\"source1_entity_id\"].is_unique, \"Duplicate S1 rows detected!\"\n",
1002
+ "\n",
1003
+ "# Export matching results\n",
1004
+ "out_match_path = OUTPUT_DIR / \"matching_results.tsv\"\n",
1005
+ "matching_results.to_csv(out_match_path, sep=\"\\t\", index=False)\n",
1006
+ "print(\"Wrote matching_results.tsv to:\", out_match_path)\n",
1007
+ "\n",
1008
+ "# Export candidate pairs\n",
1009
+ "out_cand_path = OUTPUT_DIR / \"candidate_pairs.tsv\"\n",
1010
+ "cand_export = test_candidate_pairs[[\"s1_id\", \"target_id\"]].rename(\n",
1011
+ " columns={\"s1_id\": \"source1_entity_id\", \"target_id\": \"matched_entity_id\"}\n",
1012
+ ")\n",
1013
+ "cand_export.to_csv(out_cand_path, sep=\"\\t\", index=False)\n",
1014
+ "print(\"Wrote candidate_pairs.tsv to:\", out_cand_path)\n"
1015
+ ],
1016
+ "execution_count": null,
1017
+ "outputs": []
1018
+ },
1019
+ {
1020
+ "cell_type": "markdown",
1021
+ "id": "52a6c3fc",
1022
+ "metadata": {},
1023
+ "source": [
1024
+ "## 14. Run Submission Validator\n",
1025
+ "Executes `utils/validate_submission.py` to confirm total compliance with the competition specification.\n"
1026
+ ]
1027
+ },
1028
+ {
1029
+ "cell_type": "code",
1030
+ "id": "bc2c9715",
1031
+ "metadata": {},
1032
+ "source": [
1033
+ "import subprocess\n",
1034
+ "\n",
1035
+ "validator_script = None\n",
1036
+ "for cand in [Path(\"utils/validate_submission.py\"), DATA_DIR / \"utils\" / \"validate_submission.py\"]:\n",
1037
+ " if cand.exists():\n",
1038
+ " validator_script = cand\n",
1039
+ " break\n",
1040
+ "\n",
1041
+ "if validator_script:\n",
1042
+ " test_dir_arg = DATA_DIR if (DATA_DIR / \"test_source1.tsv\").exists() else (DATA_DIR / \"test\")\n",
1043
+ " print(f\"Running official validator using {validator_script}...\")\n",
1044
+ " res = subprocess.run([\n",
1045
+ " sys.executable, str(validator_script),\n",
1046
+ " \"--output_dir\", str(OUTPUT_DIR),\n",
1047
+ " \"--test_dir\", str(test_dir_arg)\n",
1048
+ " ], capture_output=True, text=True)\n",
1049
+ " print(res.stdout)\n",
1050
+ " if res.stderr:\n",
1051
+ " print(res.stderr)\n",
1052
+ " if res.returncode == 0:\n",
1053
+ " print(\"\\n>>> SUBMISSION FULLY VALIDATED AND READY TO SUBMIT! <<<\")\n",
1054
+ " else:\n",
1055
+ " print(\"Validator returned non-zero code. Inspect messages above.\")\n",
1056
+ "else:\n",
1057
+ " print(\"validate_submission.py not found in local path \u2014 verify output files manually.\")\n"
1058
+ ],
1059
+ "execution_count": null,
1060
+ "outputs": []
1061
+ }
1062
+ ],
1063
+ "metadata": {
1064
+ "kernelspec": {
1065
+ "display_name": "Python 3",
1066
+ "language": "python",
1067
+ "name": "python3"
1068
+ },
1069
+ "language_info": {
1070
+ "name": "python",
1071
+ "version": "3.10"
1072
+ }
1073
+ },
1074
+ "nbformat": 4,
1075
+ "nbformat_minor": 5
1076
+ }