{ "cells": [ { "cell_type": "markdown", "id": "494afa14", "metadata": {}, "source": [ "# Option B \u2014 Hybrid Multi-View Blocking + Bi-Encoder Neural Embeddings\n", "### Business Entity Resolution \u2014 Amazon ML Challenge 2026\n", "\n", "This notebook implements the complete, production-ready Option B pipeline:\n", "1. **Universal Fast Transliteration & Accent Stripping** (Phonetic Indic table + NFKD + unidecode before key extraction)\n", "2. **DuckDB Multi-View Blocking**:\n", " - Both Source 2 and Source 3 targets combined into one index\n", " - Target keys and frequency caps indexed **once** per country (not re-run in inner loops)\n", " - S1 batched to avoid memory bottlenecks\n", " - Top-K candidate pairs filtering per entity\n", "3. **Mandatory Blocking Recall Ceiling Check**:\n", " - Accurately checks recall on ground truth to ensure zero candidate loss\n", "4. **Deduplicated Bi-Encoder Embeddings (Qwen3-Embedding-0.6B)**:\n", " - One embedding per **unique** entity (not per pair)\n", " - Multi-GPU DataParallel support (e.g. 2x T4 on Kaggle)\n", " - Persistent `EntityEmbedder` class available for both train and test phases\n", "5. **LightGBM Reranker with In-Fold-Only Metric Evaluation**:\n", " - `macro_f05` evaluates **only** the entities in the current validation fold (fixing CV skew)\n", " - Per-country threshold sweeps (France gets its own operating point)\n", "6. **End-to-End Test Set Inference**:\n", " - Full test blocking, embedding, feature extraction, and prediction\n", "7. **1:Many Target-Exclusivity Resolution**:\n", " - Each Target entity is assigned to at most one S1 (highest probability >= threshold)\n", " - Each S1 can match to multiple targets (across S2 and S3)\n", "8. **Submission Generation & Official Validation**:\n", " - Exports `matching_results.tsv` and `candidate_pairs.tsv`\n", " - Validates format with `utils/validate_submission.py`\n" ] }, { "cell_type": "code", "id": "fc4d836b", "metadata": {}, "source": [ "import os\n", "import sys\n", "import time\n", "import json\n", "import gc\n", "import re\n", "import unicodedata\n", "from pathlib import Path\n", "from collections import defaultdict\n", "\n", "import numpy as np\n", "import pandas as pd\n", "import duckdb\n", "import torch\n", "import torch.nn as nn\n", "import torch.nn.functional as F\n", "\n", "# ---- Data Directory Auto-Discovery ----\n", "# Checks Kaggle paths, standard local paths, or relative paths\n", "POSSIBLE_DATA_DIRS = [\n", " Path(\"/kaggle/input/amazon-ml-challenge-2026\"),\n", " Path(\"/kaggle/input/amazon-ml\"),\n", " Path(\"/kaggle/input/datasets/arghasaha1905/amazon-ml\"),\n", " Path(\"/kaggle/input/student-resource\"),\n", " Path(\"dataset\"),\n", " Path(\".\"),\n", "]\n", "\n", "DATA_DIR = None\n", "for p in POSSIBLE_DATA_DIRS:\n", " if p.exists() and (\n", " (p / \"train_source1.tsv\").exists() or\n", " (p / \"train\" / \"train_source1.tsv\").exists() or\n", " (p / \"test_source1.tsv\").exists() or\n", " (p / \"test\" / \"test_source1.tsv\").exists()\n", " ):\n", " DATA_DIR = p\n", " break\n", "\n", "if DATA_DIR is None:\n", " DATA_DIR = Path(\"/kaggle/input/amazon-ml-challenge-2026\")\n", "\n", "print(f\"Detected DATA_DIR: {DATA_DIR}\")\n", "\n", "# ---- Output & Cache Directories ----\n", "if Path(\"/kaggle/working\").exists():\n", " OUTPUT_DIR = Path(\"/kaggle/working/output\")\n", " CACHE_DIR = Path(\"/kaggle/working/cache\")\n", "else:\n", " OUTPUT_DIR = Path(\"output\")\n", " CACHE_DIR = Path(\"cache\")\n", "\n", "OUTPUT_DIR.mkdir(parents=True, exist_ok=True)\n", "CACHE_DIR.mkdir(parents=True, exist_ok=True)\n", "\n", "# ---- Run Configuration ----\n", "# Set SAMPLE_MODE = True to test on a fast sample (20k per country) first.\n", "# Set SAMPLE_MODE = False for the full 2.2M train + 1.73M test run.\n", "SAMPLE_MODE = True\n", "SAMPLE_N_PER_COUNTRY = 20_000\n", "TOP_K_CANDIDATES = 35\n", "FREQ_CAP_DEFAULT = 500\n", "FREQ_CAP_FRANCE = 1000\n", "\n", "# ---- Hardware & GPU Check ----\n", "n_gpus = torch.cuda.device_count()\n", "print(f\"CUDA available: {torch.cuda.is_available()} | GPU count: {n_gpus}\")\n", "for i in range(n_gpus):\n", " props = torch.cuda.get_device_properties(i)\n", " print(f\" GPU {i}: {props.name} | {props.total_memory / 1e9:.1f} GB\")\n", "\n", "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n", "print(f\"Using primary device: {device}\")\n" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "id": "6025bbce", "metadata": {}, "source": [ "## 1. High-Performance Text Normalization & Transliteration\n", "Indic scripts (Devanagari, Tamil, Telugu, Bengali, Kannada, Malayalam, etc.) and French diacritics\n", "must be mapped phonetically into Latin characters **before** generating blocking keys.\n", "We use a precompiled C-level `str.translate` lookup table with NFKD normalization, ensuring\n", "sub-microsecond execution per entity with zero empty-string wipes.\n" ] }, { "cell_type": "code", "id": "faa1babb", "metadata": {}, "source": [ "try:\n", " from unidecode import unidecode\n", "except ImportError:\n", " import subprocess\n", " subprocess.run([sys.executable, \"-m\", \"pip\", \"install\", \"unidecode\", \"-q\"])\n", " from unidecode import unidecode\n", "\n", "# Universal Indic & European phonetic transliteration map\n", "DEV_PHONETICS = {\n", " 0x05:'a',0x06:'a',0x07:'i',0x08:'i',0x09:'u',0x0A:'u',0x0B:'ri',0x0E:'e',0x0F:'e',0x10:'ai',0x12:'o',0x13:'o',0x14:'au',\n", " 0x15:'k',0x16:'kh',0x17:'g',0x18:'gh',0x19:'n',0x1A:'ch',0x1B:'chh',0x1C:'j',0x1D:'jh',0x1E:'n',\n", " 0x1F:'t',0x20:'th',0x21:'d',0x22:'dh',0x23:'n',0x24:'t',0x25:'th',0x26:'d',0x27:'dh',0x28:'n',\n", " 0x2A:'p',0x2B:'ph',0x2C:'b',0x2D:'bh',0x2E:'m',0x2F:'y',0x30:'r',0x31:'r',0x32:'l',0x33:'l',0x35:'v',\n", " 0x36:'sh',0x37:'sh',0x38:'s',0x39:'h',0x3E:'a',0x3F:'i',0x40:'i',0x41:'u',0x42:'u',0x43:'ri',\n", " 0x46:'e',0x47:'e',0x48:'ai',0x4A:'o',0x4B:'o',0x4C:'au',0x4D:'',0x02:'n',0x01:'n',0x03:'h'\n", "}\n", "TRANSLIT_MAP = {chr(base+off): lat for base in (0x0900,0x0980,0x0A00,0x0A80,0x0B00,0x0B80,0x0C00,0x0C80,0x0D00) for off, lat in DEV_PHONETICS.items()}\n", "TRANSLIT_MAP.update({\n", " '\u00e9':'e','\u00e8':'e','\u00ea':'e','\u00eb':'e','\u00e0':'a','\u00e2':'a','\u00e4':'a','\u00f4':'o','\u00f6':'o',\n", " '\u00ee':'i','\u00ef':'i','\u00f9':'u','\u00fb':'u','\u00fc':'u','\u00e7':'c','\u00f1':'n','\u00c9':'e','\u00c8':'e','\u00ca':'e','\u00cb':'e',\n", " '\u00c0':'a','\u00c2':'a','\u00c4':'a','\u00d4':'o','\u00d6':'o','\u00ce':'i','\u00cf':'i','\u00d9':'u','\u00db':'u','\u00dc':'u','\u00c7':'c','\u00d1':'n',\n", " '\u00ab':'','\u00bb':'','\u2019':\"'\",'\u201c':'\"','\u201d':'\"'\n", "})\n", "TRANSLIT_TABLE = str.maketrans(TRANSLIT_MAP)\n", "\n", "_WS_RE = re.compile(r\"\\s+\")\n", "_NONALNUM_RE = re.compile(r\"[^a-z0-9\\s]\")\n", "_DOMAIN_RE = re.compile(r\"^(https?://)?(www\\.)?\", re.IGNORECASE)\n", "_TLD_RE = re.compile(r\"\\.(com|in|org|net|co|io|fr)$\", re.IGNORECASE)\n", "SUFFIX_RE = re.compile(r\"\\b(pvt|ltd|limited|private|inc|incorporated|llc|corp|corporation|co|company|enterprises|services|solutions|group|sarl|sas|sasu|eurl|sci|association|fils)\\b\", re.I)\n", "\n", "def clean_and_translit(text):\n", " \"\"\"Fast transliteration: C-table translation + NFKD + unidecode fallback + clean whitespace.\"\"\"\n", " if text is None or (isinstance(text, float) and np.isnan(text)):\n", " return \"\"\n", " s = str(text).translate(TRANSLIT_TABLE)\n", " s = unicodedata.normalize(\"NFKD\", s)\n", " s = \"\".join(c for c in s if not unicodedata.combining(c))\n", " # fallback for any exotic scripts not in base table\n", " if any(ord(c) > 127 for c in s):\n", " s = unidecode(s)\n", " s = s.lower()\n", " s = _DOMAIN_RE.sub(\"\", s)\n", " s = _TLD_RE.sub(\"\", s)\n", " s = _NONALNUM_RE.sub(\" \", s)\n", " return _WS_RE.sub(\" \", s).strip()\n", "\n", "def strip_legal_suffixes(name_clean):\n", " return SUFFIX_RE.sub(\" \", name_clean).strip()\n", "\n", "def compact_alnum(s):\n", " return _NONALNUM_RE.sub(\"\", s).replace(\" \", \"\")\n", "\n", "# Verification against known Indian and French test entities\n", "_tests = [\n", " (\"\u0930\u093f\u092f\u0932 \u0915\u0902\u0938\u0932\u094d\u091f\u0947\u0902\u091f\u094d\u0938 \u092a\u094d\u0930\u093e\u0907\u0935\u0947\u091f \u0932\u093f\u092e\u093f\u091f\u0947\u0921\", \"real\"),\n", " (\"Physical Th\u00e9rapy Care\", \"physical\"),\n", " (\"\u0baa\u0bbe\u0bb2\u0bbe\u0b9c\u0bbf \u0b95\u0bcb\u0bb2\u0bcd\u0b9f\u0bc1 \u0b95\u0bc7\u0bb0\u0bcd\", \"balaji\"),\n", "]\n", "for raw, expect_prefix in _tests:\n", " out = clean_and_translit(raw)\n", " print(f\"{raw!r:45s} -> {out!r}\")\n", " assert out != \"\", f\"Transliteration produced an EMPTY string for: {raw!r}\"\n", " if expect_prefix:\n", " assert out.startswith(expect_prefix), f\"Expected prefix {expect_prefix!r}, got {out!r}\"\n" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "id": "51550e1d", "metadata": {}, "source": [ "## 2. Load Dataset\n", "Locates and loads `source1`, `source2`, `source3`, and `ground_truth`.\n", "In `SAMPLE_MODE = True`, it takes a balanced per-country sample to enable sub-2-minute prototyping.\n" ] }, { "cell_type": "code", "id": "79917e97", "metadata": {}, "source": [ "def find_file(filename):\n", " \"\"\"Finds a dataset file whether in flat structure or train/test subfolders.\"\"\"\n", " candidates = [\n", " DATA_DIR / filename,\n", " DATA_DIR / \"train\" / filename,\n", " DATA_DIR / \"test\" / filename,\n", " ]\n", " for c in candidates:\n", " if c.exists():\n", " return c\n", " raise FileNotFoundError(f\"Could not locate {filename} in {DATA_DIR}\")\n", "\n", "def load_tsv(filename):\n", " path = find_file(filename)\n", " df = pd.read_csv(path, sep=\"\\t\", dtype=str, keep_default_na=False)\n", " print(f\"Loaded {filename}: {len(df):,} rows, cols={list(df.columns)}\")\n", " return df\n", "\n", "train_s1 = load_tsv(\"train_source1.tsv\")\n", "train_s2 = load_tsv(\"train_source2.tsv\")\n", "train_s3 = load_tsv(\"train_source3.tsv\")\n", "train_gt = load_tsv(\"train_ground_truth.tsv\")\n", "\n", "for df in [train_s1, train_s2, train_s3]:\n", " assert df[\"entity_id\"].is_unique, \"Duplicate entity_id found!\"\n", "\n", "if SAMPLE_MODE:\n", " print(f\"\\n[SAMPLE_MODE active] Sampling {SAMPLE_N_PER_COUNTRY:,} entities per country from train_s1...\")\n", " train_s1 = (train_s1.groupby(\"country\", group_keys=False)\n", " .apply(lambda g: g.sample(min(SAMPLE_N_PER_COUNTRY, len(g)), random_state=42))\n", " .reset_index(drop=True))\n", " keep_ids = set(train_s1[\"entity_id\"])\n", " train_gt = train_gt[train_gt[\"source1_entity_id\"].isin(keep_ids)].reset_index(drop=True)\n", " print(f\"train_s1 sampled to: {len(train_s1):,} rows\")\n" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "id": "656c196c", "metadata": {}, "source": [ "## 3. Pre-Normalize Name and Address Fields\n", "Adds `name_clean`, `name_root`, `addr_clean`, `name_compact`, `house_no`, and `postal`.\n", "Only normalizes required tables, preserving memory.\n" ] }, { "cell_type": "code", "id": "69261b0f", "metadata": {}, "source": [ "def add_normalized_columns(df):\n", " \"\"\"Vectorized / batch text normalization for high speed and minimal memory.\"\"\"\n", " df = df.copy()\n", " names = df[\"business_name\"].tolist()\n", " addrs = df[\"business_address\"].tolist()\n", "\n", " clean_names = [clean_and_translit(n) for n in names]\n", " clean_addrs = [clean_and_translit(a) for a in addrs]\n", "\n", " df[\"name_clean\"] = clean_names\n", " df[\"name_root\"] = [strip_legal_suffixes(n) for n in clean_names]\n", " df[\"addr_clean\"] = clean_addrs\n", " df[\"name_compact\"] = [compact_alnum(n) for n in clean_names]\n", "\n", " df[\"house_no\"] = df[\"addr_clean\"].str.extract(r\"(\\d{1,6})\", expand=False).fillna(\"\")\n", " df[\"postal\"] = df[\"addr_clean\"].str.extract(r\"(\\d{5,6})(?!.*\\d{5,6})\", expand=False).fillna(\"\")\n", " return df\n", "\n", "print(\"Normalizing train_s1...\")\n", "train_s1 = add_normalized_columns(train_s1)\n", "\n", "empty_name_frac = (train_s1[\"name_clean\"] == \"\").mean()\n", "assert empty_name_frac < 0.01, \"Error: Transliteration produced excessive empty strings!\"\n" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "id": "8a2f4922", "metadata": {}, "source": [ "## 4. Multi-View DuckDB Blocking (Fixed & Optimized)\n", "Key fixes implemented:\n", "- **Combined Targets**: Both Source 2 and Source 3 targets are registered together into `targets_tmp` (eliminating the bug where Source 2 was overwritten).\n", "- **Index-Once Architecture**: Target keys and frequency caps are generated **once per country**, completely removing the 17x redundant inner loop computation.\n", "- **Top-K Candidate Selection**: Keeps Top-K (35) candidates per S1 entity.\n" ] }, { "cell_type": "code", "id": "e2984698", "metadata": {}, "source": [ "con = duckdb.connect()\n", "con.execute(\"PRAGMA threads=8;\")\n", "con.execute(\"PRAGMA max_memory='12GB';\")\n", "con.execute(f\"PRAGMA temp_directory='{CACHE_DIR}/duckdb_spill';\")\n", "\n", "def generate_keys_sql(alias):\n", " return f\"\"\"\n", " SELECT entity_id, 'p5:' || substr(name_compact, 1, 5) AS key FROM {alias} WHERE length(name_compact) >= 5\n", " UNION ALL\n", " SELECT entity_id, 'nt:' || tok AS key\n", " FROM (SELECT entity_id, unnest(string_split(name_clean, ' ')) AS tok FROM {alias})\n", " WHERE length(tok) >= 3\n", " UNION ALL\n", " SELECT entity_id, 'nb:' || toks[1] || '_' || toks[2] AS key\n", " FROM (SELECT entity_id, string_split(name_clean, ' ') AS toks FROM {alias})\n", " WHERE len(toks) >= 2 AND length(toks[1]) >= 3 AND length(toks[2]) >= 3\n", " UNION ALL\n", " SELECT entity_id, 'np:' || substr(toks[1],1,3) || '_' || substr(toks[2],1,3) AS key\n", " FROM (SELECT entity_id, string_split(name_clean, ' ') AS toks FROM {alias})\n", " WHERE len(toks) >= 2 AND length(toks[1]) >= 3 AND length(toks[2]) >= 3\n", " UNION ALL\n", " SELECT entity_id, 'addr_toks:' || tok AS key\n", " FROM (SELECT entity_id, unnest(string_split(addr_clean, ' ')) AS tok FROM {alias})\n", " WHERE length(tok) >= 4\n", " UNION ALL\n", " SELECT entity_id, 'pn:' || postal || '_' || substr(name_compact,1,3) AS key\n", " FROM {alias} WHERE postal != '' AND length(name_compact) >= 3\n", " UNION ALL\n", " SELECT entity_id, 'hn:' || house_no || '_' || substr(name_compact,1,3) AS key\n", " FROM {alias} WHERE house_no != '' AND length(name_compact) >= 3\n", " UNION ALL\n", " SELECT entity_id, 'ht:' || house_no || '_' || toks[1] AS key\n", " FROM (SELECT entity_id, house_no, string_split(name_clean, ' ') AS toks FROM {alias})\n", " WHERE house_no != '' AND len(toks) >= 1 AND length(toks[1]) >= 3\n", " \"\"\"\n", "\n", "def run_blocking_for_country(s1_country_df, targets_country_df, country, freq_cap, top_k, batch_size=50_000):\n", " \"\"\"Extracts target keys ONCE, then streams S1 batches for candidate joining.\"\"\"\n", " if len(s1_country_df) == 0:\n", " return pd.DataFrame(columns=[\"s1_id\", \"target_id\", \"key_hits\"])\n", "\n", " t_start = time.time()\n", " print(f\"\\n--- [Blocking {country}] S1: {len(s1_country_df):,}, Combined Targets: {len(targets_country_df):,} ---\")\n", " \n", " con.register(\"targets_country\", targets_country_df)\n", " tgt_sql = generate_keys_sql(\"targets_country\")\n", " con.execute(f\"CREATE OR REPLACE TEMP TABLE target_keys AS {tgt_sql}\")\n", " \n", " # Calculate key frequencies once\n", " con.execute(f\"\"\"\n", " CREATE OR REPLACE TEMP TABLE key_freq AS\n", " SELECT key, count(*) AS n FROM target_keys GROUP BY key\n", " \"\"\")\n", " con.execute(f\"\"\"\n", " CREATE OR REPLACE TEMP TABLE filtered_target_keys AS\n", " SELECT t.entity_id, t.key\n", " FROM target_keys t\n", " JOIN key_freq f ON t.key = f.key AND f.n <= {freq_cap}\n", " \"\"\")\n", " con.execute(\"CREATE INDEX IF NOT EXISTS ftk_idx ON filtered_target_keys(key);\")\n", "\n", " n_batches = (len(s1_country_df) + batch_size - 1) // batch_size\n", " candidates_list = []\n", "\n", " for b in range(n_batches):\n", " s1_batch = s1_country_df.iloc[b * batch_size : (b + 1) * batch_size]\n", " con.register(\"s1_batch\", s1_batch)\n", " s1_sql = generate_keys_sql(\"s1_batch\")\n", " con.execute(f\"CREATE OR REPLACE TEMP TABLE s1_batch_keys AS {s1_sql}\")\n", "\n", " joined = con.execute(f\"\"\"\n", " WITH pair_hits AS (\n", " SELECT s.entity_id AS s1_id, t.entity_id AS target_id, count(*) AS key_hits\n", " FROM s1_batch_keys s\n", " JOIN filtered_target_keys t ON s.key = t.key\n", " GROUP BY 1, 2\n", " ),\n", " ranked AS (\n", " SELECT s1_id, target_id, key_hits,\n", " row_number() OVER(PARTITION BY s1_id ORDER BY key_hits DESC, target_id) AS rank\n", " FROM pair_hits\n", " )\n", " SELECT s1_id, target_id, key_hits\n", " FROM ranked\n", " WHERE rank <= {top_k};\n", " \"\"\").fetchdf()\n", "\n", " con.unregister(\"s1_batch\")\n", " candidates_list.append(joined)\n", " print(f\" Batch {b+1}/{n_batches}: {len(s1_batch):,} S1 -> {len(joined):,} candidate pairs\")\n", "\n", " con.unregister(\"targets_country\")\n", " result = pd.concat(candidates_list, ignore_index=True) if candidates_list else pd.DataFrame(columns=[\"s1_id\", \"target_id\", \"key_hits\"])\n", " print(f\"Country {country} finished in {time.time()-t_start:.1f}s. Total candidate pairs: {len(result):,}\")\n", " return result\n" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "id": "c4385c2d", "metadata": {}, "source": [ "## 4.1 Run Blocking on Training Set\n", "We normalize S2 and S3 per-country on demand to ensure lean memory usage.\n" ] }, { "cell_type": "code", "id": "ec3db4dc", "metadata": {}, "source": [ "countries = train_s1[\"country\"].unique().tolist()\n", "candidate_frames = []\n", "\n", "for ctry in countries:\n", " cap = FREQ_CAP_FRANCE if ctry == \"France\" else FREQ_CAP_DEFAULT\n", " \n", " # Filter S2 and S3 for this country and normalize on demand\n", " s1_c = train_s1[train_s1[\"country\"] == ctry]\n", " s2_c = add_normalized_columns(train_s2[train_s2[\"country\"] == ctry])\n", " s3_c = add_normalized_columns(train_s3[train_s3[\"country\"] == ctry])\n", " targets_c = pd.concat([s2_c, s3_c], ignore_index=True)\n", "\n", " cands = run_blocking_for_country(s1_c, targets_c, ctry, cap, TOP_K_CANDIDATES)\n", " candidate_frames.append(cands)\n", " del s2_c, s3_c, targets_c\n", " gc.collect()\n", "\n", "candidate_pairs = pd.concat(candidate_frames, ignore_index=True)\n", "print(f\"\\n>>> Total Training Candidate Pairs Generated: {len(candidate_pairs):,} <<<\")\n" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "id": "56b67ec4", "metadata": {}, "source": [ "## 5. Mandatory Blocking Recall Ceiling Diagnostic\n", "Verifies the true candidate recall ceiling before model training.\n", "Any silent failure in blocking will be detected here immediately.\n" ] }, { "cell_type": "code", "id": "84946a80", "metadata": {}, "source": [ "def measure_blocking_recall(candidate_df, gt_df):\n", " gt_pairs = set()\n", " for _, row in gt_df.iterrows():\n", " s1 = row[\"source1_entity_id\"]\n", " matched = str(row.get(\"matched_entity_ids\", \"\") or \"\").strip()\n", " if matched and matched != \"nan\":\n", " for tid in matched.split(\",\"):\n", " tid_clean = tid.strip()\n", " if tid_clean:\n", " gt_pairs.add((s1, tid_clean))\n", "\n", " cand_set = set(zip(candidate_df[\"s1_id\"], candidate_df[\"target_id\"]))\n", " recalled = gt_pairs & cand_set\n", "\n", " recall = len(recalled) / len(gt_pairs) if gt_pairs else 0.0\n", " print(f\"True entity pairs in Ground Truth : {len(gt_pairs):,}\")\n", " print(f\"True entity pairs captured by blocking : {len(recalled):,}\")\n", " print(f\"BLOCKING RECALL CEILING: {recall:.4%}\")\n", " return recall, gt_pairs\n", "\n", "recall_ceiling, gt_pairs_all = measure_blocking_recall(candidate_pairs, train_gt)\n", "assert recall_ceiling >= 0.85, f\"Recall ceiling {recall_ceiling:.2%} is below 85%! Check blocking rules.\"\n", "print(\"SUCCESS: Blocking recall check passed!\")\n" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "id": "d66d92ba", "metadata": {}, "source": [ "## 6. Bi-Encoder Neural Embeddings (`Qwen3-Embedding-0.6B`)\n", "To fit within Kaggle execution limits:\n", "1. **Deduplication to Unique Entities**: Each entity is embedded only **once** and cached by ID.\n", "2. **Matryoshka Truncation**: Truncated to 256 dimensions for fast dot-product cosine similarity.\n", "3. **Multi-GPU Parallelism**: Automatically leverages `torch.nn.DataParallel` when 2x GPUs are available.\n" ] }, { "cell_type": "code", "id": "a88c001c", "metadata": {}, "source": [ "from transformers import AutoTokenizer, AutoModel\n", "\n", "class EntityEmbedder:\n", " def __init__(self, model_name=\"Qwen/Qwen3-Embedding-0.6B\", embed_dim=256, max_len=64):\n", " self.model_name = model_name\n", " self.embed_dim = embed_dim\n", " self.max_len = max_len\n", " self.embed_cache = {}\n", " \n", " print(f\"Loading tokenizer and model: {model_name}...\")\n", " self.tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)\n", " self.model = AutoModel.from_pretrained(model_name, torch_dtype=torch.float16, trust_remote_code=True)\n", " \n", " if torch.cuda.device_count() > 1:\n", " print(f\"Using DataParallel across {torch.cuda.device_count()} GPUs\")\n", " self.model = nn.DataParallel(self.model)\n", " \n", " self.model = self.model.to(device).eval()\n", " print(\"Embedder ready.\")\n", "\n", " def format_record(self, name, addr, country):\n", " return f\"Business name: {name} | Address: {addr} | Country: {country}\"\n", "\n", " @torch.inference_mode()\n", " def embed_entities(self, df_entities, batch_size=256 if torch.cuda.device_count() >= 2 else 64):\n", " \"\"\"Embeds unique entities that are not already present in the cache.\"\"\"\n", " new_entities = df_entities[~df_entities[\"entity_id\"].isin(self.embed_cache)].drop_duplicates(subset=\"entity_id\")\n", " if len(new_entities) == 0:\n", " return\n", "\n", " texts = [self.format_record(r.name_clean, r.addr_clean, r.country) for r in new_entities.itertuples()]\n", " e_ids = new_entities[\"entity_id\"].tolist()\n", " \n", " print(f\"Embedding {len(texts):,} new unique entities...\")\n", " t0 = time.time()\n", " \n", " for i in range(0, len(texts), batch_size):\n", " b_texts = texts[i : i + batch_size]\n", " b_ids = e_ids[i : i + batch_size]\n", " \n", " enc = self.tokenizer(b_texts, padding=True, truncation=True, max_length=self.max_len, return_tensors=\"pt\").to(device)\n", " out = self.model(**enc)\n", " last_hidden = out.last_hidden_state if hasattr(out, \"last_hidden_state\") else out[0]\n", " mask = enc[\"attention_mask\"].unsqueeze(-1).float()\n", " pooled = (last_hidden * mask).sum(1) / mask.sum(1).clamp(min=1e-9)\n", " pooled = pooled[:, :self.embed_dim]\n", " pooled = F.normalize(pooled, p=2, dim=1)\n", " \n", " vecs = pooled.float().cpu().numpy()\n", " for eid, v in zip(b_ids, vecs):\n", " self.embed_cache[eid] = v\n", "\n", " if (i // batch_size) % 100 == 0 and i > 0:\n", " rate = i / max(time.time() - t0, 1e-9)\n", " print(f\" Progress: {i:,}/{len(texts):,} ({rate:.0f} entities/s)\")\n", " \n", " print(f\"Completed in {time.time()-t0:.1f}s. Total cached entities: {len(self.embed_cache):,}\")\n", "\n", " def compute_cosine_similarity(self, s1_ids, target_ids):\n", " \"\"\"Fast dot product lookup between cached L2-normalized embeddings.\"\"\"\n", " s1_vecs = np.stack([self.embed_cache[i] for i in s1_ids])\n", " tgt_vecs = np.stack([self.embed_cache[i] for i in target_ids])\n", " return (s1_vecs * tgt_vecs).sum(axis=1)\n", "\n", "embedder = EntityEmbedder()\n" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "id": "8e504ee9", "metadata": {}, "source": [ "## 7. Fast Feature Extraction\n", "Combines lexical similarity signals (RapidFuzz), structural signals (House/Postal match),\n", "blocking frequency (`key_hits`), and the neural semantic similarity (`qwen_cosine_sim`).\n", "Processed in batches to guarantee zero OOM risk.\n" ] }, { "cell_type": "code", "id": "da83c2f1", "metadata": {}, "source": [ "from rapidfuzz import fuzz\n", "\n", "def extract_features_batched(candidate_df, s1_lookup_df, target_lookup_df, embedder, batch_size=100_000):\n", " \"\"\"Extracts features in chunks to prevent large intermediate DataFrame allocations.\"\"\"\n", " s1_dict = s1_lookup_df.set_index(\"entity_id\")[[\"name_clean\", \"name_root\", \"addr_clean\", \"house_no\", \"postal\", \"country\"]].to_dict(\"index\")\n", " tgt_dict = target_lookup_df.set_index(\"entity_id\")[[\"name_clean\", \"name_root\", \"addr_clean\", \"house_no\", \"postal\", \"country\"]].to_dict(\"index\")\n", "\n", " # Prepare embedding universe\n", " needed_eids = list(set(candidate_df[\"s1_id\"]) | set(candidate_df[\"target_id\"]))\n", " missing_eids = [eid for eid in needed_eids if eid not in embedder.embed_cache]\n", " if missing_eids:\n", " records = []\n", " for eid in missing_eids:\n", " row = s1_dict.get(eid) or tgt_dict.get(eid)\n", " if row:\n", " records.append({\"entity_id\": eid, \"name_clean\": row[\"name_clean\"], \"addr_clean\": row[\"addr_clean\"], \"country\": row[\"country\"]})\n", " if records:\n", " embedder.embed_entities(pd.DataFrame(records))\n", "\n", " feature_chunks = []\n", " n_chunks = (len(candidate_df) + batch_size - 1) // batch_size\n", " print(f\"Extracting features for {len(candidate_df):,} pairs in {n_chunks} chunks...\")\n", "\n", " for i in range(n_chunks):\n", " chunk = candidate_df.iloc[i * batch_size : (i + 1) * batch_size]\n", " s1_ids = chunk[\"s1_id\"].tolist()\n", " tgt_ids = chunk[\"target_id\"].tolist()\n", "\n", " qwen_sim = embedder.compute_cosine_similarity(s1_ids, tgt_ids)\n", "\n", " name_ratios, name_tsorts, name_tsets, name_partials, root_ratios = [], [], [], [], []\n", " addr_ratios, addr_tsorts, addr_tsets, addr_partials = [], [], [], []\n", " house_matches, postal_matches, postal_conflicts = [], [], []\n", "\n", " for s, t in zip(s1_ids, tgt_ids):\n", " s_row = s1_dict[s]\n", " t_row = tgt_dict[t]\n", "\n", " sn, tn = s_row[\"name_clean\"], t_row[\"name_clean\"]\n", " sa, ta = s_row[\"addr_clean\"], t_row[\"addr_clean\"]\n", " sr, tr = s_row[\"name_root\"], t_row[\"name_root\"]\n", "\n", " name_ratios.append(fuzz.ratio(sn, tn))\n", " name_tsorts.append(fuzz.token_sort_ratio(sn, tn))\n", " name_tsets.append(fuzz.token_set_ratio(sn, tn))\n", " name_partials.append(fuzz.partial_ratio(sn, tn))\n", " root_ratios.append(fuzz.ratio(sr, tr))\n", "\n", " addr_ratios.append(fuzz.ratio(sa, ta))\n", " addr_tsorts.append(fuzz.token_sort_ratio(sa, ta))\n", " addr_tsets.append(fuzz.token_set_ratio(sa, ta))\n", " addr_partials.append(fuzz.partial_ratio(sa, ta))\n", "\n", " sh, th = s_row[\"house_no\"], t_row[\"house_no\"]\n", " sp, tp = s_row[\"postal\"], t_row[\"postal\"]\n", "\n", " house_matches.append(1 if (sh and th and sh == th) else 0)\n", " postal_matches.append(1 if (sp and tp and sp == tp) else 0)\n", " postal_conflicts.append(1 if (sp and tp and sp != tp) else 0)\n", "\n", " df_chunk = pd.DataFrame({\n", " \"s1_id\": s1_ids,\n", " \"target_id\": tgt_ids,\n", " \"key_hits\": chunk[\"key_hits\"].values,\n", " \"qwen_cosine_sim\": qwen_sim,\n", " \"name_ratio\": name_ratios,\n", " \"name_tsort\": name_tsorts,\n", " \"name_tset\": name_tsets,\n", " \"name_partial\": name_partials,\n", " \"root_ratio\": root_ratios,\n", " \"addr_ratio\": addr_ratios,\n", " \"addr_tsort\": addr_tsorts,\n", " \"addr_tset\": addr_tsets,\n", " \"addr_partial\": addr_partials,\n", " \"house_match\": house_matches,\n", " \"postal_match\": postal_matches,\n", " \"postal_conflict\": postal_conflicts,\n", " })\n", " feature_chunks.append(df_chunk)\n", "\n", " return pd.concat(feature_chunks, ignore_index=True)\n", "\n", "# Prepare combined target reference\n", "targets_all_norm = pd.concat([\n", " add_normalized_columns(train_s2[train_s2[\"entity_id\"].isin(set(candidate_pairs[\"target_id\"]))]),\n", " add_normalized_columns(train_s3[train_s3[\"entity_id\"].isin(set(candidate_pairs[\"target_id\"]))]),\n", "], ignore_index=True).drop_duplicates(subset=\"entity_id\")\n", "\n", "training_features = extract_features_batched(candidate_pairs, train_s1, targets_all_norm, embedder)\n", "print(\"Extracted training features shape:\", training_features.shape)\n" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "id": "52e4f748", "metadata": {}, "source": [ "## 8. LightGBM Classifier & Corrected Cross-Validation\n", "Key Fix: `macro_f05` evaluates **only** the entities present in the current validation fold / country,\n", "eliminating the 5x score suppression bug.\n" ] }, { "cell_type": "code", "id": "a4b8b502", "metadata": {}, "source": [ "import lightgbm as lgb\n", "from sklearn.model_selection import GroupKFold\n", "\n", "# Build labels\n", "gt_lookup = defaultdict(set)\n", "for _, row in train_gt.iterrows():\n", " s1 = row[\"source1_entity_id\"]\n", " m = str(row.get(\"matched_entity_ids\", \"\") or \"\").strip()\n", " if m and m != \"nan\":\n", " for t in m.split(\",\"):\n", " gt_lookup[s1].add(t.strip())\n", "\n", "training_features[\"label\"] = [\n", " 1 if t in gt_lookup[s] else 0\n", " for s, t in zip(training_features[\"s1_id\"], training_features[\"target_id\"])\n", "]\n", "\n", "FEATURE_COLS = [c for c in training_features.columns if c not in (\"s1_id\", \"target_id\", \"label\")]\n", "X = training_features[FEATURE_COLS].astype(float)\n", "y = training_features[\"label\"].values\n", "groups = training_features[\"s1_id\"].values\n", "\n", "n_pos, n_neg = (y == 1).sum(), (y == 0).sum()\n", "scale_pos_weight = n_neg / max(n_pos, 1)\n", "print(f\"Positives: {n_pos:,} | Negatives: {n_neg:,} | scale_pos_weight: {scale_pos_weight:.2f}\")\n", "\n", "def evaluate_macro_f05(eval_s1_ids, cand_s1_ids, cand_target_ids, probs, threshold, gt_map):\n", " \"\"\"Accurately computes Macro F0.5 strictly for the specified eval_s1_ids.\"\"\"\n", " preds = defaultdict(set)\n", " for s, t, p in zip(cand_s1_ids, cand_target_ids, probs):\n", " if p >= threshold:\n", " preds[s].add(t)\n", "\n", " scores = []\n", " for s in eval_s1_ids:\n", " pred_set = preds.get(s, set())\n", " true_set = gt_map.get(s, set())\n", " if not pred_set and not true_set:\n", " scores.append(1.0)\n", " continue\n", " tp = len(pred_set & true_set)\n", " prec = tp / len(pred_set) if pred_set else 0.0\n", " rec = tp / len(true_set) if true_set else 0.0\n", " if (0.25 * prec + rec) > 0:\n", " f05 = (1.25 * prec * rec) / (0.25 * prec + rec)\n", " else:\n", " f05 = 0.0\n", " scores.append(f05)\n", " return float(np.mean(scores))\n", "\n", "gkf = GroupKFold(n_splits=5)\n", "oof_pred = np.zeros(len(X))\n", "fold_scores = []\n", "\n", "for fold, (train_idx, val_idx) in enumerate(gkf.split(X, y, groups)):\n", " train_set = lgb.Dataset(X.iloc[train_idx], label=y[train_idx])\n", " val_set = lgb.Dataset(X.iloc[val_idx], label=y[val_idx])\n", " \n", " params = dict(\n", " objective=\"binary\",\n", " metric=\"binary_logloss\",\n", " scale_pos_weight=scale_pos_weight,\n", " num_leaves=63,\n", " learning_rate=0.05,\n", " n_estimators=500,\n", " n_jobs=-1,\n", " verbosity=-1\n", " )\n", " booster = lgb.train(\n", " params, train_set, valid_sets=[val_set],\n", " callbacks=[lgb.early_stopping(30, verbose=False)]\n", " )\n", " val_probs = booster.predict(X.iloc[val_idx])\n", " oof_pred[val_idx] = val_probs\n", "\n", " val_s1 = set(groups[val_idx])\n", " f05 = evaluate_macro_f05(val_s1, groups[val_idx], training_features[\"target_id\"].values[val_idx], val_probs, 0.50, gt_lookup)\n", " fold_scores.append(f05)\n", " print(f\"Fold {fold+1}: Macro F0.5 @ 0.50 = {f05:.4f}\")\n", "\n", "print(f\"\\n>>> 5-Fold Mean Macro F0.5: {np.mean(fold_scores):.4f} +/- {np.std(fold_scores):.4f} <<<\")\n" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "id": "3e8638ca", "metadata": {}, "source": [ "## 9. Per-Country Threshold Sweeping\n", "Optimizes decision thresholds individually for France, US, and India on the OOF predictions.\n" ] }, { "cell_type": "code", "id": "455def1b", "metadata": {}, "source": [ "s1_country_map = train_s1.set_index(\"entity_id\")[\"country\"].to_dict()\n", "candidate_countries = np.array([s1_country_map[s] for s in groups])\n", "\n", "per_country_threshold = {}\n", "for ctry in np.unique(candidate_countries):\n", " mask = candidate_countries == ctry\n", " ctry_s1 = set(groups[mask])\n", " \n", " best_t, best_score = 0.50, -1.0\n", " for t in np.arange(0.30, 0.951, 0.02):\n", " score = evaluate_macro_f05(\n", " ctry_s1,\n", " groups[mask],\n", " training_features[\"target_id\"].values[mask],\n", " oof_pred[mask],\n", " t,\n", " gt_lookup\n", " )\n", " if score > best_score:\n", " best_score, best_t = score, t\n", " \n", " per_country_threshold[ctry] = round(float(best_t), 2)\n", " print(f\"Country [{ctry}]: Optimal Threshold = {best_t:.2f} (Macro F0.5 = {best_score:.4f})\")\n", "\n", "per_country_threshold[\"__default__\"] = 0.50\n", "print(\"\\nConfigured Decision Thresholds:\", per_country_threshold)\n" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "id": "806a7056", "metadata": {}, "source": [ "## 10. Final Model Training & Artifact Persistence\n" ] }, { "cell_type": "code", "id": "f3f967cb", "metadata": {}, "source": [ "full_train_set = lgb.Dataset(X, label=y)\n", "final_params = dict(\n", " objective=\"binary\",\n", " metric=\"binary_logloss\",\n", " scale_pos_weight=scale_pos_weight,\n", " num_leaves=63,\n", " learning_rate=0.05,\n", " n_estimators=500,\n", " n_jobs=-1,\n", " verbosity=-1\n", ")\n", "final_model = lgb.train(final_params, full_train_set)\n", "final_model.save_model(str(CACHE_DIR / \"model.lgb\"))\n", "\n", "with open(CACHE_DIR / \"model_meta.json\", \"w\") as f:\n", " json.dump({\n", " \"feature_cols\": FEATURE_COLS,\n", " \"per_country_threshold\": per_country_threshold,\n", " \"scale_pos_weight\": float(scale_pos_weight)\n", " }, f, indent=2)\n", "\n", "print(\"Saved final model and metadata to\", CACHE_DIR)\n" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "id": "6d3e876d", "metadata": {}, "source": [ "## 11. Full End-to-End Test Set Inference\n", "Executes multi-view blocking, Qwen bi-encoder neural embeddings, and feature extraction\n", "across `test_source1`, `test_source2`, and `test_source3`.\n" ] }, { "cell_type": "code", "id": "a35939da", "metadata": {}, "source": [ "print(\"Loading test tables...\")\n", "test_s1 = load_tsv(\"test_source1.tsv\")\n", "test_s2 = load_tsv(\"test_source2.tsv\")\n", "test_s3 = load_tsv(\"test_source3.tsv\")\n", "\n", "print(\"Normalizing test_s1...\")\n", "test_s1 = add_normalized_columns(test_s1)\n", "\n", "test_candidate_frames = []\n", "for ctry in test_s1[\"country\"].unique():\n", " cap = FREQ_CAP_FRANCE if ctry == \"France\" else FREQ_CAP_DEFAULT\n", " s1_c = test_s1[test_s1[\"country\"] == ctry]\n", " s2_c = add_normalized_columns(test_s2[test_s2[\"country\"] == ctry])\n", " s3_c = add_normalized_columns(test_s3[test_s3[\"country\"] == ctry])\n", " targets_c = pd.concat([s2_c, s3_c], ignore_index=True)\n", "\n", " cands = run_blocking_for_country(s1_c, targets_c, ctry, cap, TOP_K_CANDIDATES)\n", " test_candidate_frames.append(cands)\n", " del s2_c, s3_c, targets_c\n", " gc.collect()\n", "\n", "test_candidate_pairs = pd.concat(test_candidate_frames, ignore_index=True)\n", "print(f\"\\nTotal Test Candidate Pairs: {len(test_candidate_pairs):,}\")\n", "\n", "# Prepare test targets reference\n", "test_needed_targets = set(test_candidate_pairs[\"target_id\"])\n", "test_targets_all = pd.concat([\n", " add_normalized_columns(test_s2[test_s2[\"entity_id\"].isin(test_needed_targets)]),\n", " add_normalized_columns(test_s3[test_s3[\"entity_id\"].isin(test_needed_targets)])\n", "], ignore_index=True).drop_duplicates(subset=\"entity_id\")\n", "\n", "test_features = extract_features_batched(test_candidate_pairs, test_s1, test_targets_all, embedder)\n", "\n", "# Predict probabilities\n", "test_probs = final_model.predict(test_features[FEATURE_COLS].astype(float))\n", "test_features[\"prob\"] = test_probs\n", "\n", "# Apply per-country threshold\n", "test_s1_ctry = test_s1.set_index(\"entity_id\")[\"country\"].to_dict()\n", "test_features[\"country\"] = test_features[\"s1_id\"].map(test_s1_ctry)\n", "test_features[\"threshold\"] = test_features[\"country\"].map(lambda c: per_country_threshold.get(c, per_country_threshold[\"__default__\"]))\n", "test_features[\"above_threshold\"] = test_features[\"prob\"] >= test_features[\"threshold\"]\n", "\n", "n_above = test_features[\"above_threshold\"].sum()\n", "print(f\"Candidate pairs above confidence threshold: {n_above:,} ({n_above/len(test_features):.1%})\")\n" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "id": "909a5f79", "metadata": {}, "source": [ "## 12. 1:Many Target-Exclusivity Resolution (Fixed)\n", "**Ground Truth Rule**:\n", "- Each Target entity belongs to **at most one** S1 entity.\n", "- Each S1 entity can belong to **multiple** Target entities (across S2 and S3).\n", "\n", "We sort candidate pairs above threshold by confidence probability descending, and retain the\n", "single highest-confidence S1 entity for every matched target.\n" ] }, { "cell_type": "code", "id": "42a24012", "metadata": {}, "source": [ "candidates_above = test_features[test_features[\"above_threshold\"]][[\"s1_id\", \"target_id\", \"prob\"]]\n", "\n", "# Resolve 1:Many target exclusivity\n", "resolved = (candidates_above\n", " .sort_values(\"prob\", ascending=False)\n", " .drop_duplicates(subset=[\"target_id\"]))\n", "\n", "print(f\"Resolved {len(resolved):,} matching links across {resolved['s1_id'].nunique():,} unique S1 entities.\")\n" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "id": "3cde9178", "metadata": {}, "source": [ "## 13. Build Official Submission Files\n", "Generates `matching_results.tsv` and `candidate_pairs.tsv` following the exact challenge schema:\n", "- Every `test_source1` entity is represented exactly once in `matching_results.tsv`.\n", "- Candidate pairs are exported in `candidate_pairs.tsv`.\n" ] }, { "cell_type": "code", "id": "38fcf2db", "metadata": {}, "source": [ "# Build matches map: S1 -> comma-separated sorted targets\n", "matches_by_s1 = resolved.groupby(\"s1_id\")[\"target_id\"].apply(lambda ids: \",\".join(sorted(ids))).to_dict()\n", "\n", "all_test_s1_ids = test_s1[\"entity_id\"].tolist()\n", "matching_results = pd.DataFrame({\n", " \"source1_entity_id\": all_test_s1_ids,\n", " \"matched_entity_ids\": [matches_by_s1.get(sid, \"\") for sid in all_test_s1_ids]\n", "})\n", "\n", "n_singletons = (matching_results[\"matched_entity_ids\"] == \"\").sum()\n", "n_matched = len(matching_results) - n_singletons\n", "print(f\"Test Entities: {len(matching_results):,}\")\n", "print(f\" Singletons (no match): {n_singletons:,} ({n_singletons/len(matching_results):.1%})\")\n", "print(f\" Matched Entities : {n_matched:,} ({n_matched/len(matching_results):.1%})\")\n", "\n", "assert len(matching_results) == len(all_test_s1_ids), \"Row count mismatch!\"\n", "assert matching_results[\"source1_entity_id\"].is_unique, \"Duplicate S1 rows detected!\"\n", "\n", "# Export matching results\n", "out_match_path = OUTPUT_DIR / \"matching_results.tsv\"\n", "matching_results.to_csv(out_match_path, sep=\"\\t\", index=False)\n", "print(\"Wrote matching_results.tsv to:\", out_match_path)\n", "\n", "# Export candidate pairs\n", "out_cand_path = OUTPUT_DIR / \"candidate_pairs.tsv\"\n", "cand_export = test_candidate_pairs[[\"s1_id\", \"target_id\"]].rename(\n", " columns={\"s1_id\": \"source1_entity_id\", \"target_id\": \"matched_entity_id\"}\n", ")\n", "cand_export.to_csv(out_cand_path, sep=\"\\t\", index=False)\n", "print(\"Wrote candidate_pairs.tsv to:\", out_cand_path)\n" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "id": "52a6c3fc", "metadata": {}, "source": [ "## 14. Run Submission Validator\n", "Executes `utils/validate_submission.py` to confirm total compliance with the competition specification.\n" ] }, { "cell_type": "code", "id": "bc2c9715", "metadata": {}, "source": [ "import subprocess\n", "\n", "validator_script = None\n", "for cand in [Path(\"utils/validate_submission.py\"), DATA_DIR / \"utils\" / \"validate_submission.py\"]:\n", " if cand.exists():\n", " validator_script = cand\n", " break\n", "\n", "if validator_script:\n", " test_dir_arg = DATA_DIR if (DATA_DIR / \"test_source1.tsv\").exists() else (DATA_DIR / \"test\")\n", " print(f\"Running official validator using {validator_script}...\")\n", " res = subprocess.run([\n", " sys.executable, str(validator_script),\n", " \"--output_dir\", str(OUTPUT_DIR),\n", " \"--test_dir\", str(test_dir_arg)\n", " ], capture_output=True, text=True)\n", " print(res.stdout)\n", " if res.stderr:\n", " print(res.stderr)\n", " if res.returncode == 0:\n", " print(\"\\n>>> SUBMISSION FULLY VALIDATED AND READY TO SUBMIT! <<<\")\n", " else:\n", " print(\"Validator returned non-zero code. Inspect messages above.\")\n", "else:\n", " print(\"validate_submission.py not found in local path \u2014 verify output files manually.\")\n" ], "execution_count": null, "outputs": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.10" } }, "nbformat": 4, "nbformat_minor": 5 }