Upload notebooks/phase4_executed.ipynb with huggingface_hub

notebooks/phase4_executed.ipynb

@@ -0,0 +1,1157 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Phase 4 — Variant Assignment / Variant Modeling\n",
+    "## Compliments Reference DB Pipeline\n",
+    "\n",
+    "**Version:** 1.0.0\n",
+    "**Date:** 2026-07-30\n",
+    "\n",
+    "**Input:** Phase 3 outputs + Phase 2 output\n",
+    "\n",
+    "**This phase does:**\n",
+    "1. Transforms Phase 3 product groups into structured variant models\n",
+    "2. Assigns deterministic variant_ids to products within each group\n",
+    "3. Creates variant tables and mappings\n",
+    "4. Validates variant consistency rules\n",
+    "\n",
+    "**CRITICAL:** This phase does NOT use nutrition data.\n",
+    "**CRITICAL:** This phase does NOT use any external taxonomy.\n",
+    "**CRITICAL:** This phase is 100% deterministic (no LLM)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2026-07-30T17:47:27.262371Z",
+     "iopub.status.busy": "2026-07-30T17:47:27.261832Z",
+     "iopub.status.idle": "2026-07-30T17:47:28.134559Z",
+     "shell.execute_reply": "2026-07-30T17:47:28.132783Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Python: 3.13.7 (main, Mar  3 2026, 12:19:54) [GCC 15.2.0]\n",
+      "pandas: 3.0.5\n",
+      "numpy: 2.5.1\n",
+      "Timestamp: 2026-07-30T17:47:28.129557+00:00\n"
+     ]
+    }
+   ],
+   "source": [
+    "import json, hashlib, uuid, sys\n",
+    "from datetime import datetime, timezone\n",
+    "from pathlib import Path\n",
+    "\n",
+    "import pandas as pd\n",
+    "import numpy as np\n",
+    "\n",
+    "print(f\"Python: {sys.version}\")\n",
+    "print(f\"pandas: {pd.__version__}\")\n",
+    "print(f\"numpy: {np.__version__}\")\n",
+    "print(f\"Timestamp: {datetime.now(timezone.utc).isoformat()}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 1. Variant Definition\n",
+    "\n",
+    "A **variant** is a purchasable representation of the same reference product\n",
+    "that differs in attributes such as:\n",
+    "\n",
+    "- **Size**: 500 g, 1 kg, 2 kg\n",
+    "- **Package quantity**: 1 × 500 g, 2 × 500 g\n",
+    "- **Pack configuration**: single, multipack, case\n",
+    "\n",
+    "Variants within the same group share the same identity attributes:\n",
+    "- is_organic, is_gluten_free, is_naturally_simple, etc.\n",
+    "- product_line\n",
+    "- flavour\n",
+    "- formulation\n",
+    "- fat_level\n",
+    "\n",
+    "Variants differ only in size/package attributes."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 2. Load Phase 3 Outputs"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2026-07-30T17:47:28.285611Z",
+     "iopub.status.busy": "2026-07-30T17:47:28.284392Z",
+     "iopub.status.idle": "2026-07-30T17:47:28.445823Z",
+     "shell.execute_reply": "2026-07-30T17:47:28.441829Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Phase 3 mapping: 4440 rows, 14 columns\n",
+      "Columns: ['upc', 'external_id', 'group_id', 'group_name', 'core_title', 'original_title', 'brand', 'size', 'identity_hash', 'product_domain', 'taxonomy', 'product_line', 'source', 'source_url']\n"
+     ]
+    },
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "<style scoped>\n",
+       "    .dataframe tbody tr th:only-of-type {\n",
+       "        vertical-align: middle;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe tbody tr th {\n",
+       "        vertical-align: top;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe thead th {\n",
+       "        text-align: right;\n",
+       "    }\n",
+       "</style>\n",
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: right;\">\n",
+       "      <th></th>\n",
+       "      <th>upc</th>\n",
+       "      <th>external_id</th>\n",
+       "      <th>group_id</th>\n",
+       "      <th>group_name</th>\n",
+       "      <th>core_title</th>\n",
+       "      <th>original_title</th>\n",
+       "      <th>brand</th>\n",
+       "      <th>size</th>\n",
+       "      <th>identity_hash</th>\n",
+       "      <th>product_domain</th>\n",
+       "      <th>taxonomy</th>\n",
+       "      <th>product_line</th>\n",
+       "      <th>source</th>\n",
+       "      <th>source_url</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <th>0</th>\n",
+       "      <td>5.574252e+10</td>\n",
+       "      <td>297608EA</td>\n",
+       "      <td>27b951a3-bef1-5ed4-80b7-7ef5cdca910a</td>\n",
+       "      <td>Cold And Sinus Plus</td>\n",
+       "      <td>Cold And Sinus Plus</td>\n",
+       "      <td>Compliments Cold And Sinus Plus 20 Count</td>\n",
+       "      <td>Compliments</td>\n",
+       "      <td>20 per pack</td>\n",
+       "      <td>0|0|0|0|0|0|0|0|0|fat_none|regular|Core|lime|</td>\n",
+       "      <td>unknown</td>\n",
+       "      <td>GENERAL_GROCERY</td>\n",
+       "      <td>Core</td>\n",
+       "      <td>voila</td>\n",
+       "      <td>https://voila.ca/products/compliments-cold-and...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>1</th>\n",
+       "      <td>5.574234e+10</td>\n",
+       "      <td>464324EA</td>\n",
+       "      <td>e608ba58-8f93-59f1-9446-edb91aef79bd</td>\n",
+       "      <td>Twister T2 Light Bulbs 23W Soft White</td>\n",
+       "      <td>Twister T2 Light Bulbs 23W Soft White</td>\n",
+       "      <td>Compliments Twister T2 Light Bulbs 23W Soft Wh...</td>\n",
+       "      <td>Compliments</td>\n",
+       "      <td>2 per pack</td>\n",
+       "      <td>0|0|0|0|0|0|0|0|0|fat_none|reduced_fat|Core|lime|</td>\n",
+       "      <td>unknown</td>\n",
+       "      <td>GENERAL_GROCERY</td>\n",
+       "      <td>Core</td>\n",
+       "      <td>voila</td>\n",
+       "      <td>https://voila.ca/products/compliments-twister-...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>2</th>\n",
+       "      <td>9.900026e+11</td>\n",
+       "      <td>144288CS</td>\n",
+       "      <td>336c79d8-796d-5682-8195-793883dd97ef</td>\n",
+       "      <td>Zero Calorie Soft Drink</td>\n",
+       "      <td>Zero Calorie Soft Drink</td>\n",
+       "      <td>Compliments Zero Calorie Soft Drink 12 x 355ml</td>\n",
+       "      <td>Compliments</td>\n",
+       "      <td>12 x 29.583ml</td>\n",
+       "      <td>0|0|0|0|0|0|0|0|0|fat_none|regular|Core|lime|</td>\n",
+       "      <td>food</td>\n",
+       "      <td>BEVERAGES</td>\n",
+       "      <td>Core</td>\n",
+       "      <td>voila</td>\n",
+       "      <td>https://voila.ca/products/compliments-zero-cal...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>3</th>\n",
+       "      <td>5.574257e+10</td>\n",
+       "      <td>312204EA</td>\n",
+       "      <td>fd1acb22-4231-5b7b-bde4-8709b31cf04c</td>\n",
+       "      <td>Kalamata Whole Olives</td>\n",
+       "      <td>Kalamata Whole Olives</td>\n",
+       "      <td>Compliments Kalamata Whole Olives 375 ml</td>\n",
+       "      <td>Compliments</td>\n",
+       "      <td>375ml</td>\n",
+       "      <td>0|0|0|0|0|0|0|0|0|fat_none|regular|Core|lime|w...</td>\n",
+       "      <td>unknown</td>\n",
+       "      <td>GENERAL_GROCERY</td>\n",
+       "      <td>Core</td>\n",
+       "      <td>voila</td>\n",
+       "      <td>https://voila.ca/products/compliments-kalamata...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>4</th>\n",
+       "      <td>5.574256e+10</td>\n",
+       "      <td>259928EA</td>\n",
+       "      <td>4ff536c8-c957-5fe8-8faa-3f539d6048ef</td>\n",
+       "      <td>Sliced Almonds</td>\n",
+       "      <td>Sliced Almonds</td>\n",
+       "      <td>Compliments Sliced Almonds 250 g</td>\n",
+       "      <td>Compliments</td>\n",
+       "      <td>250g</td>\n",
+       "      <td>0|0|0|0|0|0|0|0|0|fat_none|regular|Core|almond...</td>\n",
+       "      <td>unknown</td>\n",
+       "      <td>GENERAL_GROCERY</td>\n",
+       "      <td>Core</td>\n",
+       "      <td>voila</td>\n",
+       "      <td>https://voila.ca/products/compliments-sliced-a...</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "            upc external_id                              group_id  \\\n",
+       "0  5.574252e+10    297608EA  27b951a3-bef1-5ed4-80b7-7ef5cdca910a   \n",
+       "1  5.574234e+10    464324EA  e608ba58-8f93-59f1-9446-edb91aef79bd   \n",
+       "2  9.900026e+11    144288CS  336c79d8-796d-5682-8195-793883dd97ef   \n",
+       "3  5.574257e+10    312204EA  fd1acb22-4231-5b7b-bde4-8709b31cf04c   \n",
+       "4  5.574256e+10    259928EA  4ff536c8-c957-5fe8-8faa-3f539d6048ef   \n",
+       "\n",
+       "                              group_name  \\\n",
+       "0                    Cold And Sinus Plus   \n",
+       "1  Twister T2 Light Bulbs 23W Soft White   \n",
+       "2                Zero Calorie Soft Drink   \n",
+       "3                  Kalamata Whole Olives   \n",
+       "4                         Sliced Almonds   \n",
+       "\n",
+       "                              core_title  \\\n",
+       "0                    Cold And Sinus Plus   \n",
+       "1  Twister T2 Light Bulbs 23W Soft White   \n",
+       "2                Zero Calorie Soft Drink   \n",
+       "3                  Kalamata Whole Olives   \n",
+       "4                         Sliced Almonds   \n",
+       "\n",
+       "                                      original_title        brand  \\\n",
+       "0           Compliments Cold And Sinus Plus 20 Count  Compliments   \n",
+       "1  Compliments Twister T2 Light Bulbs 23W Soft Wh...  Compliments   \n",
+       "2     Compliments Zero Calorie Soft Drink 12 x 355ml  Compliments   \n",
+       "3           Compliments Kalamata Whole Olives 375 ml  Compliments   \n",
+       "4                   Compliments Sliced Almonds 250 g  Compliments   \n",
+       "\n",
+       "            size                                      identity_hash  \\\n",
+       "0    20 per pack      0|0|0|0|0|0|0|0|0|fat_none|regular|Core|lime|   \n",
+       "1     2 per pack  0|0|0|0|0|0|0|0|0|fat_none|reduced_fat|Core|lime|   \n",
+       "2  12 x 29.583ml      0|0|0|0|0|0|0|0|0|fat_none|regular|Core|lime|   \n",
+       "3          375ml  0|0|0|0|0|0|0|0|0|fat_none|regular|Core|lime|w...   \n",
+       "4           250g  0|0|0|0|0|0|0|0|0|fat_none|regular|Core|almond...   \n",
+       "\n",
+       "  product_domain         taxonomy product_line source  \\\n",
+       "0        unknown  GENERAL_GROCERY         Core  voila   \n",
+       "1        unknown  GENERAL_GROCERY         Core  voila   \n",
+       "2           food        BEVERAGES         Core  voila   \n",
+       "3        unknown  GENERAL_GROCERY         Core  voila   \n",
+       "4        unknown  GENERAL_GROCERY         Core  voila   \n",
+       "\n",
+       "                                          source_url  \n",
+       "0  https://voila.ca/products/compliments-cold-and...  \n",
+       "1  https://voila.ca/products/compliments-twister-...  \n",
+       "2  https://voila.ca/products/compliments-zero-cal...  \n",
+       "3  https://voila.ca/products/compliments-kalamata...  \n",
+       "4  https://voila.ca/products/compliments-sliced-a...  "
+      ]
+     },
+     "execution_count": 2,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "# Colab-compatible paths\n",
+    "PHASE3_URL = \"https://huggingface.co/saraNour/compliments-reference-phase3\"\n",
+    "PHASE2_URL = \"https://huggingface.co/saraNour/compliments-reference-phase2\"\n",
+    "\n",
+    "# For local execution\n",
+    "PHASE3_DIR = Path(\"/home/sara/gsoc/compliments-reference-db/phase3\")\n",
+    "PHASE2_DIR = Path(\"/home/sara/gsoc/compliments-reference-db/phase2\")\n",
+    "\n",
+    "# Load Phase 3 mapping\n",
+    "mapping = pd.read_csv(PHASE3_DIR / \"outputs\" / \"product_group_mapping.csv\")\n",
+    "print(f\"Phase 3 mapping: {len(mapping)} rows, {len(mapping.columns)} columns\")\n",
+    "print(f\"Columns: {list(mapping.columns)}\")\n",
+    "mapping.head()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 3. Load Phase 2 Output (for variant-relevant columns)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2026-07-30T17:47:28.450980Z",
+     "iopub.status.busy": "2026-07-30T17:47:28.450441Z",
+     "iopub.status.idle": "2026-07-30T17:47:28.560182Z",
+     "shell.execute_reply": "2026-07-30T17:47:28.555275Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Phase 2 output: 4440 rows, 38 columns\n",
+      "\n",
+      "Variant-relevant columns:\n",
+      "  external_id: 4440/4440 non-null\n",
+      "  variant_attributes: 4440/4440 non-null\n",
+      "  flavour: 4440/4440 non-null\n",
+      "  formulation: 4440/4440 non-null\n",
+      "  fat_level: 4440/4440 non-null\n",
+      "  fat_percentage: 54/4440 non-null\n",
+      "  size_amount: 4133/4440 non-null\n",
+      "  size_unit: 4133/4440 non-null\n",
+      "  size_qty: 4440/4440 non-null\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Load Phase 2 output for variant-relevant columns\n",
+    "p2 = pd.read_parquet(PHASE2_DIR / \"outputs\" / \"phase2_output.parquet\")\n",
+    "print(f\"Phase 2 output: {len(p2)} rows, {len(p2.columns)} columns\")\n",
+    "\n",
+    "# Show variant-relevant columns\n",
+    "variant_cols = [\"external_id\", \"variant_attributes\", \"flavour\", \"formulation\",\n",
+    "                \"fat_level\", \"fat_percentage\", \"size_amount\", \"size_unit\", \"size_qty\"]\n",
+    "print(f\"\\nVariant-relevant columns:\")\n",
+    "for col in variant_cols:\n",
+    "    if col in p2.columns:\n",
+    "        non_null = p2[col].notna().sum()\n",
+    "        print(f\"  {col}: {non_null}/{len(p2)} non-null\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 4. Merge Phase 3 with Phase 2 Variant Data"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2026-07-30T17:47:28.565519Z",
+     "iopub.status.busy": "2026-07-30T17:47:28.565008Z",
+     "iopub.status.idle": "2026-07-30T17:47:28.595177Z",
+     "shell.execute_reply": "2026-07-30T17:47:28.592197Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Merged: 4440 rows, 22 columns\n",
+      "\n",
+      "Columns: ['upc', 'external_id', 'group_id', 'group_name', 'core_title', 'original_title', 'brand', 'size', 'identity_hash', 'product_domain', 'taxonomy', 'product_line', 'source', 'source_url', 'variant_attributes', 'flavour', 'formulation', 'fat_level', 'fat_percentage', 'size_amount', 'size_unit', 'size_qty']\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Merge\n",
+    "df = mapping.merge(p2[variant_cols], on=\"external_id\", how=\"left\")\n",
+    "print(f\"Merged: {len(df)} rows, {len(df.columns)} columns\")\n",
+    "print(f\"\\nColumns: {list(df.columns)}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 5. Variant Key Generation"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2026-07-30T17:47:28.600357Z",
+     "iopub.status.busy": "2026-07-30T17:47:28.599840Z",
+     "iopub.status.idle": "2026-07-30T17:47:29.449331Z",
+     "shell.execute_reply": "2026-07-30T17:47:29.446016Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Unique variant keys: 631\n",
+      "\n",
+      "Sample variant keys:\n",
+      "                                                     original_title           size             variant_key\n",
+      "0                          Compliments Cold And Sinus Plus 20 Count    20 per pack                 count20\n",
+      "1            Compliments Twister T2 Light Bulbs 23W Soft White 2 EA     2 per pack                  count2\n",
+      "2                    Compliments Zero Calorie Soft Drink 12 x 355ml  12 x 29.583ml  amt29.583|unitml|qty12\n",
+      "3                          Compliments Kalamata Whole Olives 375 ml          375ml         amt375.0|unitml\n",
+      "4                                  Compliments Sliced Almonds 250 g           250g          amt250.0|unitg\n",
+      "5                Compliments Unisex Protective Underwear L 18 Count    18 per pack                 count18\n",
+      "6        Compliments Latex Gloves Unlined Waxed Medium/Large 1 Pack            NaN                  nosize\n",
+      "7  Compliments Gluten-Free Tuxedo Chocolate Chip Cookie Dough 400 g           400g          amt400.0|unitg\n",
+      "8                          Compliments Taco Kit Hard And Soft 393 g           393g          amt393.0|unitg\n",
+      "9                    Compliments Naturally Simple Turkey Jerky 80 g            80g           amt80.0|unitg\n"
+     ]
+    }
+   ],
+   "source": [
+    "def build_variant_key(row):\n",
+    "    \"\"\"Build deterministic variant key from variant attributes.\"\"\"\n",
+    "    va_str = row.get(\"variant_attributes\", \"{}\")\n",
+    "    if isinstance(va_str, str):\n",
+    "        try:\n",
+    "            va = json.loads(va_str)\n",
+    "        except (json.JSONDecodeError, TypeError):\n",
+    "            va = {}\n",
+    "    elif isinstance(va_str, dict):\n",
+    "        va = va_str\n",
+    "    else:\n",
+    "        va = {}\n",
+    "    \n",
+    "    amount = va.get(\"amount\")\n",
+    "    unit = va.get(\"unit\", \"\")\n",
+    "    qty = va.get(\"qty\", 1)\n",
+    "    count = va.get(\"count\")\n",
+    "    multiplier = va.get(\"multiplier\", 1)\n",
+    "    raw = va.get(\"raw\", \"\")\n",
+    "    \n",
+    "    parts = []\n",
+    "    if count is not None:\n",
+    "        parts.append(f\"count{int(count)}\")\n",
+    "    elif amount is not None:\n",
+    "        parts.append(f\"amt{amount}\")\n",
+    "        if unit:\n",
+    "            parts.append(f\"unit{unit}\")\n",
+    "        if qty and qty > 1:\n",
+    "            parts.append(f\"qty{int(qty)}\")\n",
+    "        if multiplier and multiplier > 1:\n",
+    "            parts.append(f\"mult{int(multiplier)}\")\n",
+    "    elif raw:\n",
+    "        parts.append(f\"raw{raw.strip().lower()}\")\n",
+    "    else:\n",
+    "        parts.append(\"nosize\")\n",
+    "    \n",
+    "    return \"|\".join(parts)\n",
+    "\n",
+    "# Apply\n",
+    "df[\"variant_key\"] = df.apply(lambda row: build_variant_key(row.to_dict()), axis=1)\n",
+    "print(f\"Unique variant keys: {df['variant_key'].nunique()}\")\n",
+    "print(f\"\\nSample variant keys:\")\n",
+    "print(df[[\"original_title\", \"size\", \"variant_key\"]].head(10).to_string())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 6. Variant ID Generation"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2026-07-30T17:47:29.454379Z",
+     "iopub.status.busy": "2026-07-30T17:47:29.453884Z",
+     "iopub.status.idle": "2026-07-30T17:47:29.672800Z",
+     "shell.execute_reply": "2026-07-30T17:47:29.668771Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Total unique variants: 4299\n",
+      "Total products: 4440\n",
+      "Total groups: 3691\n"
+     ]
+    }
+   ],
+   "source": [
+    "def generate_variant_id(group_id, variant_key):\n",
+    "    \"\"\"Generate deterministic variant_id from group_id + variant_key.\"\"\"\n",
+    "    namespace = uuid.NAMESPACE_DNS\n",
+    "    name = f\"{group_id}||{variant_key}\"\n",
+    "    return str(uuid.uuid5(namespace, name))\n",
+    "\n",
+    "# Apply\n",
+    "df[\"variant_id\"] = df.apply(\n",
+    "    lambda row: generate_variant_id(row[\"group_id\"], row[\"variant_key\"]),\n",
+    "    axis=1\n",
+    ")\n",
+    "\n",
+    "print(f\"Total unique variants: {df['variant_id'].nunique()}\")\n",
+    "print(f\"Total products: {len(df)}\")\n",
+    "print(f\"Total groups: {df['group_id'].nunique()}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 7. Build Output Tables"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2026-07-30T17:47:29.683063Z",
+     "iopub.status.busy": "2026-07-30T17:47:29.682522Z",
+     "iopub.status.idle": "2026-07-30T17:47:46.834219Z",
+     "shell.execute_reply": "2026-07-30T17:47:46.823795Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Variant table: 4299 variants\n",
+      "Variant mapping: 4440 products\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Variant summary: 3691 groups\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Variant table (one row per unique variant)\n",
+    "variant_rows = []\n",
+    "for vid, grp in df.groupby(\"variant_id\"):\n",
+    "    row = grp.iloc[0]\n",
+    "    sizes = grp[\"size\"].dropna().unique().tolist()\n",
+    "    variant_rows.append({\n",
+    "        \"variant_id\": vid,\n",
+    "        \"group_id\": row[\"group_id\"],\n",
+    "        \"group_name\": row[\"group_name\"],\n",
+    "        \"brand\": row[\"brand\"],\n",
+    "        \"product_line\": row[\"product_line\"],\n",
+    "        \"product_domain\": row[\"product_domain\"],\n",
+    "        \"taxonomy\": row[\"taxonomy\"],\n",
+    "        \"core_title\": row[\"core_title\"],\n",
+    "        \"variant_key\": row[\"variant_key\"],\n",
+    "        \"size\": sizes[0] if len(sizes) == 1 else \"; \".join(str(s) for s in sizes),\n",
+    "        \"size_amount\": row.get(\"size_amount\"),\n",
+    "        \"size_unit\": row.get(\"size_unit\"),\n",
+    "        \"size_qty\": row.get(\"size_qty\"),\n",
+    "        \"flavour\": row.get(\"flavour\"),\n",
+    "        \"formulation\": row.get(\"formulation\"),\n",
+    "        \"fat_level\": row.get(\"fat_level\"),\n",
+    "        \"fat_percentage\": row.get(\"fat_percentage\"),\n",
+    "        \"product_count\": len(grp),\n",
+    "        \"unique_upcs\": grp[\"upc\"].nunique(),\n",
+    "        \"source\": row[\"source\"],\n",
+    "        \"source_url\": row[\"source_url\"],\n",
+    "    })\n",
+    "variant_table = pd.DataFrame(variant_rows)\n",
+    "print(f\"Variant table: {len(variant_table)} variants\")\n",
+    "\n",
+    "# Variant mapping\n",
+    "variant_mapping = df[[\"external_id\", \"upc\", \"group_id\", \"variant_id\",\n",
+    "                      \"core_title\", \"original_title\", \"brand\", \"size\",\n",
+    "                      \"variant_key\", \"source\", \"source_url\"]].copy()\n",
+    "print(f\"Variant mapping: {len(variant_mapping)} products\")\n",
+    "\n",
+    "# Variant summary\n",
+    "summary_rows = []\n",
+    "for gid, grp in df.groupby(\"group_id\"):\n",
+    "    variant_ids = grp[\"variant_id\"].unique().tolist()\n",
+    "    row = grp.iloc[0]\n",
+    "    summary_rows.append({\n",
+    "        \"group_id\": gid,\n",
+    "        \"group_name\": row[\"group_name\"],\n",
+    "        \"brand\": row[\"brand\"],\n",
+    "        \"product_domain\": row[\"product_domain\"],\n",
+    "        \"taxonomy\": row[\"taxonomy\"],\n",
+    "        \"variant_count\": len(variant_ids),\n",
+    "        \"variant_ids\": \", \".join(variant_ids),\n",
+    "        \"product_count\": len(grp),\n",
+    "    })\n",
+    "variant_summary = pd.DataFrame(summary_rows)\n",
+    "print(f\"Variant summary: {len(variant_summary)} groups\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 8. Validation"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2026-07-30T17:47:46.850462Z",
+     "iopub.status.busy": "2026-07-30T17:47:46.849394Z",
+     "iopub.status.idle": "2026-07-30T17:47:46.900741Z",
+     "shell.execute_reply": "2026-07-30T17:47:46.899086Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "=== VALIDATION ===\n",
+      "\n",
+      "Rule 1 - Every product has one variant: True (max variants per product: 1)\n",
+      "Rule 2 - Every variant belongs to one group: True (max groups per variant: 1)\n",
+      "Rule 3 - No product maps to multiple variants: True (products with multiple: 0)\n",
+      "Rule 4 - Variant count matches: True (mapping: 4299, table: 4299)\n",
+      "Rule 5 - All products have group_id: True (null: 0)\n",
+      "Rule 6 - All products have variant_id: True (null: 0)\n",
+      "\n",
+      "Overall: PASS\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Validation rules\n",
+    "print(\"=== VALIDATION ===\")\n",
+    "print()\n",
+    "\n",
+    "# Rule 1: Every product has exactly one variant\n",
+    "max_variants_per_product = df.groupby(\"external_id\")[\"variant_id\"].nunique().max()\n",
+    "print(f\"Rule 1 - Every product has one variant: {max_variants_per_product == 1} (max variants per product: {max_variants_per_product})\")\n",
+    "\n",
+    "# Rule 2: Every variant belongs to exactly one group\n",
+    "max_groups_per_variant = variant_table.groupby(\"variant_id\")[\"group_id\"].nunique().max()\n",
+    "print(f\"Rule 2 - Every variant belongs to one group: {max_groups_per_variant == 1} (max groups per variant: {max_groups_per_variant})\")\n",
+    "\n",
+    "# Rule 3: No product maps to multiple variants\n",
+    "products_with_multiple = (df.groupby(\"external_id\")[\"variant_id\"].nunique() > 1).sum()\n",
+    "print(f\"Rule 3 - No product maps to multiple variants: {products_with_multiple == 0} (products with multiple: {products_with_multiple})\")\n",
+    "\n",
+    "# Rule 4: Variant count matches\n",
+    "mapping_variants = variant_mapping[\"variant_id\"].nunique()\n",
+    "table_variants = len(variant_table)\n",
+    "print(f\"Rule 4 - Variant count matches: {mapping_variants == table_variants} (mapping: {mapping_variants}, table: {table_variants})\")\n",
+    "\n",
+    "# Rule 5: All products have group_id\n",
+    "null_groups = df[\"group_id\"].isna().sum()\n",
+    "print(f\"Rule 5 - All products have group_id: {null_groups == 0} (null: {null_groups})\")\n",
+    "\n",
+    "# Rule 6: All products have variant_id\n",
+    "null_variants = df[\"variant_id\"].isna().sum()\n",
+    "print(f\"Rule 6 - All products have variant_id: {null_variants == 0} (null: {null_variants})\")\n",
+    "\n",
+    "print()\n",
+    "print(\"Overall: PASS\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 9. Variant Distribution Statistics"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2026-07-30T17:47:46.915155Z",
+     "iopub.status.busy": "2026-07-30T17:47:46.914599Z",
+     "iopub.status.idle": "2026-07-30T17:47:46.952803Z",
+     "shell.execute_reply": "2026-07-30T17:47:46.950330Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "=== VARIANT DISTRIBUTION ===\n",
+      "Total products: 4440\n",
+      "Total groups: 3691\n",
+      "Total variants: 4299\n",
+      "\n",
+      "Groups by variant count:\n",
+      "  1 variant(s): 3235 groups\n",
+      "  2 variant(s): 357 groups\n",
+      "  3 variant(s): 67 groups\n",
+      "  4 variant(s): 22 groups\n",
+      "  5 variant(s): 5 groups\n",
+      "  6 variant(s): 2 groups\n",
+      "  7 variant(s): 2 groups\n",
+      "  10 variant(s): 1 groups\n",
+      "\n",
+      "Groups with 1 variant: 3235\n",
+      "Groups with >1 variants: 456\n",
+      "Max variants per group: 10\n",
+      "Avg variants per group: 1.16\n",
+      "\n",
+      "Missing size:\n",
+      "  Products with missing size: 215\n",
+      "  Products with no-size variant: 215\n"
+     ]
+    }
+   ],
+   "source": [
+    "print(\"=== VARIANT DISTRIBUTION ===\")\n",
+    "print(f\"Total products: {len(df)}\")\n",
+    "print(f\"Total groups: {df['group_id'].nunique()}\")\n",
+    "print(f\"Total variants: {df['variant_id'].nunique()}\")\n",
+    "print()\n",
+    "\n",
+    "print(\"Groups by variant count:\")\n",
+    "vc = variant_summary[\"variant_count\"].value_counts().sort_index()\n",
+    "for count, num_groups in vc.items():\n",
+    "    print(f\"  {count} variant(s): {num_groups} groups\")\n",
+    "print()\n",
+    "\n",
+    "print(f\"Groups with 1 variant: {(variant_summary['variant_count'] == 1).sum()}\")\n",
+    "print(f\"Groups with >1 variants: {(variant_summary['variant_count'] > 1).sum()}\")\n",
+    "print(f\"Max variants per group: {variant_summary['variant_count'].max()}\")\n",
+    "print(f\"Avg variants per group: {variant_summary['variant_count'].mean():.2f}\")\n",
+    "print()\n",
+    "\n",
+    "print(\"Missing size:\")\n",
+    "print(f\"  Products with missing size: {df['size'].isna().sum()}\")\n",
+    "print(f\"  Products with no-size variant: {(df['variant_key'] == 'nosize').sum()}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 10. Peanut Butter Example"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2026-07-30T17:47:46.980676Z",
+     "iopub.status.busy": "2026-07-30T17:47:46.980188Z",
+     "iopub.status.idle": "2026-07-30T17:47:47.082551Z",
+     "shell.execute_reply": "2026-07-30T17:47:47.080237Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "=== PEANUT BUTTER EXAMPLE ===\n",
+      "Total products: 29\n",
+      "Groups: 21\n",
+      "\n",
+      "\n",
+      "Group: 03e904ad...\n",
+      "  Products: 1\n",
+      "    - Compliments Milk Chocolate Peanut Butter Bites 340 g\n",
+      "      size: 340g, variant_key: amt340.0|unitg\n",
+      "      variant_id: 8420e30e...\n",
+      "\n",
+      "Group: 0da73841...\n",
+      "  Products: 1\n",
+      "    - Compliments Snack Peanut Butter Wafer Rolls Dippers Raspberry 240 g\n",
+      "      size: 240g, variant_key: amt240.0|unitg\n",
+      "      variant_id: c3b0a6db...\n",
+      "\n",
+      "Group: 104375ed...\n",
+      "  Products: 1\n",
+      "    - Compliments Peanut Butter With Honey 500 g\n",
+      "      size: 500g, variant_key: amt500.0|unitg\n",
+      "      variant_id: 32a679d6...\n",
+      "\n",
+      "Group: 1375b508...\n",
+      "  Products: 1\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "    - Compliments Ice Cream Chocolate Peanut Butter 1.5 L\n",
+      "      size: 1.5L, variant_key: amt1.5|unitl\n",
+      "      variant_id: 0e76097e...\n",
+      "\n",
+      "Group: 1845cac0...\n",
+      "  Products: 1\n",
+      "    - Compliments Filled Pretzels Peanut Butter 680 g\n",
+      "      size: 680g, variant_key: amt680.0|unitg\n",
+      "      variant_id: 8be4f42a...\n",
+      "\n",
+      "Group: 1ce8dc8f...\n",
+      "  Products: 1\n",
+      "    - Compliments S'mores Kit Peanut Butter 446 g\n",
+      "      size: 446g, variant_key: amt446.0|unitg\n",
+      "      variant_id: 5f4a17f1...\n",
+      "\n",
+      "Group: 1d7958d7...\n",
+      "  Products: 1\n",
+      "    - Compliments Snack Peanut Butter Wafer Rolls With Dipper Strawberry 240\n",
+      "      size: 240g, variant_key: amt240.0|unitg\n",
+      "      variant_id: 09227ea6...\n",
+      "\n",
+      "Group: 35e31165...\n",
+      "  Products: 1\n",
+      "    - Compliments Dipper Sticks Peanut Butter Chocolate 240 g\n",
+      "      size: 240g, variant_key: amt240.0|unitg\n",
+      "      variant_id: 5dc08c1f...\n",
+      "\n",
+      "Group: 3822ff9a...\n",
+      "  Products: 1\n",
+      "    - Compliments Organic Peanut Butter Crunchy 500 g\n",
+      "      size: 500g, variant_key: amt500.0|unitg\n",
+      "      variant_id: f8187cb1...\n",
+      "\n",
+      "Group: 5bfec681...\n",
+      "  Products: 3\n",
+      "    - Compliments Light Smooth Peanut Butter 2 kg\n",
+      "      size: 2kg, variant_key: amt2.0|unitkg\n",
+      "      variant_id: fb9c8e53...\n",
+      "    - Compliments Smooth Light Peanut Butter 1 kg\n",
+      "      size: 1kg, variant_key: amt1.0|unitkg\n",
+      "      variant_id: 55dff053...\n",
+      "    - Compliments Smooth Light Peanut Butter 500 g\n",
+      "      size: 500g, variant_key: amt500.0|unitg\n",
+      "      variant_id: 6b9bfb30...\n",
+      "\n",
+      "Group: 805b971b...\n",
+      "  Products: 1\n",
+      "    - Compliments Smooth Peanut Butter Dark Roast 1 kg\n",
+      "      size: 1kg, variant_key: amt1.0|unitkg\n",
+      "      variant_id: f3d988cd...\n",
+      "\n",
+      "Group: 97ad569d...\n",
+      "  Products: 1\n",
+      "    - Compliments Naturally Simple Peanut Butter Crunchy 100% Natural 500 g\n",
+      "      size: 500g, variant_key: amt500.0|unitg\n",
+      "      variant_id: 3f35082c...\n",
+      "\n",
+      "Group: b09ff83f...\n",
+      "  Products: 3\n",
+      "    - Compliments Smooth Peanut Butter 500 g\n",
+      "      size: 500g, variant_key: amt500.0|unitg\n",
+      "      variant_id: 9d9449be...\n",
+      "    - Compliments Peanut Butter Smooth 2 kg\n",
+      "      size: 2kg, variant_key: amt2.0|unitkg\n",
+      "      variant_id: eb2853a9...\n",
+      "    - Compliments Smooth Peanut Butter 1 kg\n",
+      "      size: 1kg, variant_key: amt1.0|unitkg\n",
+      "      variant_id: c61a76b9...\n",
+      "\n",
+      "Group: bc4d1309...\n",
+      "  Products: 3\n",
+      "    - Compliments Peanut Butter Crunchy 2 kg\n",
+      "      size: 2kg, variant_key: amt2.0|unitkg\n",
+      "      variant_id: fab2bbd7...\n",
+      "    - Compliments Peanut Butter Crunchy 500 g\n",
+      "      size: 500g, variant_key: amt500.0|unitg\n",
+      "      variant_id: 59e1c2c7...\n",
+      "    - Compliments Peanut Butter Crunchy 1 kg\n",
+      "      size: 1kg, variant_key: amt1.0|unitkg\n",
+      "      variant_id: 8f9d4c80...\n",
+      "\n",
+      "Group: d36e44e9...\n",
+      "  Products: 1\n",
+      "    - Compliments Dipped Granola Bars Peanut Butter 156 g\n",
+      "      size: 156g, variant_key: amt156.0|unitg\n",
+      "      variant_id: 9a7500bb...\n",
+      "\n",
+      "Group: d9eb5924...\n",
+      "  Products: 1\n",
+      "    - Compliments Baking Chips Peanut Butter 225 g\n",
+      "      size: 225g, variant_key: amt225.0|unitg\n",
+      "      variant_id: 72b1a7ad...\n",
+      "\n",
+      "Group: e78cdee5...\n",
+      "  Products: 1\n",
+      "    - Compliments Organic Peanut Butter Smooth 500 g\n",
+      "      size: 500g, variant_key: amt500.0|unitg\n",
+      "      variant_id: 47be7869...\n",
+      "\n",
+      "Group: e9bc48d6...\n",
+      "  Products: 2\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "    - Compliments Naturally Simple 100% Natural Smooth Peanut Butter 1 kg\n",
+      "      size: 1kg, variant_key: amt1.0|unitkg\n",
+      "      variant_id: d5988560...\n",
+      "    - Compliments Naturally Simple 100% Natural Smooth Peanut Butter 500 g\n",
+      "      size: 500g, variant_key: amt500.0|unitg\n",
+      "      variant_id: 288a8f40...\n",
+      "\n",
+      "Group: f1c7a47b...\n",
+      "  Products: 1\n",
+      "    - Compliments Peanut Butter Spread Cinnamon Sugar 500 g\n",
+      "      size: 500g, variant_key: amt500.0|unitg\n",
+      "      variant_id: f49632b9...\n",
+      "\n",
+      "Group: f570c38e...\n",
+      "  Products: 2\n",
+      "    - Compliments Cookies Peanut Butter 300 g\n",
+      "      size: 300g, variant_key: amt300.0|unitg\n",
+      "      variant_id: b06ef2f1...\n",
+      "    - Compliments Cookies Peanut Butter 300 g\n",
+      "      size: 300g, variant_key: amt300.0|unitg\n",
+      "      variant_id: b06ef2f1...\n",
+      "\n",
+      "Group: fea9d818...\n",
+      "  Products: 1\n",
+      "    - Compliments Dog Treat Chewies Peanut Butter 80 g\n",
+      "      size: 80g, variant_key: amt80.0|unitg\n",
+      "      variant_id: 3e98cc6a...\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Find peanut butter products\n",
+    "pb = df[df[\"original_title\"].str.contains(\"peanut butter\", case=False, na=False)].copy()\n",
+    "print(f\"=== PEANUT BUTTER EXAMPLE ===\")\n",
+    "print(f\"Total products: {len(pb)}\")\n",
+    "print(f\"Groups: {pb['group_id'].nunique()}\")\n",
+    "print()\n",
+    "\n",
+    "for gid, grp in pb.groupby(\"group_id\"):\n",
+    "    print(f\"\\nGroup: {gid[:8]}...\")\n",
+    "    print(f\"  Products: {len(grp)}\")\n",
+    "    for _, row in grp.iterrows():\n",
+    "        print(f\"    - {row['original_title'][:70]}\")\n",
+    "        print(f\"      size: {row['size']}, variant_key: {row['variant_key']}\")\n",
+    "        print(f\"      variant_id: {row['variant_id'][:8]}...\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 11. Save Outputs"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2026-07-30T17:47:47.086903Z",
+     "iopub.status.busy": "2026-07-30T17:47:47.086320Z",
+     "iopub.status.idle": "2026-07-30T17:47:47.631025Z",
+     "shell.execute_reply": "2026-07-30T17:47:47.629735Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Outputs saved:\n",
+      "  reference_product_variants.parquet (4299 rows)\n",
+      "  product_variant_mapping.parquet (4440 rows)\n",
+      "  reference_product_variant_summary.parquet (3691 rows)\n",
+      "  phase4_peanut_butter_example.parquet (29 rows)\n"
+     ]
+    }
+   ],
+   "source": [
+    "OUTPUT_DIR = Path(\"/home/sara/gsoc/compliments-reference-db/phase4/outputs\")\n",
+    "VALIDATION_DIR = Path(\"/home/sara/gsoc/compliments-reference-db/phase4/validation\")\n",
+    "STATISTICS_DIR = Path(\"/home/sara/gsoc/compliments-reference-db/phase4/statistics\")\n",
+    "AUDIT_DIR = Path(\"/home/sara/gsoc/compliments-reference-db/phase4/audit\")\n",
+    "\n",
+    "OUTPUT_DIR.mkdir(parents=True, exist_ok=True)\n",
+    "VALIDATION_DIR.mkdir(parents=True, exist_ok=True)\n",
+    "STATISTICS_DIR.mkdir(parents=True, exist_ok=True)\n",
+    "AUDIT_DIR.mkdir(parents=True, exist_ok=True)\n",
+    "\n",
+    "# Save Parquet outputs\n",
+    "variant_table.to_parquet(OUTPUT_DIR / \"reference_product_variants.parquet\", index=False)\n",
+    "variant_mapping.to_parquet(OUTPUT_DIR / \"product_variant_mapping.parquet\", index=False)\n",
+    "variant_summary.to_parquet(OUTPUT_DIR / \"reference_product_variant_summary.parquet\", index=False)\n",
+    "\n",
+    "# Save CSV for inspection\n",
+    "variant_table.to_csv(OUTPUT_DIR / \"reference_product_variants.csv\", index=False)\n",
+    "variant_mapping.to_csv(OUTPUT_DIR / \"product_variant_mapping.csv\", index=False)\n",
+    "variant_summary.to_csv(OUTPUT_DIR / \"reference_product_variant_summary.csv\", index=False)\n",
+    "\n",
+    "# Save Peanut Butter example\n",
+    "pb.to_parquet(AUDIT_DIR / \"phase4_peanut_butter_example.parquet\", index=False)\n",
+    "\n",
+    "print(\"Outputs saved:\")\n",
+    "print(f\"  reference_product_variants.parquet ({len(variant_table)} rows)\")\n",
+    "print(f\"  product_variant_mapping.parquet ({len(variant_mapping)} rows)\")\n",
+    "print(f\"  reference_product_variant_summary.parquet ({len(variant_summary)} rows)\")\n",
+    "print(f\"  phase4_peanut_butter_example.parquet ({len(pb)} rows)\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 12. Verify Outputs"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2026-07-30T17:47:47.636242Z",
+     "iopub.status.busy": "2026-07-30T17:47:47.635650Z",
+     "iopub.status.idle": "2026-07-30T17:47:47.691128Z",
+     "shell.execute_reply": "2026-07-30T17:47:47.688410Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "=== VERIFY OUTPUTS ===\n",
+      "reference_product_variants.parquet: 4299 rows, 21 columns\n",
+      "product_variant_mapping.parquet: 4440 rows, 11 columns\n",
+      "reference_product_variant_summary.parquet: 3691 rows, 8 columns\n",
+      "\n",
+      "All outputs verified successfully!\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Verify outputs can be reloaded\n",
+    "print(\"=== VERIFY OUTPUTS ===\")\n",
+    "\n",
+    "vt = pd.read_parquet(OUTPUT_DIR / \"reference_product_variants.parquet\")\n",
+    "print(f\"reference_product_variants.parquet: {len(vt)} rows, {len(vt.columns)} columns\")\n",
+    "\n",
+    "vm = pd.read_parquet(OUTPUT_DIR / \"product_variant_mapping.parquet\")\n",
+    "print(f\"product_variant_mapping.parquet: {len(vm)} rows, {len(vm.columns)} columns\")\n",
+    "\n",
+    "vs = pd.read_parquet(OUTPUT_DIR / \"reference_product_variant_summary.parquet\")\n",
+    "print(f\"reference_product_variant_summary.parquet: {len(vs)} rows, {len(vs.columns)} columns\")\n",
+    "\n",
+    "print()\n",
+    "print(\"All outputs verified successfully!\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 13. Output Summary"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2026-07-30T17:47:47.696006Z",
+     "iopub.status.busy": "2026-07-30T17:47:47.695480Z",
+     "iopub.status.idle": "2026-07-30T17:47:47.711328Z",
+     "shell.execute_reply": "2026-07-30T17:47:47.707589Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "=== PHASE 4 COMPLETE ===\n",
+      "Version: 1.0.0\n",
+      "Input:  4440 products, 3691 groups\n",
+      "Output: 4299 variants, 4440 products mapped\n",
+      "Validation: PASS\n",
+      "\n",
+      "No LLM used.\n",
+      "No nutrition data used.\n",
+      "No external taxonomy used.\n"
+     ]
+    }
+   ],
+   "source": [
+    "print(\"=== PHASE 4 COMPLETE ===\")\n",
+    "print(f\"Version: 1.0.0\")\n",
+    "print(f\"Input:  {len(df)} products, {df['group_id'].nunique()} groups\")\n",
+    "print(f\"Output: {len(variant_table)} variants, {len(variant_mapping)} products mapped\")\n",
+    "print(f\"Validation: PASS\")\n",
+    "print(f\"\\nNo LLM used.\")\n",
+    "print(f\"No nutrition data used.\")\n",
+    "print(f\"No external taxonomy used.\")"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.13.7"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}