saraNour/compliments-reference-phase4

Phase 4 — Variant Assignment / Variant Modeling

Compliments Reference DB Pipeline

Version: 1.0.0 Date: 2026-07-30 Status: PASS

---

Purpose

Phase 4 transforms the Phase 3 product-group representation into a structured product variant model. Products belonging to the same reference product group can have different purchasable variants such as size, package quantity, formulation, flavour, fat level, etc.

---

Key Concepts

Reference Product Group (from Phase 3)

A reference product group represents the same underlying product identity. All products in a group share the same identity attributes (organic, gluten-free, fat level, flavour, formulation, product_line).

Variant

A variant is a purchasable representation of the same reference product that differs in attributes such as:

• Size: 500 g, 1 kg, 2 kg
• Package quantity: 1 × 500 g, 2 × 500 g
• Pack configuration: single, multipack, case

Identifier Hierarchy

Identifier	Level	Description
external_id	Source product	Voila retailer product ID (unique per product)
upc	Source product	Barcode (may be reused across variants)
group_id	Reference group	Deterministic UUID5 from Phase 3
variant_id	Variant	Deterministic UUID5 from group_id + variant_key

---

Input Datasets

Dataset	Source	Description
Phase 3 mapping	phase3/outputs/product_group_mapping.csv	Product-to-group mapping
Phase 2 output	phase2/outputs/phase2_output.parquet	Variant-relevant attributes

---

What Phase 4 Does

1. Loads Phase 3 outputs — product_group_mapping.csv
2. Loads Phase 2 output — for variant_attributes, flavour, formulation, fat_level, fat_percentage
3. Merges data — combines group assignments with variant attributes
4. Builds variant keys — deterministic keys from size/package attributes
5. Generates variant IDs — deterministic UUID5 from group_id + variant_key
6. Creates variant tables — main variant table, mapping, summary
7. Validates — checks all consistency rules
8. Generates audit — Peanut Butter example

---

Variant Key Logic

The variant key captures size/package information:

If count-based: "count{count}"
If size-based: "amt{amount}|unit{unit}[|qty{qty}][|mult{multiplier}]"
If raw: "raw{raw_string}"
If missing: "nosize"

Examples:

• "amt500.0|unitg" → 500 g
• "amt1.0|unitkg" → 1 kg
• "amt12|unitml|qty12" → 12 × 12 ml
• "count20" → 20 per pack
• "nosize" → size missing

---

Variant ID Generation

variant_id = uuid5(NAMESPACE_DNS, f"{group_id}||{variant_key}")

This is:

• Deterministic: Same input → same output
• Reproducible: Runs produce identical IDs
• Collision-resistant: UUID5 namespace-based

---

Output Files

reference_product_variants.parquet

One row per unique variant.

Column	Type	Description
variant_id	str	Deterministic UUID5
group_id	str	Reference group UUID
group_name	str	Most frequent core_title
brand	str	Normalized brand
product_line	str	Product line
product_domain	str	food/non_food/unknown
taxonomy	str	Taxonomy category
core_title	str	Core product title
variant_key	str	Deterministic variant key
size	str	Size string
size_amount	float	Numeric size amount
size_unit	str	Size unit
size_qty	int	Quantity multiplier
flavour	list	Flavour keywords
formulation	list	Formulation keywords
fat_level	str	Fat level
fat_percentage	float	Fat percentage
product_count	int	Products in this variant
unique_upcs	int	Unique UPCs in this variant
source	str	Data source
source_url	str	Source URL

product_variant_mapping.parquet

Product-to-variant mapping.

Column	Type	Description
external_id	str	Source product ID
upc	str	Barcode
group_id	str	Reference group UUID
variant_id	str	Variant UUID
core_title	str	Core title
original_title	str	Raw title
brand	str	Normalized brand
size	str	Size string
variant_key	str	Deterministic variant key
source	str	Data source
source_url	str	Source URL

reference_product_variant_summary.parquet

Group-level variant summary.

Column	Type	Description
group_id	str	Reference group UUID
group_name	str	Most frequent core_title
brand	str	Normalized brand
product_domain	str	food/non_food/unknown
taxonomy	str	Taxonomy category
variant_count	int	Number of variants
variant_ids	str	Comma-separated variant IDs
product_count	int	Total products in group

---

Validation Rules

Rule	Description	Status
1	Every product has exactly one variant	PASS
2	Every variant belongs to exactly one group	PASS
3	No product maps to multiple variants	PASS
4	Variant count matches across tables	PASS
5	All products have group_id	PASS
6	All products have variant_id	PASS

---

Handling Missing Size

• Products with missing size receive variant_key = "nosize"
• They still get deterministic variant_ids
• They are NOT merged together just because size is missing
• Phase 3 group identity remains authoritative

---

Files

phase4/
├── README.md
├── src/
│   └── phase4.py
├── notebooks/
│   ├── phase4.ipynb
│   └── phase4_executed.ipynb
├── outputs/
│   ├── reference_product_variants.parquet
│   ├── reference_product_variants.csv
│   ├── product_variant_mapping.parquet
│   ├── product_variant_mapping.csv
│   ├── reference_product_variant_summary.parquet
│   └── reference_product_variant_summary.csv
├── validation/
│   └── phase4_validation.json
├── statistics/
│   └── phase4_statistics.json
└── audit/
    └── phase4_peanut_butter_example.parquet

---

How to Run

cd /home/sara/gsoc/compliments-reference-db
python -m phase4.src.phase4

---

LLM Usage

None. All variant assignment is deterministic.

Nutrition Usage

None. This phase does NOT load or use nutrition data.

External Datasets

None. This phase does NOT use GPC, Agribalyse, or any external taxonomy.

---

Traceability

source_of_truth/products.parquet
        ↓
Phase 1 output (phase1_output.parquet)
        ↓
Phase 2 output (phase2_output.parquet)
        ↓
Phase 3 product_group_mapping.csv
        ↓
Phase 4 product_variant_mapping.parquet
        ↓
Phase 4 reference_product_variants.parquet

---

Known Limitations

• Variant assignment is based solely on size/package attributes
• Products with identical size but different packaging (e.g., bottle vs jar) cannot be distinguished
• The current dataset does not contain packaging type information

---

Recommendation for Phase 5

Phase 5 (Nutrition Merge) should:

1. Load nutrition.parquet from the authoritative source
2. Join with product_variant_mapping.parquet on external_id
3. Create a merged product-nutrition table
4. Preserve all variant and group information