saraNour commited on
Commit
eb3c596
·
verified ·
1 Parent(s): f55abbf

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +153 -0
README.md ADDED
@@ -0,0 +1,153 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Phase 1 — Load / Validate / Provenance
2
+
3
+ ## Compliments Reference DB Pipeline
4
+
5
+ **Version:** 1.0.0
6
+ **Date:** 2026-07-30
7
+ **Status:** PASS
8
+
9
+ ---
10
+
11
+ ## Purpose
12
+
13
+ Phase 1 loads the authoritative Compliments product dataset, validates its schema, inspects data quality, documents provenance, and produces a clean validated output for Phase 2.
14
+
15
+ ---
16
+
17
+ ## Authoritative Input
18
+
19
+ | Property | Value |
20
+ |----------|-------|
21
+ | Source | `saraNour/compliments-brand/source_of_truth/products.parquet` |
22
+ | URL | https://huggingface.co/datasets/saraNour/compliments-brand/blob/main/source_of_truth/products.parquet |
23
+ | Type | HuggingFace dataset (private) |
24
+ | Original Source | Voila.ca (Loblaw) Compliments private-label products |
25
+ | Row Count | 4,440 |
26
+ | Column Count | 16 |
27
+
28
+ ---
29
+
30
+ ## What Phase 1 Does
31
+
32
+ 1. **Downloads** the authoritative `products.parquet` from HuggingFace
33
+ 2. **Validates** schema (column names, dtypes, row count)
34
+ 3. **Inspects** nulls, duplicates, UPC patterns, brand values, size fields
35
+ 4. **Documents** provenance of every column
36
+ 5. **Drops** 100% null columns (`size_per_unit`, `size_total`) with explicit documentation
37
+ 6. **Produces** clean output + validation + statistics
38
+
39
+ ---
40
+
41
+ ## Phase 1 Does NOT
42
+
43
+ - Modify source data values
44
+ - Normalize brands or titles
45
+ - Extract identity attributes
46
+ - Perform any grouping
47
+ - Use any external data sources
48
+
49
+ ---
50
+
51
+ ## Output Schema
52
+
53
+ The Phase 1 output (`phase1_output.parquet`) contains 14 columns:
54
+
55
+ | # | Column | Type | Description |
56
+ |---|--------|------|-------------|
57
+ | 1 | upc | str | Universal Product Code (1 null, 3,271 unique) |
58
+ | 2 | external_id | str | Voila retailer product ID (4,440 unique) |
59
+ | 3 | brand | str | Product brand (11 variants) |
60
+ | 4 | title | str | Raw product title from Voila |
61
+ | 5 | price | float64 | Price in CAD |
62
+ | 6 | price_currency | str | Always "CAD" |
63
+ | 7 | size | str | Raw size string (631 unique) |
64
+ | 8 | size_amount | float64 | Parsed numeric amount (307 nulls) |
65
+ | 9 | size_unit | str | Parsed unit: g, kg, ml, l, count (307 nulls) |
66
+ | 10 | size_qty | int64 | Quantity multiplier (usually 1) |
67
+ | 11 | size_unit_norm | str | Normalized unit (307 nulls) |
68
+ | 12 | image_url | str | Product image URL |
69
+ | 13 | source | str | Always "voila" |
70
+ | 14 | source_url | str | Product page URL |
71
+
72
+ ### Dropped Columns
73
+
74
+ | Column | Reason |
75
+ |--------|--------|
76
+ | size_per_unit | 100% null (4,440/4,440 values are NaN) |
77
+ | size_total | 100% null (4,440/4,440 values are NaN) |
78
+
79
+ ---
80
+
81
+ ## Validation Results
82
+
83
+ | Check | Result |
84
+ |-------|--------|
85
+ | Row count = 4,440 | PASS |
86
+ | Column count = 16 | PASS |
87
+ | Column names match expected | PASS |
88
+ | Full row duplicates = 0 | PASS |
89
+ | external_id duplicates = 0 | PASS |
90
+ | Unexpected 100% null columns | NONE |
91
+ | **Overall** | **PASS** |
92
+
93
+ ### Non-trivial Nulls
94
+
95
+ | Column | Null Count | Null % |
96
+ |--------|-----------|--------|
97
+ | upc | 1 | 0.02% |
98
+ | size_amount | 307 | 6.91% |
99
+ | size_unit | 307 | 6.91% |
100
+ | size_unit_norm | 307 | 6.91% |
101
+
102
+ ---
103
+
104
+ ## Key Findings
105
+
106
+ - **1168 reused UPCs**: Some barcodes are shared across product variants (same product, different sizes). This is expected.
107
+ - **1 null UPC**: `external_id` 24061EA ("Compliments Foil Containers with Lids 5-lb 2 Pack").
108
+ - **307 products (6.9%)** have null `size_amount`/`size_unit`/`size_unit_norm` — these have non-parseable or missing size strings.
109
+ - **11 brand variants** of "Compliments" and "Sensations" exist — normalization will happen in Phase 2.
110
+
111
+ ---
112
+
113
+ ## Files
114
+
115
+ ```
116
+ phase1/
117
+ README.md # This file
118
+ src/phase1.py # Production Python script
119
+ notebooks/phase1.ipynb # Interactive notebook (unexecuted)
120
+ outputs/
121
+ phase1_output.parquet # Clean validated output (4,440 rows, 14 cols)
122
+ phase1_provenance.json # Column provenance documentation
123
+ phase1_executed.ipynb # Executed notebook with all outputs
124
+ validation/
125
+ phase1_validation.json # Validation report (PASS/FAIL)
126
+ statistics/
127
+ phase1_statistics.json # Data statistics
128
+ ```
129
+
130
+ ---
131
+
132
+ ## How to Run
133
+
134
+ ```bash
135
+ # Using the Python script
136
+ cd phase1/
137
+ python3 src/phase1.py
138
+
139
+ # Using the notebook
140
+ jupyter notebook notebooks/phase1.ipynb
141
+ ```
142
+
143
+ ---
144
+
145
+ ## Handoff to Phase 2
146
+
147
+ Phase 2 must read:
148
+ - `phase1/outputs/phase1_output.parquet`
149
+
150
+ Phase 2 must NOT:
151
+ - Re-download from HuggingFace
152
+ - Read from any other source
153
+ - Skip Phase 1 validation