saraNour commited on
Commit
44114a5
Β·
verified Β·
1 Parent(s): 1cd137c

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +416 -72
README.md CHANGED
@@ -1,12 +1,35 @@
1
- # Reference DB system
2
 
3
- A reusable data pipeline for building retailer-level reference datasets for Open Food Facts Canada.
 
 
 
 
 
 
4
 
5
  ## Overview
6
 
7
- Reference DB transforms raw retailer product data into standardized, validated, classified, grouped, and enriched product datasets.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
8
 
9
- The pipeline is designed to support multiple retailers while keeping retailer-specific extraction logic isolated from the generic processing pipeline.
 
 
10
 
11
  ## Architecture
12
 
@@ -19,7 +42,7 @@ Ingestion
19
  ↓
20
  Phase 1 β€” Validation
21
  ↓
22
- Phase 2 β€” Identity
23
  ↓
24
  Phase 3 β€” Classification + Taxonomy + Grouping
25
  ↓
@@ -29,15 +52,138 @@ Phase 5 β€” Nutrition
29
  ↓
30
  Phase 6 β€” Scores & Enrichment
31
  ↓
 
 
32
  Retailer Reference Dataset
33
  ```
34
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
35
  ## Project Structure
36
 
37
  ```text
38
- reference-db/
39
  β”‚
40
  β”œβ”€β”€ contracts/
 
41
  β”‚ β”œβ”€β”€ ingestion/
42
  β”‚ β”œβ”€β”€ phase_1/
43
  β”‚ β”œβ”€β”€ phase_2/
@@ -46,103 +192,239 @@ reference-db/
46
  β”‚ β”œβ”€β”€ phase_5/
47
  β”‚ └── phase_6/
48
  β”‚
49
- β”œβ”€β”€ reference_data/
50
- β”‚ └── taxonomy/
 
 
 
 
 
 
 
 
 
 
51
  β”‚
52
  β”œβ”€β”€ src/
53
  β”‚ └── reference_db/
54
  β”‚ β”œβ”€β”€ adapters/
 
55
  β”‚ β”œβ”€β”€ ingestion/
56
  β”‚ β”œβ”€β”€ phase_1/
57
  β”‚ β”œβ”€β”€ phase_2/
58
  β”‚ β”œβ”€β”€ phase_3/
 
 
 
59
  β”‚ β”œβ”€β”€ phase_4/
60
  β”‚ β”œβ”€β”€ phase_5/
61
- β”‚ └── phase_6/
 
 
 
 
 
62
  β”‚
63
  β”œβ”€β”€ tests/
64
- β”œβ”€β”€ data/
65
- β”œβ”€β”€ README.md
66
  β”œβ”€β”€ pyproject.toml
 
67
  └── .gitignore
68
  ```
69
 
 
 
70
  ## Retailer Adapters
71
 
72
- The current implementation includes:
73
 
74
  * Compliments
 
 
 
 
75
 
76
- The adapter architecture is designed to support additional retailers such as Walmart, Costco, Metro, and VoilΓ  / Sobeys. New retailer-specific adapters can be added without changing the generic pipeline phases.
77
 
78
- ## Technology Stack
79
 
80
- * Python
81
- * dlt β€” data ingestion
82
- * DuckDB β€” analytical storage
83
- * pytest β€” testing
84
 
85
- ## Design Principles
86
 
87
- * Keep retailer-specific logic inside adapters.
88
- * Keep processing phases retailer-agnostic.
89
- * Preserve source identifiers and provenance.
90
- * Never invent missing source values.
91
- * Keep identity separate from variants.
92
- * Keep classification, taxonomy, and product grouping as separate concerns.
93
- * Preserve traceability from processed records back to source data.
94
- * Validate data at phase boundaries using executable data contracts.
95
- * Prefer simple implementations before introducing additional infrastructure.
96
 
97
- ## Pipeline Phases
98
 
99
- ### Phase 1 β€” Validation
 
 
 
 
 
 
100
 
101
- Validates the ingested product dataset against the ingestion contract and checks required fields, data types, identifiers, and duplicate records.
102
 
103
- ### Phase 2 β€” Identity
 
 
104
 
105
- Builds a standardized product identity representation and separates identity attributes from variant attributes.
106
 
107
- ### Phase 3 β€” Classification, Taxonomy & Grouping
 
 
 
 
 
 
 
 
 
 
108
 
109
- Classifies products, assigns the Reference DB taxonomy, and creates product groups based on product identity.
110
 
111
- ### Phase 4 β€” Variants
112
 
113
- Models variants within existing product groups.
114
 
115
- ### Phase 5 β€” Nutrition
116
 
117
- Standardizes and validates nutrition data and links it to products.
 
 
 
 
 
118
 
119
- ### Phase 6 β€” Scores & Enrichment
120
 
121
- Calculates available product scores and assigns enrichment references such as Nutri-Score and Agribalyse references where possible.
122
 
123
- ## Data Contracts
 
 
124
 
125
- Each pipeline stage has a versioned contract defining:
126
 
127
- * Expected input schema
128
- * Expected output schema
129
- * Required and optional fields
130
- * Processing responsibilities
131
- * Quality expectations
132
- * Primary identifiers
133
- * Lineage
134
 
135
- Contracts are stored under `contracts/`.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
136
 
137
  ## Data
138
 
139
- Raw and generated datasets are not committed to this repository by default.
 
 
140
 
141
- The `data/` directory is intended for local or generated pipeline data.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
142
 
143
  ## Development
144
 
145
- Create and activate the virtual environment:
146
 
147
  ```bash
148
  python -m venv .venv
@@ -155,36 +437,98 @@ Install the project in editable mode:
155
  pip install -e .
156
  ```
157
 
158
- Run tests:
159
 
160
  ```bash
161
- pytest
162
  ```
163
 
164
- ## Status
165
 
166
- The repository currently contains the project architecture, contracts, and implementation skeleton.
 
 
167
 
168
- Pipeline phases are being implemented incrementally and tested independently before being integrated into the complete workflow.
169
 
170
- ## Quality Gates & Review
 
 
171
 
172
- Each phase produces a quality result that determines whether its output can continue.
173
 
174
- Possible statuses are:
175
 
176
- - `PASS` β€” continue normally.
177
- - `WARNING` β€” continue and report the issue.
178
- - `AMBIGUOUS` β€” quarantine the record for review.
179
- - `FAIL` β€” stop processing the affected pipeline path.
180
 
181
- Ambiguity is detected by the phase responsible for the decision.
182
 
183
- For example:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
184
 
185
- - Phase 2 β†’ ambiguous identity or normalization.
186
- - Phase 3 β†’ ambiguous classification, taxonomy, or grouping.
187
- - Phase 4 β†’ ambiguous variant.
188
 
189
- Review is not a separate pipeline phase. After a decision is recorded, the affected record can be reprocessed from the phase where the ambiguity occurred.
190
 
 
 
1
+ # Reference DB System
2
 
3
+ A reusable data pipeline for building retailer-level reference datasets for **Open Food Facts Canada**.
4
+
5
+ Reference DB transforms retailer product data into standardized, validated, classified, grouped, and enriched product datasets while preserving source identifiers, provenance, and traceability.
6
+
7
+ The system is designed to support multiple Canadian retailers while keeping retailer-specific extraction and mapping logic isolated from the generic processing pipeline.
8
+
9
+ ---
10
 
11
  ## Overview
12
 
13
+ Retailer product data is often inconsistent across sources. Product names, identifiers, attributes, categories, nutrition information, and variants may use different structures and conventions.
14
+
15
+ Reference DB provides a structured pipeline that transforms this source data into a consistent retailer-level reference dataset.
16
+
17
+ The pipeline separates:
18
+
19
+ * Source-specific ingestion
20
+ * Data validation
21
+ * Product identity
22
+ * Classification and taxonomy
23
+ * Product grouping
24
+ * Variant modeling
25
+ * Nutrition processing
26
+ * Scores and enrichment
27
+ * Production outputs
28
+ * Quality validation and review
29
 
30
+ The resulting data can be used as a reference layer for Open Food Facts Canada workflows and downstream applications.
31
+
32
+ ---
33
 
34
  ## Architecture
35
 
 
42
  ↓
43
  Phase 1 β€” Validation
44
  ↓
45
+ Phase 2 β€” Identity & Normalization
46
  ↓
47
  Phase 3 β€” Classification + Taxonomy + Grouping
48
  ↓
 
52
  ↓
53
  Phase 6 β€” Scores & Enrichment
54
  ↓
55
+ Production Outputs
56
+ ↓
57
  Retailer Reference Dataset
58
  ```
59
 
60
+ The architecture is intentionally modular so that retailer-specific logic does not have to be duplicated across processing phases.
61
+
62
+ ---
63
+
64
+ ## Processing Phases
65
+
66
+ ### Phase 1 β€” Validation
67
+
68
+ Validates the ingested product dataset against the ingestion and validation contracts.
69
+
70
+ Checks include:
71
+
72
+ * Required fields
73
+ * Data types
74
+ * Product identifiers
75
+ * Duplicate records
76
+ * Basic structural consistency
77
+ * Source data integrity
78
+
79
+ ---
80
+
81
+ ### Phase 2 β€” Identity & Normalization
82
+
83
+ Builds a standardized representation of product identity.
84
+
85
+ This phase:
86
+
87
+ * Normalizes identity-related attributes
88
+ * Separates identity attributes from variant attributes
89
+ * Preserves source identifiers
90
+ * Generates deterministic identity information
91
+ * Provides the foundation for product grouping
92
+
93
+ Identity and variant information are deliberately kept separate so that different variants of the same product can remain within the same product group.
94
+
95
+ ---
96
+
97
+ ### Phase 3 β€” Classification, Taxonomy & Grouping
98
+
99
+ Phase 3 combines three related but distinct concepts:
100
+
101
+ **Classification**
102
+
103
+ Assigns product-level classifications based on product information and domain rules.
104
+
105
+ **Reference Taxonomy**
106
+
107
+ Maps products to the Reference DB taxonomy rather than relying entirely on retailer-specific categories.
108
+
109
+ **Product Grouping**
110
+
111
+ Groups records representing the same underlying product identity.
112
+
113
+ The grouping pipeline uses candidate generation, matching signals, identity information, and explicit decision logic.
114
+
115
+ Ambiguous cases can be isolated for review rather than being silently merged.
116
+
117
+ ---
118
+
119
+ ### Phase 4 β€” Variants
120
+
121
+ Models product variants within established product groups.
122
+
123
+ Examples of variant-level differences include:
124
+
125
+ * Size
126
+ * Quantity
127
+ * Flavor
128
+ * Format
129
+ * Packaging
130
+ * Other product-specific attributes
131
+
132
+ Variants remain linked to their product group while preserving their own identifiers and attributes.
133
+
134
+ ---
135
+
136
+ ### Phase 5 β€” Nutrition
137
+
138
+ Standardizes and validates available nutrition information and links nutrition data to products.
139
+
140
+ Missing nutrition information is treated as **missing data**, not as zero.
141
+
142
+ Nutrition processing also includes validation to prevent invalid or suspicious source values from silently becoming production data.
143
+
144
+ ---
145
+
146
+ ### Phase 6 β€” Scores & Enrichment
147
+
148
+ Calculates available product scores and enrichment information.
149
+
150
+ Depending on the available source data, this phase can include:
151
+
152
+ * Nutri-Score
153
+ * Agribalyse references
154
+ * Fruit and vegetable-related enrichment
155
+ * Score eligibility
156
+ * Other derived product attributes
157
+
158
+ Only products satisfying the required eligibility conditions receive the corresponding score or enrichment.
159
+
160
+ ---
161
+
162
+ ## Production Outputs
163
+
164
+ The pipeline produces production-oriented datasets rather than exposing processing artifacts as the final interface.
165
+
166
+ Current production outputs include:
167
+
168
+ ```text
169
+ production_metadata
170
+ production_groups
171
+ production_variants
172
+ production_nutrition
173
+ production_scores
174
+ ```
175
+
176
+ These outputs provide a stable representation of the processed Reference DB data while keeping intermediate processing artifacts separate.
177
+
178
+ ---
179
+
180
  ## Project Structure
181
 
182
  ```text
183
+ reference-db-system/
184
  β”‚
185
  β”œβ”€β”€ contracts/
186
+ β”‚ β”œβ”€β”€ adapters/
187
  β”‚ β”œβ”€β”€ ingestion/
188
  β”‚ β”œβ”€β”€ phase_1/
189
  β”‚ β”œβ”€β”€ phase_2/
 
192
  β”‚ β”œβ”€β”€ phase_5/
193
  β”‚ └── phase_6/
194
  β”‚
195
+ β”œβ”€β”€ data/
196
+ β”‚ └── README.md
197
+ β”‚
198
+ β”œβ”€β”€ scripts/
199
+ β”‚ β”œβ”€β”€ phase1.py
200
+ β”‚ β”œβ”€β”€ phase5.py
201
+ β”‚ β”œβ”€β”€ phase6.py
202
+ β”‚ β”œβ”€β”€ publish_phase3.py
203
+ β”‚ β”œβ”€β”€ regression_after.py
204
+ β”‚ β”œβ”€β”€ regression_check.py
205
+ β”‚ └── qa/
206
+ β”‚ └── end_to_end_qa.py
207
  β”‚
208
  β”œβ”€β”€ src/
209
  β”‚ └── reference_db/
210
  β”‚ β”œβ”€β”€ adapters/
211
+ β”‚ β”œβ”€β”€ classification/
212
  β”‚ β”œβ”€β”€ ingestion/
213
  β”‚ β”œβ”€β”€ phase_1/
214
  β”‚ β”œβ”€β”€ phase_2/
215
  β”‚ β”œβ”€β”€ phase_3/
216
+ β”‚ β”‚ β”œβ”€β”€ classification/
217
+ β”‚ β”‚ β”œβ”€β”€ grouping/
218
+ β”‚ β”‚ └── taxonomy/
219
  β”‚ β”œβ”€β”€ phase_4/
220
  β”‚ β”œβ”€β”€ phase_5/
221
+ β”‚ β”œβ”€β”€ phase_6/
222
+ β”‚ β”œβ”€β”€ production/
223
+ β”‚ β”œβ”€β”€ publishing/
224
+ β”‚ β”œβ”€β”€ quality/
225
+ β”‚ β”œβ”€β”€ review/
226
+ β”‚ └── taxonomy/
227
  β”‚
228
  β”œβ”€β”€ tests/
229
+ β”‚
 
230
  β”œβ”€β”€ pyproject.toml
231
+ β”œβ”€β”€ README.md
232
  └── .gitignore
233
  ```
234
 
235
+ ---
236
+
237
  ## Retailer Adapters
238
 
239
+ The repository contains retailer-specific adapters for:
240
 
241
  * Compliments
242
+ * Costco
243
+ * Metro
244
+ * VoilΓ 
245
+ * Walmart
246
 
247
+ The adapter architecture isolates retailer-specific source mapping and extraction logic from the generic Reference DB processing phases.
248
 
249
+ This makes it possible to add or modify a retailer without rewriting the core processing pipeline.
250
 
251
+ ---
 
 
 
252
 
253
+ ## Data Contracts
254
 
255
+ Each major pipeline stage has a versioned contract describing the expected behavior and data structure.
 
 
 
 
 
 
 
 
256
 
257
+ Contracts define aspects such as:
258
 
259
+ * Expected input schema
260
+ * Expected output schema
261
+ * Required and optional fields
262
+ * Primary identifiers
263
+ * Processing responsibilities
264
+ * Quality expectations
265
+ * Data lineage
266
 
267
+ Contracts are stored under:
268
 
269
+ ```text
270
+ contracts/
271
+ ```
272
 
273
+ The current contracts cover:
274
 
275
+ ```text
276
+ contracts/
277
+ β”œβ”€β”€ adapters/
278
+ β”œβ”€β”€ ingestion/
279
+ β”œβ”€β”€ phase_1/
280
+ β”œβ”€β”€ phase_2/
281
+ β”œβ”€β”€ phase_3/
282
+ β”œβ”€β”€ phase_4/
283
+ β”œβ”€β”€ phase_5/
284
+ └── phase_6/
285
+ ```
286
 
287
+ ---
288
 
289
+ ## Quality Gates & Review
290
 
291
+ Quality checks are performed at phase boundaries to prevent invalid or ambiguous data from silently progressing through the pipeline.
292
 
293
+ Possible quality states include:
294
 
295
+ | Status | Meaning |
296
+ | ----------- | ----------------------------------------------------------------------------- |
297
+ | `PASS` | The output satisfies the required checks. |
298
+ | `WARNING` | An issue was detected but processing can continue. |
299
+ | `AMBIGUOUS` | The decision cannot be made safely and the record can be isolated for review. |
300
+ | `FAIL` | The affected pipeline path should not continue. |
301
 
302
+ Ambiguity is detected by the phase responsible for the corresponding decision.
303
 
304
+ For example:
305
 
306
+ * Phase 2 β†’ ambiguous identity or normalization
307
+ * Phase 3 β†’ ambiguous classification, taxonomy, or grouping
308
+ * Phase 4 β†’ ambiguous variant decisions
309
 
310
+ Review is not treated as a separate processing phase. Once a decision is resolved, the affected record can be reprocessed from the relevant phase.
311
 
312
+ ---
313
+
314
+ ## Testing & Validation
315
+
316
+ The repository includes automated tests and an end-to-end QA workflow.
 
 
317
 
318
+ The current validation workflow includes:
319
+
320
+ ```bash
321
+ ruff check src scripts tests
322
+ python -m compileall -q src scripts tests
323
+ pytest -q
324
+ python scripts/qa/end_to_end_qa.py
325
+ ```
326
+
327
+ The end-to-end QA validates:
328
+
329
+ * Phase outputs
330
+ * External ID coverage
331
+ * Group integrity
332
+ * Variant integrity
333
+ * Nutrition and score relationships
334
+ * Production output consistency
335
+ * Orphan detection
336
+ * Basic nutrition quality
337
+ * Score eligibility
338
+
339
+ The current test suite contains **74 passing tests**, and the end-to-end QA workflow passes on the current implementation.
340
+
341
+ ---
342
+
343
+ ## Current QA Snapshot
344
+
345
+ The current end-to-end QA run processes:
346
+
347
+ | Output | Rows Γ— Columns |
348
+ | ---------------------- | -------------: |
349
+ | Phase 1 | 4,440 Γ— 14 |
350
+ | Phase 2 | 4,440 Γ— 34 |
351
+ | Phase 3 Groups | 4,440 Γ— 2 |
352
+ | Phase 3 Classification | 4,440 Γ— 12 |
353
+ | Phase 3 Taxonomy | 4,440 Γ— 7 |
354
+ | Phase 4 Variants | 4,320 Γ— 6 |
355
+ | Phase 4 Mapping | 4,440 Γ— 4 |
356
+ | Phase 5 Nutrition | 4,440 Γ— 77 |
357
+ | Phase 6 Scores | 4,440 Γ— 39 |
358
+ | Production Metadata | 4,440 Γ— 34 |
359
+ | Production Groups | 3,845 Γ— 2 |
360
+ | Production Variants | 4,320 Γ— 6 |
361
+ | Production Nutrition | 4,440 Γ— 77 |
362
+ | Production Scores | 4,440 Γ— 39 |
363
+
364
+ The QA checks currently report:
365
+
366
+ * External ID missing/extra records: **0 / 0**
367
+ * Null production `group_id`: **0**
368
+ * Production group mismatches: **0**
369
+ * Orphan variants: **0**
370
+ * Orphan nutrition records: **0**
371
+ * Orphan score records: **0**
372
+ * Production metadata/nutrition/score mismatches: **0**
373
+
374
+ These figures describe the current validated test dataset and are not claims about the size of all Reference DB source datasets.
375
+
376
+ ---
377
 
378
  ## Data
379
 
380
+ Raw source data and generated pipeline outputs are intentionally **not committed to this code repository by default**.
381
+
382
+ The `data/` directory is reserved for local or generated pipeline data and currently contains documentation only.
383
 
384
+ Reference datasets and source collections are maintained separately on Hugging Face.
385
+
386
+ ### ReferenceDB Datasets
387
+
388
+ The ReferenceDB dataset collection contains Canadian retailer, brand, and source datasets used by the project.
389
+
390
+ [ReferenceDB Hugging Face Collection](https://huggingface.co/collections/offCanada/referencedb?utm_source=chatgpt.com)
391
+
392
+ ---
393
+
394
+ ## Technology Stack
395
+
396
+ * **Python** β€” pipeline implementation
397
+ * **dlt** β€” data ingestion and loading
398
+ * **DuckDB** β€” local analytical processing and storage
399
+ * **pytest** β€” automated testing
400
+ * **Ruff** β€” linting and code quality checks
401
+
402
+ The implementation intentionally keeps the core processing logic lightweight and modular.
403
+
404
+ ---
405
+
406
+ ## Design Principles
407
+
408
+ The system follows several core design principles:
409
+
410
+ * Keep retailer-specific logic inside adapters.
411
+ * Keep processing phases retailer-agnostic.
412
+ * Preserve source identifiers and provenance.
413
+ * Never invent missing source values.
414
+ * Keep identity separate from variants.
415
+ * Keep classification, taxonomy, and product grouping as separate concerns.
416
+ * Preserve traceability from processed records back to source data.
417
+ * Validate data at phase boundaries.
418
+ * Use executable data contracts where appropriate.
419
+ * Isolate ambiguous decisions rather than silently forcing them.
420
+ * Keep processing artifacts separate from production outputs.
421
+ * Prefer simple implementations before introducing additional infrastructure.
422
+
423
+ ---
424
 
425
  ## Development
426
 
427
+ Create and activate a virtual environment:
428
 
429
  ```bash
430
  python -m venv .venv
 
437
  pip install -e .
438
  ```
439
 
440
+ Run the test suite:
441
 
442
  ```bash
443
+ pytest -q
444
  ```
445
 
446
+ Run linting:
447
 
448
+ ```bash
449
+ ruff check src scripts tests
450
+ ```
451
 
452
+ Run the end-to-end QA:
453
 
454
+ ```bash
455
+ python scripts/qa/end_to_end_qa.py
456
+ ```
457
 
458
+ ---
459
 
460
+ ## Repository Roles
461
 
462
+ The project uses different repositories and resources for different purposes.
 
 
 
463
 
464
+ ### GitHub β€” Source Code
465
 
466
+ The GitHub repository is the primary development repository containing the Reference DB implementation, contracts, tests, and pipeline scripts.
467
+
468
+ [Reference DB System on GitHub](https://github.com/offCanada/reference-db-system?utm_source=chatgpt.com)
469
+
470
+ ### Hugging Face β€” Code Snapshot
471
+
472
+ A Hugging Face copy of the code is maintained as a convenient snapshot for review and reproducibility.
473
+
474
+ [Reference DB System on Hugging Face](https://huggingface.co/offCanada/reference-db-system?utm_source=chatgpt.com)
475
+
476
+ ### Hugging Face β€” Datasets
477
+
478
+ Reference datasets are maintained separately in the ReferenceDB collection.
479
+
480
+ [ReferenceDB Collection](https://huggingface.co/collections/offCanada/referencedb?utm_source=chatgpt.com)
481
+
482
+ ### Notion β€” Architecture Documentation
483
+
484
+ The architectural decisions, processing model, data flow, and system design are documented separately.
485
+
486
+ [Reference DB Architecture Documentation](https://app.notion.com/p/Reference-DB-Architecture-23e0995da28a836c8dd7814651b06c0f?utm_source=chatgpt.com)
487
+
488
+ ---
489
+
490
+ ## Project Status
491
+
492
+ The core Reference DB processing pipeline is implemented and integrated across Phases 1–6.
493
+
494
+ The current repository includes:
495
+
496
+ * Retailer adapters
497
+ * Ingestion and validation
498
+ * Identity normalization
499
+ * Classification
500
+ * Reference taxonomy
501
+ * Product grouping
502
+ * Variant modeling
503
+ * Nutrition processing
504
+ * Scores and enrichment
505
+ * Production dataset construction
506
+ * Quality gates
507
+ * Review and quarantine components
508
+ * Data contracts
509
+ * Automated tests
510
+ * End-to-end QA
511
+ * Publishing utilities
512
+
513
+ The current implementation has been validated through automated tests and an end-to-end QA workflow.
514
+
515
+ The repository is intended to provide a reusable foundation for continuing Reference DB development and onboarding additional Canadian data sources.
516
+
517
+ ---
518
+
519
+ ## Related Resources
520
+
521
+ * **Source code:** [GitHub Repository](https://github.com/offCanada/reference-db-system?utm_source=chatgpt.com)
522
+ * **Code review snapshot:** [Hugging Face Code Repository](https://huggingface.co/offCanada/reference-db-system?utm_source=chatgpt.com)
523
+ * **Datasets:** [ReferenceDB Collection](https://huggingface.co/collections/offCanada/referencedb?utm_source=chatgpt.com)
524
+ * **Architecture:** [Reference DB Architecture](https://app.notion.com/p/Reference-DB-Architecture-23e0995da28a836c8dd7814651b06c0f?utm_source=chatgpt.com)
525
+
526
+ ---
527
+
528
+ ## GSoC 2026
529
 
530
+ This repository contains the implementation developed as part of the **Open Food Facts Canada Reference DB project during Google Summer of Code 2026**.
 
 
531
 
532
+ The repository should be read together with the architecture documentation and ReferenceDB datasets to understand the complete project.
533
 
534
+ The final project overview and work-product documentation provide the broader context, implementation summary, results, challenges, and project status.