product-embedding-e5-base / training /docs /product-function-training.md
beno-rohman's picture
Update product-function similarity checkpoint and concise model card
4aafc75 verified
|
Raw History Blame Contribute Delete
9.2 kB
# Product-function similarity training
## Objective
Learn graded relationships between products, across brands and models:
1. Different descriptions of the same identified product should be very close.
2. Different products with the same function/type should be close.
3. Related but different functional types should have intermediate similarity.
4. Unrelated functions, including misleading shared-name comparisons, should be
farther apart.
The primary objective is product-to-product similarity. Accessories are individual
functional types within the training set, not the only form of negative example.
## Data and automatic supervision
The input is the existing cleaned WDC v2 corpus at
`artifacts/prepared/offers.parquet`. No additional product corpus or user-supplied
annotations are required. The source predates many current product models, so
modern model names are a generalization question, not assumed training coverage.
`product_types.py` defines 84 functional-type evidence rules, organized into
broader families. Examples include consoles, laptops, desktop computers, phones,
headphones, cameras, running shoes, boots, shirts, chairs, desks, coffee makers,
kettles, musical instruments, fitness equipment, power tools, and pet food.
Rules assign **silver labels** from primary product-title evidence. They use
conservative role/compatibility exclusions and omit uncertain titles. Specific
accessory heads precede compatible main-device names. Secondary content after
words such as "with" is not treated as the main product head. They are not a
comprehensive product ontology and can mislabel ambiguous or noisy descriptions.
Mining processes 1,648,748 title candidates selected from the existing Parquet
corpus by a union of the functional patterns. Identity-level hash sampling bounds
each type/split pool at 1,800 products. Products with conflicting type assignments
within an identity are excluded. Existing train/validation/test identity splits
are retained. Exact shortened-title overlaps across splits are quarantined, and
shortened-title duplicates within each split are removed.
The audited pools support all 84 functional types. Actual distributions, rules,
conflicts, and leakage-removal counts are stored in
`artifacts/type-data/pool-manifest.json`.
## Pair construction
Each type contributes 140 training anchor selections. Smaller eligible classes
are sampled with replacement to avoid losing rare types. Same-type reference
products come from different identities, preferably different populated brands.
Other pairs sample broader-family and unrelated relationships. Character n-gram
TF-IDF mines lexically similar products assigned a different functional type.
Targets are similarity-regression training targets, not calibrated probabilities:
| Relationship | Target cosine similarity | Generated training pairs |
|---|---:|---:|
| Same identified product | 0.97 | 3,500 |
| Product and its automatically inferred type description | 0.90 | 11,760 |
| Different products, same functional type | 0.88 | 11,760 |
| Same broader family, different types | 0.40 | 11,620 |
| Different type, lexical hard negative | 0.15 | 11,760 |
| Unrelated broader families | 0.05 | 11,760 |
| **Total** | | **62,160** |
Type descriptions verbalize the inferred label, such as "game consoles" or
"running shoes". They are not fabricated product listings. Exact-identity pairs
reuse the existing GTIN-derived training data. All other product texts come from
the current WDC corpus.
For 40% of main-device anchor selections, the pipeline derives a shorter name by
removing an explicit brand prefix, capacity, and generic/color terms. This creates
short-name inputs from actual listings. Accessory role words are preserved;
"sticker Nintendo Switch" is never shortened to "Nintendo Switch" by that rule.
The console example strings used for the final demonstration are not inserted
as handcrafted training pairs.
## Training
Configuration: `configs/type-training.json`.
- Start from original `intfloat/multilingual-e5-base`, revision
`d128750597153bb5987e10b1c3493a34e5a4502a`.
- Keep 768-dimensional mean-pooled, normalized embeddings.
- Prefix both pair texts with `query: `; maximum length 96 tokens.
- Use Sentence Transformers `CosineSimilarityLoss`: MSE between predicted cosine
similarity and the graded silver target.
- Pairwise regression does not treat another same-type item in the batch as an
automatic negative.
- One epoch, batch size 8, gradient accumulation 1, FP16, fused AdamW,
learning rate `1e-5`, linear schedule, 8% warmup, seed 42.
- Evaluate 1,500 fixed sampled validation pairs and save every 1,000 steps.
Retain two checkpoints and load the checkpoint with lowest validation loss.
- Zero data-loader workers and two Torch CPU threads.
The final model is written to `artifacts/training/e5-product-type/model`. Its
training manifest records actual steps, runtime, peak allocated CUDA memory,
versions, data fingerprint, and selected checkpoint.
## Evaluation
Validation and test use different product identities from training. Their
relationships are also inferred automatically; these are silver-label tests,
not independently human-annotated ground truth. There are 12,788 validation pair
relations and 12,792 test pair relations, with retrieval tasks spanning 84 types.
Compare original E5, the earlier identity model, and the product-function model:
- Macro type-consistent nearest-neighbor accuracy.
- Macro same-type fraction among the five nearest candidates.
- Same-type versus lexical different-type ranking accuracy.
- A separate product-versus-product ranking slice excluding accessory types.
- Graded-pair Spearman correlation and MSE.
- Same-type/hard-negative cosine distributions and per-type results.
- The earlier exact-identity retrieval benchmark, to record the objective trade-off.
Separate demonstration probes cover consoles, phones, laptops, footwear, and
kitchen appliances. They contain ordinary short descriptions and current product
names; they are diagnostic examples, not a replacement for the larger benchmark.
Reports are stored under `artifacts/type-data/evaluation`. The model becomes the
visualizer default only after training, evaluation, and explicit objective checks
finish and write a completed `validated.json` marker.
## Inference and visualization
At inference, the web interface sends raw product descriptions to the selected
encoder with the query prefix. It does not run the labeling rules, look up a type,
prepend type labels, or position nodes using class membership. PCA/t-SNE is
computed from the actual resulting normalized embeddings. The inspector ranks
neighbors using full-dimensional cosine similarity.
The model selector retains all three checkpoints for comparison. Product-function
similarity uses the new checkpoint; product identity uses the earlier one; the
original E5 remains available as the baseline. The Hugging Face identity-model
repository is a separate artifact from this local training run.
## Commands
```powershell
.\.venv\Scripts\python.exe -m product_embedding.type_data --execute
.\.venv\Scripts\python.exe -m product_embedding.type_training --execute
```
To reuse mined pools and regenerate the audited graded pairs:
```powershell
.\.venv\Scripts\python.exe -m product_embedding.type_data --execute --reuse-pools
```
Do not regenerate training inputs while a training process is reading them. Use
fresh data/output paths for a different experiment. Run commands without
`--execute` to preview configurations.
## Limitations
Labels inherit rule errors, web extraction noise, uneven language coverage, and
the selected functional taxonomy. Relationships are deliberately coarser than
exact identity, and within-type capacity/variant distinctions can weaken. Short
ambiguous names and types outside the automatically labeled set may still be
poorly represented. Three-dimensional projections can discard useful distances.
Completion and improvements must be established from actual reports and raw
inference results, not from the training objective alone.
## Completed run
The configured epoch completed in 878.08 seconds including final save, with
7,770 steps and 62,160 pairs. The selected checkpoint at step 7,770 had sampled
validation MSE 0.016543. All final objective checks passed. The API/UI now exposes
the validated checkpoint as `product-type` and selects it by default.
On the 3,208-query / 3,216-candidate silver-label test, macro same-type top-1
improved from 86.49% (original E5) to 98.75%. Main-product versus main-product
ranking improved from 51.34% to 98.93%. Every console/console score in the
diagnostic example exceeded every console/sticker score. All five diagnostic
groups passed their specified same-function contrast ordering checks.
The original identity test exposes a substantial trade-off: new Recall@1 is
42.1% at 96 tokens, compared with the earlier identity checkpoint's 97.5% at 128
tokens. The original checkpoint remains available for that task. Full metrics,
limits, provenance, usage, and example matrices are in the
[product-function model card](product-function-model-card.md) and local reports.