Spaces:

alphamike
/

GraphAttributeLearning

Sleeping

Ashish Mehta

Add initial project structure and configuration files for Adjective-Aware Object Embedding

8e53dc6 8 months ago

9.58 kB

# Adjective-Aware Object Embedding

## 1) Goal and Scope

Build an adjective-aware representation pipeline for object images (starting with `chair`) that:

- predicts multi-label attributes (for example `broken`, `blue`, `wooden`, `plastic`)
- reshapes embedding space so items cluster by adjectives, not only by object identity
- supports both offline analysis (embedding quality) and online inference (attribute scoring)

Initial scope is single-object class (`chair`) to de-risk data and modeling, then generalize to multi-class.

---

## 2) Target Outcomes (Definition of Done)

The project is complete when all conditions below are met:

- **Model quality**
  - mAP improves over baseline by >= 8-12% relative.
  - Macro-F1 improves over baseline by >= 5% relative.
  - At least 4 key attributes (`broken`, `blue`, `wooden`, `plastic`) achieve per-class AP >= 0.70 on validation (adjustable by data quality).
- **Embedding quality**
  - UMAP/t-SNE shows tighter adjective clusters than baseline.
  - Retrieval@K for same-adjective neighbors is measurably improved.
- **Engineering quality**
  - Reproducible training with fixed seeds and config-driven runs.
  - End-to-end training + inference scripts run without manual edits.
  - Experiment logs and checkpoints are versioned and comparable.

---

## 3) Technical Strategy

### 3.1 Core Idea

Use a dual-encoder visual backbone:

- CLIP image features (semantic alignment)
- DINOv2 image features (visual structure)

Fuse both features and perform message passing over an object-attribute graph, where:

- object nodes use fused visual features
- attribute nodes use CLIP text embeddings (or learnable vectors as ablation)
- edges encode object-attribute membership (+ optional attribute-attribute priors)

### 3.2 Modeling Choice Rationale

- CLIP alone is strong semantically but can miss fine visual states.
- DINOv2 adds complementary visual detail.
- GNN introduces relational inductive bias, useful for correlated attributes (`plastic` with `material`, `broken` with `damage`).

### 3.3 Baselines (Mandatory)

Implement before full GNN rollout:

1. **Baseline A:** CLIP-only + MLP multi-label head
2. **Baseline B:** DINO-only + MLP multi-label head
3. **Baseline C:** CLIP+DINO fusion + MLP head (no graph)
4. **Proposed:** CLIP+DINO + GNN

All gains must be reported against Baseline C to isolate graph contribution.

---

## 4) Data Strategy

## 4.1 Dataset Selection (Phase-1 Priority)

Use this order:

1. **Visual Genome** (object + attributes, fastest path for graph supervision)
2. **MIT States / SUN Attributes** (state-centric signals, useful for adjective robustness)
3. **Open Images V7** (scale-up if labels fit target adjectives)

Avoid LAION in Phase-1 training due to noisy weak labels; use only for optional pretraining or pseudo-label mining.

## 4.2 Label Taxonomy

Normalize attributes into controlled groups:

- **Color:** blue, red, white, black, ...
- **Material:** wooden, plastic, metal, ...
- **Condition/State:** broken, clean, dirty, old, ...

Rules:

- lowercase, lemmatized labels
- synonym merge (`wood` -> `wooden`, `damaged` -> `broken` when valid)
- keep only attributes with minimum support threshold (for example >= 200 samples)

## 4.3 Data Splits

- train/val/test by image IDs (no leakage)
- stratified split by high-level attribute groups
- maintain long-tail report for rare attributes

## 4.4 Data Quality Gates

Before model training, enforce:

- duplicate image check
- invalid/empty label filtering
- per-attribute frequency report
- noisy label audit on random sample (manual spot check)

---

## 5) Graph Construction Plan

## 5.1 Graph Types

Build two graph variants for experiments:

- **Variant 1 (Bipartite):** object <-> attribute
- **Variant 2 (Enriched):** object <-> attribute + attribute <-> attribute priors

Start with Variant 1 for stability, then ablate Variant 2.

## 5.2 Node Features

- **Object node:** fused visual embedding (`[clip || dino]`)
- **Attribute node (default):** CLIP text embedding from prompt template:
  - `a {attribute} chair`
- **Attribute node (ablation):** trainable vectors

## 5.3 Edge Weights

Use weighted edges for confidence:

- labeled positive edge: weight = 1.0
- weak/pseudo edge (if used): weight in `[0.3, 0.8]`
- optional attribute prior edge from co-occurrence PMI or normalized co-frequency

---

## 6) Model Architecture

## 6.1 Feature Fusion

Start with concatenation:

- `z = concat(z_clip, z_dino)`

Then ablate:

- gated fusion (`z = g * z_clip + (1 - g) * z_dino`)
- projected fusion (linear layers to shared dimension before concat)

## 6.2 GNN Head

Recommended start:

- `GATConv(input_dim -> 512, heads=4)`
- `ReLU + Dropout(0.2)`
- `GATConv(512 -> 256, heads=2)`
- `Linear(256 -> num_attributes)` for object-node logits

Alternative for stability if needed: GraphSAGE.

## 6.3 Loss Design

Primary:

- weighted BCE with class-frequency weights

Secondary:

- contrastive / InfoNCE on object embeddings:
  - positives: same adjective group
  - negatives: conflicting adjective groups

Total:

- `L = L_bce + lambda_contrast * L_contrast`

Tune `lambda_contrast` in `{0.05, 0.1, 0.2}`.

---

## 7) Training Strategy

## 7.1 Two-Stage Optimization

Stage 1:

- freeze CLIP and DINO encoders
- train fusion + GNN head

Stage 2:

- unfreeze top N layers of encoders (N configurable)
- fine-tune with lower LR for encoders

This reduces instability and GPU cost early.

## 7.2 Hyperparameter Defaults

- optimizer: AdamW
- LR head: `1e-3`
- LR encoder: `1e-5` (stage 2)
- weight decay: `1e-4`
- batch size: maximize stable GPU usage (target 32+ effective via grad accumulation)
- epochs: 30-60 with early stopping on val mAP

## 7.3 Training Controls

- deterministic seed support
- mixed precision enabled
- gradient clipping (`1.0`)
- checkpoint by best val mAP
- scheduler: cosine decay with warmup

---

## 8) Evaluation Strategy

## 8.1 Core Metrics

- mAP (primary)
- Macro-F1 and Micro-F1
- per-attribute AP and F1
- calibration metrics (ECE / reliability bins) if serving probabilities

## 8.2 Embedding Metrics

- retrieval precision@K for same-adjective neighbors
- cluster compactness/separation (silhouette on adjective labels)
- UMAP/t-SNE visual audits (fixed random seed)

## 8.3 Error Analysis

Every run should include:

- top false positives and false negatives per attribute
- confusion among correlated materials/colors
- performance by attribute frequency bucket (head/mid/tail)

---

## 9) Ablation Matrix (Required)

Minimum experiment grid:

1. CLIP-only vs DINO-only vs fused
2. no-graph vs bipartite graph vs enriched graph
3. text-initialized attribute nodes vs trainable attribute nodes
4. BCE only vs BCE + contrastive
5. frozen encoders vs partial fine-tuning

Keep one variable changed per ablation for clean conclusions.

---

## 10) Execution Sequence (No Timeline)

### Phase A: Data and Baselines

- finalize attribute taxonomy and label normalization
- build data loader + split pipeline
- train Baseline A/B/C
- output: baseline report and dataset diagnostics

### Phase B: Graph Pipeline

- implement graph builder (Variant 1)
- train GNN model with frozen encoders
- evaluate against Baseline C
- output: first graph-vs-no-graph comparison

### Phase C: Optimization and Ablations

- run stage-2 fine-tuning
- run required ablations
- tune `lambda_contrast`, fusion, graph depth
- output: ablation table + selected final config

### Phase D: Robustness and Packaging

- run long-tail and error analysis
- package inference script, checkpoints, and reproducibility docs
- add minimal Gradio demo interface for inference
- generate final report with metrics and visualizations
- output: final model + execution docs + demo outputs

Gradio demo is an end-stage task and starts only after training, ablations, and final model selection are complete.

---

## 11) Repository Execution Structure

Suggested structure:

- `configs/` (dataset/model/train/eval yaml)
- `data/` (raw + processed metadata, ignore large binaries in git)
- `src/encoders/` (clip, dino wrappers)
- `src/graph/` (graph builder, priors, batching)
- `src/models/` (fusion, gnn heads, mlp baselines)
- `src/train/` (trainer, losses, schedulers)
- `src/eval/` (metrics, retrieval, visualization)
- `scripts/` (train, eval, infer)
- `reports/` (experiment tables, plots, analysis)

---

## 12) Risks and Mitigations

- **Risk:** label noise in attributes  
  **Mitigation:** filtering, confidence weights, manual audit sample.

- **Risk:** class imbalance hurts rare adjectives  
  **Mitigation:** weighted BCE, focal loss fallback, frequency-aware sampling.

- **Risk:** overfitting with small attribute subsets  
  **Mitigation:** stronger augmentation, early stopping, regularization.

- **Risk:** graph complexity without measurable gain  
  **Mitigation:** enforce baseline comparison and remove graph if gain is negligible.

---

## 13) Final Deliverables

- trained adjective-aware embedding model
- reproducible training and inference pipeline
- ablation and baseline comparison report
- embedding visualization and retrieval analysis
- concise model card (dataset, metrics, limits, known failure cases)
- minimal Gradio demo app for interactive attribute prediction

---

## 14) Immediate Next Actions

1. Lock Phase-1 dataset (`Visual Genome`) and attribute list.
2. Implement data normalization and split scripts.
3. Train Baseline C (`CLIP+DINO+MLP`) as reference.
4. Build bipartite graph constructor and run first GNN experiment.
5. Record metrics in a single comparison table from day one.

	# Adjective-Aware Object Embedding

	## 1) Goal and Scope

	Build an adjective-aware representation pipeline for object images (starting with `chair`) that:

	- predicts multi-label attributes (for example `broken`, `blue`, `wooden`, `plastic`)
	- reshapes embedding space so items cluster by adjectives, not only by object identity
	- supports both offline analysis (embedding quality) and online inference (attribute scoring)

	Initial scope is single-object class (`chair`) to de-risk data and modeling, then generalize to multi-class.

	---

	## 2) Target Outcomes (Definition of Done)

	The project is complete when all conditions below are met:

	- Model quality
	- mAP improves over baseline by >= 8-12% relative.
	- Macro-F1 improves over baseline by >= 5% relative.
	- At least 4 key attributes (`broken`, `blue`, `wooden`, `plastic`) achieve per-class AP >= 0.70 on validation (adjustable by data quality).
	- Embedding quality
	- UMAP/t-SNE shows tighter adjective clusters than baseline.
	- Retrieval@K for same-adjective neighbors is measurably improved.
	- Engineering quality
	- Reproducible training with fixed seeds and config-driven runs.
	- End-to-end training + inference scripts run without manual edits.
	- Experiment logs and checkpoints are versioned and comparable.

	---

	## 3) Technical Strategy

	### 3.1 Core Idea

	Use a dual-encoder visual backbone:

	- CLIP image features (semantic alignment)
	- DINOv2 image features (visual structure)

	Fuse both features and perform message passing over an object-attribute graph, where:

	- object nodes use fused visual features
	- attribute nodes use CLIP text embeddings (or learnable vectors as ablation)
	- edges encode object-attribute membership (+ optional attribute-attribute priors)

	### 3.2 Modeling Choice Rationale

	- CLIP alone is strong semantically but can miss fine visual states.
	- DINOv2 adds complementary visual detail.
	- GNN introduces relational inductive bias, useful for correlated attributes (`plastic` with `material`, `broken` with `damage`).

	### 3.3 Baselines (Mandatory)

	Implement before full GNN rollout:

	1. Baseline A: CLIP-only + MLP multi-label head
	2. Baseline B: DINO-only + MLP multi-label head
	3. Baseline C: CLIP+DINO fusion + MLP head (no graph)
	4. Proposed: CLIP+DINO + GNN

	All gains must be reported against Baseline C to isolate graph contribution.

	---

	## 4) Data Strategy

	## 4.1 Dataset Selection (Phase-1 Priority)

	Use this order:

	1. Visual Genome (object + attributes, fastest path for graph supervision)
	2. MIT States / SUN Attributes (state-centric signals, useful for adjective robustness)
	3. Open Images V7 (scale-up if labels fit target adjectives)

	Avoid LAION in Phase-1 training due to noisy weak labels; use only for optional pretraining or pseudo-label mining.

	## 4.2 Label Taxonomy

	Normalize attributes into controlled groups:

	- Color: blue, red, white, black, ...
	- Material: wooden, plastic, metal, ...
	- Condition/State: broken, clean, dirty, old, ...

	Rules:

	- lowercase, lemmatized labels
	- synonym merge (`wood` -> `wooden`, `damaged` -> `broken` when valid)
	- keep only attributes with minimum support threshold (for example >= 200 samples)

	## 4.3 Data Splits

	- train/val/test by image IDs (no leakage)
	- stratified split by high-level attribute groups
	- maintain long-tail report for rare attributes

	## 4.4 Data Quality Gates

	Before model training, enforce:

	- duplicate image check
	- invalid/empty label filtering
	- per-attribute frequency report
	- noisy label audit on random sample (manual spot check)

	---

	## 5) Graph Construction Plan

	## 5.1 Graph Types

	Build two graph variants for experiments:

	- Variant 1 (Bipartite): object <-> attribute
	- Variant 2 (Enriched): object <-> attribute + attribute <-> attribute priors

	Start with Variant 1 for stability, then ablate Variant 2.

	## 5.2 Node Features

	- Object node: fused visual embedding (`[clip \|\| dino]`)
	- Attribute node (default): CLIP text embedding from prompt template:
	- `a {attribute} chair`
	- Attribute node (ablation): trainable vectors

	## 5.3 Edge Weights

	Use weighted edges for confidence:

	- labeled positive edge: weight = 1.0
	- weak/pseudo edge (if used): weight in `[0.3, 0.8]`
	- optional attribute prior edge from co-occurrence PMI or normalized co-frequency

	---

	## 6) Model Architecture

	## 6.1 Feature Fusion

	Start with concatenation:

	- `z = concat(z_clip, z_dino)`

	Then ablate:

	- gated fusion (`z = g * z_clip + (1 - g) * z_dino`)
	- projected fusion (linear layers to shared dimension before concat)

	## 6.2 GNN Head

	Recommended start:

	- `GATConv(input_dim -> 512, heads=4)`
	- `ReLU + Dropout(0.2)`
	- `GATConv(512 -> 256, heads=2)`
	- `Linear(256 -> num_attributes)` for object-node logits

	Alternative for stability if needed: GraphSAGE.

	## 6.3 Loss Design

	Primary:

	- weighted BCE with class-frequency weights

	Secondary:

	- contrastive / InfoNCE on object embeddings:
	- positives: same adjective group
	- negatives: conflicting adjective groups

	Total:

	- `L = L_bce + lambda_contrast * L_contrast`

	Tune `lambda_contrast` in `{0.05, 0.1, 0.2}`.

	---

	## 7) Training Strategy

	## 7.1 Two-Stage Optimization

	Stage 1:

	- freeze CLIP and DINO encoders
	- train fusion + GNN head

	Stage 2:

	- unfreeze top N layers of encoders (N configurable)
	- fine-tune with lower LR for encoders

	This reduces instability and GPU cost early.

	## 7.2 Hyperparameter Defaults

	- optimizer: AdamW
	- LR head: `1e-3`
	- LR encoder: `1e-5` (stage 2)
	- weight decay: `1e-4`
	- batch size: maximize stable GPU usage (target 32+ effective via grad accumulation)
	- epochs: 30-60 with early stopping on val mAP

	## 7.3 Training Controls

	- deterministic seed support
	- mixed precision enabled
	- gradient clipping (`1.0`)
	- checkpoint by best val mAP
	- scheduler: cosine decay with warmup

	---

	## 8) Evaluation Strategy

	## 8.1 Core Metrics

	- mAP (primary)
	- Macro-F1 and Micro-F1
	- per-attribute AP and F1
	- calibration metrics (ECE / reliability bins) if serving probabilities

	## 8.2 Embedding Metrics

	- retrieval precision@K for same-adjective neighbors
	- cluster compactness/separation (silhouette on adjective labels)
	- UMAP/t-SNE visual audits (fixed random seed)

	## 8.3 Error Analysis

	Every run should include:

	- top false positives and false negatives per attribute
	- confusion among correlated materials/colors
	- performance by attribute frequency bucket (head/mid/tail)

	---

	## 9) Ablation Matrix (Required)

	Minimum experiment grid:

	1. CLIP-only vs DINO-only vs fused
	2. no-graph vs bipartite graph vs enriched graph
	3. text-initialized attribute nodes vs trainable attribute nodes
	4. BCE only vs BCE + contrastive
	5. frozen encoders vs partial fine-tuning

	Keep one variable changed per ablation for clean conclusions.

	---

	## 10) Execution Sequence (No Timeline)

	### Phase A: Data and Baselines

	- finalize attribute taxonomy and label normalization
	- build data loader + split pipeline
	- train Baseline A/B/C
	- output: baseline report and dataset diagnostics

	### Phase B: Graph Pipeline

	- implement graph builder (Variant 1)
	- train GNN model with frozen encoders
	- evaluate against Baseline C
	- output: first graph-vs-no-graph comparison

	### Phase C: Optimization and Ablations

	- run stage-2 fine-tuning
	- run required ablations
	- tune `lambda_contrast`, fusion, graph depth
	- output: ablation table + selected final config

	### Phase D: Robustness and Packaging

	- run long-tail and error analysis
	- package inference script, checkpoints, and reproducibility docs
	- add minimal Gradio demo interface for inference
	- generate final report with metrics and visualizations
	- output: final model + execution docs + demo outputs

	Gradio demo is an end-stage task and starts only after training, ablations, and final model selection are complete.

	---

	## 11) Repository Execution Structure

	Suggested structure:

	- `configs/` (dataset/model/train/eval yaml)
	- `data/` (raw + processed metadata, ignore large binaries in git)
	- `src/encoders/` (clip, dino wrappers)
	- `src/graph/` (graph builder, priors, batching)
	- `src/models/` (fusion, gnn heads, mlp baselines)
	- `src/train/` (trainer, losses, schedulers)
	- `src/eval/` (metrics, retrieval, visualization)
	- `scripts/` (train, eval, infer)
	- `reports/` (experiment tables, plots, analysis)

	---

	## 12) Risks and Mitigations

	- Risk: label noise in attributes
	Mitigation: filtering, confidence weights, manual audit sample.

	- Risk: class imbalance hurts rare adjectives
	Mitigation: weighted BCE, focal loss fallback, frequency-aware sampling.

	- Risk: overfitting with small attribute subsets
	Mitigation: stronger augmentation, early stopping, regularization.

	- Risk: graph complexity without measurable gain
	Mitigation: enforce baseline comparison and remove graph if gain is negligible.

	---

	## 13) Final Deliverables

	- trained adjective-aware embedding model
	- reproducible training and inference pipeline
	- ablation and baseline comparison report
	- embedding visualization and retrieval analysis
	- concise model card (dataset, metrics, limits, known failure cases)
	- minimal Gradio demo app for interactive attribute prediction

	---

	## 14) Immediate Next Actions

	1. Lock Phase-1 dataset (`Visual Genome`) and attribute list.
	2. Implement data normalization and split scripts.
	3. Train Baseline C (`CLIP+DINO+MLP`) as reference.
	4. Build bipartite graph constructor and run first GNN experiment.
	5. Record metrics in a single comparison table from day one.