Hugging Face's logo

Spaces:
alphamike
/
GraphAttributeLearning
Sleeping

App Files Community
GraphAttributeLearning / plan /plan.md
Ashish Mehta
Add initial project structure and configuration files for Adjective-Aware Object Embedding
8e53dc6
preview code
|
Raw History Blame Contribute Delete
9.58 kB

A newer version of the Gradio SDK is available: 6.31.0

Upgrade

Adjective-Aware Object Embedding

1) Goal and Scope

Build an adjective-aware representation pipeline for object images (starting with chair) that:

  • predicts multi-label attributes (for example broken, blue, wooden, plastic)
  • reshapes embedding space so items cluster by adjectives, not only by object identity
  • supports both offline analysis (embedding quality) and online inference (attribute scoring)

Initial scope is single-object class (chair) to de-risk data and modeling, then generalize to multi-class.

2) Target Outcomes (Definition of Done)

The project is complete when all conditions below are met:

  • Model quality
    • mAP improves over baseline by >= 8-12% relative.
    • Macro-F1 improves over baseline by >= 5% relative.
    • At least 4 key attributes (broken, blue, wooden, plastic) achieve per-class AP >= 0.70 on validation (adjustable by data quality).
  • Embedding quality
    • UMAP/t-SNE shows tighter adjective clusters than baseline.
    • Retrieval@K for same-adjective neighbors is measurably improved.
  • Engineering quality
    • Reproducible training with fixed seeds and config-driven runs.
    • End-to-end training + inference scripts run without manual edits.
    • Experiment logs and checkpoints are versioned and comparable.

3) Technical Strategy

3.1 Core Idea

Use a dual-encoder visual backbone:

  • CLIP image features (semantic alignment)
  • DINOv2 image features (visual structure)

Fuse both features and perform message passing over an object-attribute graph, where:

  • object nodes use fused visual features
  • attribute nodes use CLIP text embeddings (or learnable vectors as ablation)
  • edges encode object-attribute membership (+ optional attribute-attribute priors)

3.2 Modeling Choice Rationale

  • CLIP alone is strong semantically but can miss fine visual states.
  • DINOv2 adds complementary visual detail.
  • GNN introduces relational inductive bias, useful for correlated attributes (plastic with material, broken with damage).

3.3 Baselines (Mandatory)

Implement before full GNN rollout:

  1. Baseline A: CLIP-only + MLP multi-label head
  2. Baseline B: DINO-only + MLP multi-label head
  3. Baseline C: CLIP+DINO fusion + MLP head (no graph)
  4. Proposed: CLIP+DINO + GNN

All gains must be reported against Baseline C to isolate graph contribution.

4) Data Strategy

4.1 Dataset Selection (Phase-1 Priority)

Use this order:

  1. Visual Genome (object + attributes, fastest path for graph supervision)
  2. MIT States / SUN Attributes (state-centric signals, useful for adjective robustness)
  3. Open Images V7 (scale-up if labels fit target adjectives)

Avoid LAION in Phase-1 training due to noisy weak labels; use only for optional pretraining or pseudo-label mining.

4.2 Label Taxonomy

Normalize attributes into controlled groups:

  • Color: blue, red, white, black, ...
  • Material: wooden, plastic, metal, ...
  • Condition/State: broken, clean, dirty, old, ...

Rules:

  • lowercase, lemmatized labels
  • synonym merge (wood -> wooden, damaged -> broken when valid)
  • keep only attributes with minimum support threshold (for example >= 200 samples)

4.3 Data Splits

  • train/val/test by image IDs (no leakage)
  • stratified split by high-level attribute groups
  • maintain long-tail report for rare attributes

4.4 Data Quality Gates

Before model training, enforce:

  • duplicate image check
  • invalid/empty label filtering
  • per-attribute frequency report
  • noisy label audit on random sample (manual spot check)

5) Graph Construction Plan

5.1 Graph Types

Build two graph variants for experiments:

  • Variant 1 (Bipartite): object <-> attribute
  • Variant 2 (Enriched): object <-> attribute + attribute <-> attribute priors

Start with Variant 1 for stability, then ablate Variant 2.

5.2 Node Features

  • Object node: fused visual embedding ([clip || dino])
  • Attribute node (default): CLIP text embedding from prompt template:
    • a {attribute} chair
  • Attribute node (ablation): trainable vectors

5.3 Edge Weights

Use weighted edges for confidence:

  • labeled positive edge: weight = 1.0
  • weak/pseudo edge (if used): weight in [0.3, 0.8]
  • optional attribute prior edge from co-occurrence PMI or normalized co-frequency

6) Model Architecture

6.1 Feature Fusion

Start with concatenation:

  • z = concat(z_clip, z_dino)

Then ablate:

  • gated fusion (z = g * z_clip + (1 - g) * z_dino)
  • projected fusion (linear layers to shared dimension before concat)

6.2 GNN Head

Recommended start:

  • GATConv(input_dim -> 512, heads=4)
  • ReLU + Dropout(0.2)
  • GATConv(512 -> 256, heads=2)
  • Linear(256 -> num_attributes) for object-node logits

Alternative for stability if needed: GraphSAGE.

6.3 Loss Design

Primary:

  • weighted BCE with class-frequency weights

Secondary:

  • contrastive / InfoNCE on object embeddings:
    • positives: same adjective group
    • negatives: conflicting adjective groups

Total:

  • L = L_bce + lambda_contrast * L_contrast

Tune lambda_contrast in {0.05, 0.1, 0.2}.

7) Training Strategy

7.1 Two-Stage Optimization

Stage 1:

  • freeze CLIP and DINO encoders
  • train fusion + GNN head

Stage 2:

  • unfreeze top N layers of encoders (N configurable)
  • fine-tune with lower LR for encoders

This reduces instability and GPU cost early.

7.2 Hyperparameter Defaults

  • optimizer: AdamW
  • LR head: 1e-3
  • LR encoder: 1e-5 (stage 2)
  • weight decay: 1e-4
  • batch size: maximize stable GPU usage (target 32+ effective via grad accumulation)
  • epochs: 30-60 with early stopping on val mAP

7.3 Training Controls

  • deterministic seed support
  • mixed precision enabled
  • gradient clipping (1.0)
  • checkpoint by best val mAP
  • scheduler: cosine decay with warmup

8) Evaluation Strategy

8.1 Core Metrics

  • mAP (primary)
  • Macro-F1 and Micro-F1
  • per-attribute AP and F1
  • calibration metrics (ECE / reliability bins) if serving probabilities

8.2 Embedding Metrics

  • retrieval precision@K for same-adjective neighbors
  • cluster compactness/separation (silhouette on adjective labels)
  • UMAP/t-SNE visual audits (fixed random seed)

8.3 Error Analysis

Every run should include:

  • top false positives and false negatives per attribute
  • confusion among correlated materials/colors
  • performance by attribute frequency bucket (head/mid/tail)

9) Ablation Matrix (Required)

Minimum experiment grid:

  1. CLIP-only vs DINO-only vs fused
  2. no-graph vs bipartite graph vs enriched graph
  3. text-initialized attribute nodes vs trainable attribute nodes
  4. BCE only vs BCE + contrastive
  5. frozen encoders vs partial fine-tuning

Keep one variable changed per ablation for clean conclusions.

10) Execution Sequence (No Timeline)

Phase A: Data and Baselines

  • finalize attribute taxonomy and label normalization
  • build data loader + split pipeline
  • train Baseline A/B/C
  • output: baseline report and dataset diagnostics

Phase B: Graph Pipeline

  • implement graph builder (Variant 1)
  • train GNN model with frozen encoders
  • evaluate against Baseline C
  • output: first graph-vs-no-graph comparison

Phase C: Optimization and Ablations

  • run stage-2 fine-tuning
  • run required ablations
  • tune lambda_contrast, fusion, graph depth
  • output: ablation table + selected final config

Phase D: Robustness and Packaging

  • run long-tail and error analysis
  • package inference script, checkpoints, and reproducibility docs
  • add minimal Gradio demo interface for inference
  • generate final report with metrics and visualizations
  • output: final model + execution docs + demo outputs

Gradio demo is an end-stage task and starts only after training, ablations, and final model selection are complete.

11) Repository Execution Structure

Suggested structure:

  • configs/ (dataset/model/train/eval yaml)
  • data/ (raw + processed metadata, ignore large binaries in git)
  • src/encoders/ (clip, dino wrappers)
  • src/graph/ (graph builder, priors, batching)
  • src/models/ (fusion, gnn heads, mlp baselines)
  • src/train/ (trainer, losses, schedulers)
  • src/eval/ (metrics, retrieval, visualization)
  • scripts/ (train, eval, infer)
  • reports/ (experiment tables, plots, analysis)

12) Risks and Mitigations

  • Risk: label noise in attributes
    Mitigation: filtering, confidence weights, manual audit sample.

  • Risk: class imbalance hurts rare adjectives
    Mitigation: weighted BCE, focal loss fallback, frequency-aware sampling.

  • Risk: overfitting with small attribute subsets
    Mitigation: stronger augmentation, early stopping, regularization.

  • Risk: graph complexity without measurable gain
    Mitigation: enforce baseline comparison and remove graph if gain is negligible.

13) Final Deliverables

  • trained adjective-aware embedding model
  • reproducible training and inference pipeline
  • ablation and baseline comparison report
  • embedding visualization and retrieval analysis
  • concise model card (dataset, metrics, limits, known failure cases)
  • minimal Gradio demo app for interactive attribute prediction

14) Immediate Next Actions

  1. Lock Phase-1 dataset (Visual Genome) and attribute list.
  2. Implement data normalization and split scripts.
  3. Train Baseline C (CLIP+DINO+MLP) as reference.
  4. Build bipartite graph constructor and run first GNN experiment.
  5. Record metrics in a single comparison table from day one.