Add initial project structure and configuration files for Adjective-Aware Object Embedding

- Created README.md detailing project objectives, structure, and execution strategy.
- Added dataset.yaml for dataset configuration, including attribute taxonomy and splits.
- Introduced eval.yaml for evaluation metrics and visualization settings.
- Established experiment.yaml for defining baseline and proposed runs.
- Configured model.yaml for encoder settings and GNN architecture.
- Developed train.yaml for training parameters and optimization strategies.
- Drafted plan_draft.md outlining project goals, architecture, and data preparation strategies.

README.md

@@ -0,0 +1,104 @@
+# Adjective-Aware Object Embedding
+
+This project builds adjective-aware object embeddings using:
+
+- CLIP image features
+- DINOv2 image features
+- Graph Neural Network (GNN) reasoning over object-attribute graphs
+
+Current scope starts with `chair` and multi-label attributes like `broken`, `blue`, `wooden`, and `plastic`.
+
+## Project Structure
+
+- `plan/plan.md` - detailed execution strategy and technical plan
+- `configs/dataset.yaml` - dataset, attributes, splits, normalization
+- `configs/model.yaml` - encoder, fusion, graph, and head settings
+- `configs/train.yaml` - optimization, staging, losses, checkpointing
+- `configs/eval.yaml` - metrics, visualization, error analysis
+- `configs/experiment.yaml` - baseline/main/ablation run definitions
+
+## Config Overview
+
+### `dataset.yaml`
+
+Defines:
+
+- dataset root/cache/processed paths
+- attribute taxonomy and synonym normalization
+- split strategy and filtering rules
+- basic augmentation
+
+### `model.yaml`
+
+Defines:
+
+- CLIP + DINO encoder settings
+- fusion type (`concat`, optional gated/projected ablations)
+- graph variant (`bipartite` / `enriched`)
+- attribute node initialization (`clip_text` / `trainable`)
+- GNN layers and baseline MLP head
+
+### `train.yaml`
+
+Defines:
+
+- two-stage training (frozen then partial fine-tuning)
+- optimizer, scheduler, AMP, grad clipping
+- weighted BCE + optional contrastive loss
+- early stopping and checkpoint monitoring
+
+### `eval.yaml`
+
+Defines:
+
+- classification metrics (mAP, macro/micro F1, per-attribute metrics)
+- embedding metrics (retrieval@K, silhouette)
+- UMAP/t-SNE options
+- error analysis outputs
+
+### `experiment.yaml`
+
+Defines:
+
+- baseline runs (A/B/C)
+- main proposed run (CLIP+DINO+GNN)
+- ablation runs (graph type, node init, loss, fine-tuning)
+
+## Execution Strategy (No Timeline)
+
+Follow this order:
+
+1. Data prep + label normalization
+2. Baselines (A/B/C)
+3. Graph model training
+4. Ablations and tuning
+5. Robustness checks + packaging
+6. Build a minimal Gradio demo interface (final step, after model is fully trained and selected)
+
+Use `plan/plan.md` as the source of truth for quality gates and deliverables.
+
+## How To Run
+
+This repo currently provides config skeletons. Connect them to your training entrypoint (for example `scripts/train.py`) and evaluation entrypoint (for example `scripts/eval.py`).
+
+Typical flow:
+
+1. Run baseline experiments from `configs/experiment.yaml`.
+2. Run the proposed GNN experiment.
+3. Run ablation experiments.
+4. Compare all runs in one metrics table.
+
+## Expected Outputs
+
+- model checkpoints
+- evaluation reports (mAP/F1/per-attribute)
+- embedding analysis artifacts (UMAP/retrieval results)
+- error analysis summaries
+- minimal Gradio demo for interactive predictions
+
+## Notes
+
+- Keep experiments reproducible with fixed seeds.
+- Track all runs with consistent config snapshots.
+- Measure gains against **Baseline C** to isolate graph contribution.
+

configs/dataset.yaml

@@ -0,0 +1,57 @@
+project:
+  name: adjective_aware_embedding
+  task: multi_label_attribute_prediction
+  primary_object_class: chair
+
+dataset:
+  name: visual_genome
+  root_dir: data/raw/visual_genome
+  processed_dir: data/processed/visual_genome
+  cache_dir: data/cache
+  image_size: 224
+  num_workers: 8
+  pin_memory: true
+
+splits:
+  train: 0.8
+  val: 0.1
+  test: 0.1
+  stratify_by_attribute_groups: true
+  split_seed: 42
+
+attributes:
+  groups:
+    color:
+      - blue
+      - red
+      - white
+      - black
+    material:
+      - wooden
+      - plastic
+      - metal
+    condition:
+      - broken
+      - clean
+      - dirty
+      - old
+  min_support: 200
+  lowercase: true
+  lemmatize: true
+  synonym_map:
+    wood: wooden
+    damaged: broken
+
+filtering:
+  drop_empty_labels: true
+  deduplicate_images: true
+  keep_only_primary_object: chair
+  max_samples_per_attribute: null
+
+augmentation:
+  enabled: true
+  horizontal_flip_prob: 0.5
+  color_jitter_prob: 0.2
+  random_resized_crop_scale:
+    - 0.8
+    - 1.0

configs/eval.yaml

@@ -0,0 +1,51 @@
+evaluation:
+  split: val
+  threshold:
+    strategy: fixed
+    value: 0.5
+
+metrics:
+  classification:
+    - map
+    - macro_f1
+    - micro_f1
+    - per_attribute_ap
+    - per_attribute_f1
+  calibration:
+    enabled: false
+    bins: 15
+    metrics:
+      - ece
+  embedding:
+    enabled: true
+    retrieval_k:
+      - 1
+      - 5
+      - 10
+    compute_silhouette: true
+
+visualization:
+  umap:
+    enabled: true
+    n_neighbors: 15
+    min_dist: 0.1
+    random_state: 42
+  tsne:
+    enabled: false
+    perplexity: 30
+    random_state: 42
+
+error_analysis:
+  enabled: true
+  top_k_false_positives: 20
+  top_k_false_negatives: 20
+  analyze_by_frequency_bucket: true
+  confusion_focus_groups:
+    - color
+    - material
+    - condition
+
+reporting:
+  save_predictions: true
+  save_embeddings: true
+  output_dir: reports/eval

configs/experiment.yaml

@@ -0,0 +1,56 @@
+experiment:
+  id: exp_clip_dino_gnn_v1
+  description: CLIP+DINO fusion with bipartite GNN and BCE+InfoNCE.
+  tags:
+    - baseline_c_plus_gnn
+    - adjective_embedding
+    - visual_genome
+
+defaults:
+  dataset: configs/dataset.yaml
+  model: configs/model.yaml
+  train: configs/train.yaml
+  eval: configs/eval.yaml
+
+baseline_runs:
+  - name: baseline_a_clip_only_mlp
+    overrides:
+      encoders.clip.enabled: true
+      encoders.dino.enabled: false
+      graph.variant: none
+      gnn.type: none
+  - name: baseline_b_dino_only_mlp
+    overrides:
+      encoders.clip.enabled: false
+      encoders.dino.enabled: true
+      graph.variant: none
+      gnn.type: none
+  - name: baseline_c_clip_dino_mlp
+    overrides:
+      encoders.clip.enabled: true
+      encoders.dino.enabled: true
+      graph.variant: none
+      gnn.type: none
+
+main_run:
+  name: proposed_clip_dino_gnn
+  overrides:
+    graph.variant: bipartite
+    gnn.type: gat
+    attribute_nodes.init_type: clip_text
+
+ablation_runs:
+  - name: graph_enriched_with_attribute_priors
+    overrides:
+      graph.variant: enriched
+      graph.include_attribute_priors: true
+  - name: trainable_attribute_nodes
+    overrides:
+      attribute_nodes.init_type: trainable
+  - name: bce_only
+    overrides:
+      loss.contrastive.enabled: false
+  - name: partial_encoder_finetune
+    overrides:
+      training.stage2.enabled: true
+      training.stage2.unfreeze_top_n_layers: 4

configs/model.yaml

@@ -0,0 +1,64 @@
+model:
+  name: clip_dino_gnn
+  num_attributes: 12
+  primary_object_class: chair
+
+encoders:
+  clip:
+    enabled: true
+    model_name: ViT-B-32
+    pretrained: true
+    output_dim: 512
+    freeze_stage1: true
+  dino:
+    enabled: true
+    model_name: vit_base_patch14_dinov2
+    pretrained: true
+    output_dim: 768
+    freeze_stage1: true
+
+fusion:
+  type: concat
+  output_dim: 1280
+  projected_fusion:
+    enabled: false
+    hidden_dim: 512
+  gated_fusion:
+    enabled: false
+    hidden_dim: 512
+
+graph:
+  variant: bipartite
+  include_attribute_priors: false
+  edge_weighting:
+    labeled_positive: 1.0
+    weak_min: 0.3
+    weak_max: 0.8
+
+attribute_nodes:
+  init_type: clip_text
+  prompt_template: "a {attribute} chair"
+  trainable_dim: 256
+
+gnn:
+  type: gat
+  layers:
+    - in_dim: 1280
+      out_dim: 512
+      heads: 4
+      dropout: 0.2
+    - in_dim: 512
+      out_dim: 256
+      heads: 2
+      dropout: 0.2
+  classifier:
+    in_dim: 256
+    out_dim: 12
+
+baseline_heads:
+  mlp:
+    enabled: true
+    hidden_dims:
+      - 512
+      - 256
+    dropout: 0.2

configs/train.yaml

@@ -0,0 +1,64 @@
+run:
+  seed: 42
+  deterministic: true
+  experiment_name: clip_dino_gnn_bipartite
+  output_dir: outputs
+  save_every_n_epochs: 1
+
+training:
+  stage1:
+    enabled: true
+    epochs: 20
+    freeze_encoders: true
+    lr_head: 0.001
+    lr_encoder: 0.0
+  stage2:
+    enabled: true
+    epochs: 20
+    freeze_encoders: false
+    unfreeze_top_n_layers: 2
+    lr_head: 0.0005
+    lr_encoder: 0.00001
+
+optimization:
+  optimizer: adamw
+  weight_decay: 0.0001
+  betas:
+    - 0.9
+    - 0.999
+  grad_clip_norm: 1.0
+  gradient_accumulation_steps: 1
+  mixed_precision: true
+
+scheduler:
+  type: cosine_with_warmup
+  warmup_ratio: 0.1
+  min_lr: 0.0000001
+
+batching:
+  batch_size: 32
+  eval_batch_size: 64
+
+loss:
+  bce:
+    enabled: true
+    pos_weight_mode: inverse_frequency
+  contrastive:
+    enabled: true
+    type: infonce
+    temperature: 0.07
+    lambda: 0.1
+    positive_sampling: same_adjective_group
+    negative_sampling: conflicting_groups
+
+early_stopping:
+  enabled: true
+  monitor: val_map
+  mode: max
+  patience: 8
+
+checkpoint:
+  enabled: true
+  monitor: val_map
+  mode: max
+  save_best_only: true

plan/plan.md

@@ -1,335 +1,340 @@
-# Adjective-Aware Object Embedding using CLIP, DINOv2, and Graph Neural Networks
+# Adjective-Aware Object Embedding
 
-## 1. Project Objective
+## 1) Goal and Scope
 
-Build a system that learns **adjectives associated with objects** (e.g.,
-chairs) and reshapes the embedding space so objects cluster based on
-attributes like:
+Build an adjective-aware representation pipeline for object images (starting with `chair`) that:
 
--   broken
--   blue
--   wooden
--   plastic
--   dirty
--   metal
+- predicts multi-label attributes (for example `broken`, `blue`, `wooden`, `plastic`)
+- reshapes embedding space so items cluster by adjectives, not only by object identity
+- supports both offline analysis (embedding quality) and online inference (attribute scoring)
 
-Encoders used: - CLIP Vision Encoder - DINOv2 Vision Encoder - Graph
-Neural Network layer for attribute reasoning
+Initial scope is single-object class (`chair`) to de-risk data and modeling, then generalize to multi-class.
 
-------------------------------------------------------------------------
+---
 
-## 2. System Architecture
+## 2) Target Outcomes (Definition of Done)
 
-Image │ ├── CLIP Encoder ├── DINOv2 Encoder │ └── Feature Fusion │ ▼
-Object Embedding │ ▼ Graph Construction │ ▼ Graph Neural Network │ ▼
-Adjective-aware embedding │ ▼ Attribute prediction
+The project is complete when all conditions below are met:
 
-------------------------------------------------------------------------
+- **Model quality**
+  - mAP improves over baseline by >= 8-12% relative.
+  - Macro-F1 improves over baseline by >= 5% relative.
+  - At least 4 key attributes (`broken`, `blue`, `wooden`, `plastic`) achieve per-class AP >= 0.70 on validation (adjustable by data quality).
+- **Embedding quality**
+  - UMAP/t-SNE shows tighter adjective clusters than baseline.
+  - Retrieval@K for same-adjective neighbors is measurably improved.
+- **Engineering quality**
+  - Reproducible training with fixed seeds and config-driven runs.
+  - End-to-end training + inference scripts run without manual edits.
+  - Experiment logs and checkpoints are versioned and comparable.
 
-## 3. Encoders
+---
 
-### CLIP Vision Encoder
+## 3) Technical Strategy
 
-Common models: - ViT-B/32 - ViT-L/14
+### 3.1 Core Idea
 
-Output embedding size: 512--768
+Use a dual-encoder visual backbone:
 
-Benefits: - aligned with language - semantic understanding
+- CLIP image features (semantic alignment)
+- DINOv2 image features (visual structure)
 
-### DINOv2 Encoder
+Fuse both features and perform message passing over an object-attribute graph, where:
 
-Common models: - vit-small - vit-base - vit-large
+- object nodes use fused visual features
+- attribute nodes use CLIP text embeddings (or learnable vectors as ablation)
+- edges encode object-attribute membership (+ optional attribute-attribute priors)
 
-Output embedding size: 768--1024
+### 3.2 Modeling Choice Rationale
 
-Benefits: - strong visual structure representation - self-supervised
-learning
+- CLIP alone is strong semantically but can miss fine visual states.
+- DINOv2 adds complementary visual detail.
+- GNN introduces relational inductive bias, useful for correlated attributes (`plastic` with `material`, `broken` with `damage`).
 
-------------------------------------------------------------------------
+### 3.3 Baselines (Mandatory)
 
-## 4. Feature Fusion
+Implement before full GNN rollout:
 
-clip_embedding = CLIP(image)\
-dino_embedding = DINO(image)
+1. **Baseline A:** CLIP-only + MLP multi-label head
+2. **Baseline B:** DINO-only + MLP multi-label head
+3. **Baseline C:** CLIP+DINO fusion + MLP head (no graph)
+4. **Proposed:** CLIP+DINO + GNN
 
-combined_embedding = concat(clip_embedding, dino_embedding)
+All gains must be reported against Baseline C to isolate graph contribution.
 
-Typical dimension:
+---
 
-1536
+## 4) Data Strategy
 
-------------------------------------------------------------------------
+## 4.1 Dataset Selection (Phase-1 Priority)
 
-## 5. Graph Design
+Use this order:
 
-Graph nodes:
+1. **Visual Genome** (object + attributes, fastest path for graph supervision)
+2. **MIT States / SUN Attributes** (state-centric signals, useful for adjective robustness)
+3. **Open Images V7** (scale-up if labels fit target adjectives)
 
-Object nodes\
-Attribute nodes
+Avoid LAION in Phase-1 training due to noisy weak labels; use only for optional pretraining or pseudo-label mining.
 
-Example:
+## 4.2 Label Taxonomy
 
-Objects: chair_1 chair_2 chair_3
+Normalize attributes into controlled groups:
 
-Attributes: blue broken wooden plastic dirty metal
+- **Color:** blue, red, white, black, ...
+- **Material:** wooden, plastic, metal, ...
+- **Condition/State:** broken, clean, dirty, old, ...
 
-Edges:
+Rules:
 
-chair → blue\
-chair → broken\
-chair → plastic
+- lowercase, lemmatized labels
+- synonym merge (`wood` -> `wooden`, `damaged` -> `broken` when valid)
+- keep only attributes with minimum support threshold (for example >= 200 samples)
 
-Optional attribute relations:
+## 4.3 Data Splits
 
-plastic ↔ material\
-blue ↔ color\
-broken ↔ damage
+- train/val/test by image IDs (no leakage)
+- stratified split by high-level attribute groups
+- maintain long-tail report for rare attributes
 
-------------------------------------------------------------------------
+## 4.4 Data Quality Gates
 
-## 6. Node Features
+Before model training, enforce:
 
-### Object nodes
+- duplicate image check
+- invalid/empty label filtering
+- per-attribute frequency report
+- noisy label audit on random sample (manual spot check)
 
-Feature = fused embedding
+---
 
-dimension = 1536
+## 5) Graph Construction Plan
 
-### Attribute nodes
+## 5.1 Graph Types
 
-Option 1 (recommended)
+Build two graph variants for experiments:
 
-Use CLIP text embeddings:
+- **Variant 1 (Bipartite):** object <-> attribute
+- **Variant 2 (Enriched):** object <-> attribute + attribute <-> attribute priors
 
-"a blue chair"\
-"a broken chair"\
-"a wooden chair"
+Start with Variant 1 for stability, then ablate Variant 2.
 
-Option 2
+## 5.2 Node Features
 
-Learn attribute embeddings (trainable vectors)
+- **Object node:** fused visual embedding (`[clip || dino]`)
+- **Attribute node (default):** CLIP text embedding from prompt template:
+  - `a {attribute} chair`
+- **Attribute node (ablation):** trainable vectors
 
-------------------------------------------------------------------------
+## 5.3 Edge Weights
 
-## 7. GNN Model
+Use weighted edges for confidence:
 
-Suggested architecture:
+- labeled positive edge: weight = 1.0
+- weak/pseudo edge (if used): weight in `[0.3, 0.8]`
+- optional attribute prior edge from co-occurrence PMI or normalized co-frequency
 
-GATConv(input_dim,512)\
-ReLU\
-GATConv(512,256)\
-Linear(256,1)
+---
 
-Prediction is computed from attribute nodes.
+## 6) Model Architecture
 
-------------------------------------------------------------------------
+## 6.1 Feature Fusion
 
-## 8. Training Objectives
+Start with concatenation:
 
-### Attribute Prediction
+- `z = concat(z_clip, z_dino)`
 
-Multi‑label classification
+Then ablate:
 
-Loss:
+- gated fusion (`z = g * z_clip + (1 - g) * z_dino`)
+- projected fusion (linear layers to shared dimension before concat)
 
-Binary Cross Entropy
+## 6.2 GNN Head
 
-Example label:
+Recommended start:
 
-chair_1: broken=1\
-blue=0\
-plastic=1
+- `GATConv(input_dim -> 512, heads=4)`
+- `ReLU + Dropout(0.2)`
+- `GATConv(512 -> 256, heads=2)`
+- `Linear(256 -> num_attributes)` for object-node logits
 
-### Contrastive Embedding Loss
+Alternative for stability if needed: GraphSAGE.
 
-Encourage clustering by adjectives.
+## 6.3 Loss Design
 
-Example:
+Primary:
 
-Positive pair: blue chair A blue chair B
+- weighted BCE with class-frequency weights
 
-Negative pair: blue chair vs broken chair
+Secondary:
 
-Loss options: - Triplet loss - Contrastive loss - InfoNCE
+- contrastive / InfoNCE on object embeddings:
+  - positives: same adjective group
+  - negatives: conflicting adjective groups
 
-------------------------------------------------------------------------
+Total:
 
-## 9. Datasets
+- `L = L_bce + lambda_contrast * L_contrast`
 
-### Visual Genome
+Tune `lambda_contrast` in `{0.05, 0.1, 0.2}`.
 
-\~108k images with objects, attributes, and relationships.
+---
 
-Example annotation:
+## 7) Training Strategy
 
-chair → wooden\
-chair → old
+## 7.1 Two-Stage Optimization
 
-Download: https://visualgenome.org
+Stage 1:
 
-------------------------------------------------------------------------
+- freeze CLIP and DINO encoders
+- train fusion + GNN head
 
-### Open Images V7
+Stage 2:
 
-\~9M images with detection and attribute labels.
+- unfreeze top N layers of encoders (N configurable)
+- fine-tune with lower LR for encoders
 
-Example:
+This reduces instability and GPU cost early.
 
-chair → plastic\
-chair → old
+## 7.2 Hyperparameter Defaults
 
-Download: https://storage.googleapis.com/openimages/web/index.html
+- optimizer: AdamW
+- LR head: `1e-3`
+- LR encoder: `1e-5` (stage 2)
+- weight decay: `1e-4`
+- batch size: maximize stable GPU usage (target 32+ effective via grad accumulation)
+- epochs: 30-60 with early stopping on val mAP
 
-------------------------------------------------------------------------
+## 7.3 Training Controls
 
-### MIT States Dataset
+- deterministic seed support
+- mixed precision enabled
+- gradient clipping (`1.0`)
+- checkpoint by best val mAP
+- scheduler: cosine decay with warmup
 
-Object‑state dataset.
+---
 
-Examples:
+## 8) Evaluation Strategy
 
-broken chair\
-painted chair\
-clean chair
+## 8.1 Core Metrics
 
-------------------------------------------------------------------------
+- mAP (primary)
+- Macro-F1 and Micro-F1
+- per-attribute AP and F1
+- calibration metrics (ECE / reliability bins) if serving probabilities
 
-### SUN Attribute Dataset
+## 8.2 Embedding Metrics
 
-Dataset containing \~102 visual attributes.
+- retrieval precision@K for same-adjective neighbors
+- cluster compactness/separation (silhouette on adjective labels)
+- UMAP/t-SNE visual audits (fixed random seed)
 
-Examples:
+## 8.3 Error Analysis
 
-rusty\
-shiny\
-old\
-colorful
+Every run should include:
 
-------------------------------------------------------------------------
+- top false positives and false negatives per attribute
+- confusion among correlated materials/colors
+- performance by attribute frequency bucket (head/mid/tail)
 
-### COCO Dataset
+---
 
-Object detection dataset (\~330k images).
+## 9) Ablation Matrix (Required)
 
-Attributes must be manually annotated.
+Minimum experiment grid:
 
-Download: https://cocodataset.org
+1. CLIP-only vs DINO-only vs fused
+2. no-graph vs bipartite graph vs enriched graph
+3. text-initialized attribute nodes vs trainable attribute nodes
+4. BCE only vs BCE + contrastive
+5. frozen encoders vs partial fine-tuning
 
-------------------------------------------------------------------------
+Keep one variable changed per ablation for clean conclusions.
 
-### LAION‑400M
+---
 
-Large image‑text dataset with captions.
+## 10) Execution Sequence (No Timeline)
 
-Example captions:
+### Phase A: Data and Baselines
 
-"a broken plastic chair"\
-"a blue chair"
+- finalize attribute taxonomy and label normalization
+- build data loader + split pipeline
+- train Baseline A/B/C
+- output: baseline report and dataset diagnostics
 
-Attributes can be extracted using NLP.
+### Phase B: Graph Pipeline
 
-Download: https://laion.ai
+- implement graph builder (Variant 1)
+- train GNN model with frozen encoders
+- evaluate against Baseline C
+- output: first graph-vs-no-graph comparison
 
-------------------------------------------------------------------------
+### Phase C: Optimization and Ablations
 
-### ImageNet‑Attribute variants
+- run stage-2 fine-tuning
+- run required ablations
+- tune `lambda_contrast`, fusion, graph depth
+- output: ablation table + selected final config
 
-Researchers often annotate ImageNet objects with attributes such as:
+### Phase D: Robustness and Packaging
 
-metal\
-wooden\
-colorful
+- run long-tail and error analysis
+- package inference script, checkpoints, and reproducibility docs
+- add minimal Gradio demo interface for inference
+- generate final report with metrics and visualizations
+- output: final model + execution docs + demo outputs
 
-Useful for transfer learning.
+Gradio demo is an end-stage task and starts only after training, ablations, and final model selection are complete.
 
-------------------------------------------------------------------------
+---
 
-## 10. Data Preparation
+## 11) Repository Execution Structure
 
-Dataset pipeline:
+Suggested structure:
 
-images ↓ object detection (chair) ↓ extract attributes ↓ build
-object‑attribute graph
+- `configs/` (dataset/model/train/eval yaml)
+- `data/` (raw + processed metadata, ignore large binaries in git)
+- `src/encoders/` (clip, dino wrappers)
+- `src/graph/` (graph builder, priors, batching)
+- `src/models/` (fusion, gnn heads, mlp baselines)
+- `src/train/` (trainer, losses, schedulers)
+- `src/eval/` (metrics, retrieval, visualization)
+- `scripts/` (train, eval, infer)
+- `reports/` (experiment tables, plots, analysis)
 
-Example dataset:
+---
 
-dataset/ chair_001.jpg chair_002.jpg
+## 12) Risks and Mitigations
 
-labels:
+- **Risk:** label noise in attributes  
+  **Mitigation:** filtering, confidence weights, manual audit sample.
 
-chair_001.jpg → \[blue,plastic\]\
-chair_002.jpg → \[broken,wooden\]
+- **Risk:** class imbalance hurts rare adjectives  
+  **Mitigation:** weighted BCE, focal loss fallback, frequency-aware sampling.
 
-------------------------------------------------------------------------
+- **Risk:** overfitting with small attribute subsets  
+  **Mitigation:** stronger augmentation, early stopping, regularization.
 
-## 11. Training Pipeline
+- **Risk:** graph complexity without measurable gain  
+  **Mitigation:** enforce baseline comparison and remove graph if gain is negligible.
 
-Image ↓ CLIP encoder ↓ DINOv2 encoder ↓ feature fusion ↓ graph
-construction ↓ GNN forward pass ↓ attribute prediction ↓ loss
-computation ↓ backpropagation
+---
 
-------------------------------------------------------------------------
+## 13) Final Deliverables
 
-## 12. Inference Pipeline
+- trained adjective-aware embedding model
+- reproducible training and inference pipeline
+- ablation and baseline comparison report
+- embedding visualization and retrieval analysis
+- concise model card (dataset, metrics, limits, known failure cases)
+- minimal Gradio demo app for interactive attribute prediction
 
-New Image ↓ CLIP + DINO ↓ embedding fusion ↓ graph node creation ↓ GNN
-inference ↓ attribute scores
+---
 
-Example output:
+## 14) Immediate Next Actions
 
-broken: 0.91\
-blue: 0.12\
-plastic: 0.87\
-wooden: 0.04
-
-------------------------------------------------------------------------
-
-## 13. Evaluation
-
-Metrics:
-
--   mean average precision
--   F1 score
--   multi‑label accuracy
-
-Embedding analysis:
-
--   t‑SNE
--   UMAP
-
-Goal: observe clusters based on adjectives.
-
-------------------------------------------------------------------------
-
-## 14. Future Extensions
-
-Graph Transformer instead of GNN.
-
-Scene graph learning:
-
-chair → broken → leg
-
-LLM integration to generate descriptions:
-
-"This is a broken blue plastic chair."
-
-------------------------------------------------------------------------
-
-## 15. Deliverables
-
--   adjective aware embedding model
--   attribute prediction model
--   graph dataset
--   clustering visualizations
-
-------------------------------------------------------------------------
-
-## 16. Summary
-
-The system combines:
-
-CLIP semantic features\
-DINOv2 visual features\
-Graph neural network reasoning
-
-to produce an **embedding space structured by object adjectives**.
+1. Lock Phase-1 dataset (`Visual Genome`) and attribute list.
+2. Implement data normalization and split scripts.
+3. Train Baseline C (`CLIP+DINO+MLP`) as reference.
+4. Build bipartite graph constructor and run first GNN experiment.
+5. Record metrics in a single comparison table from day one.

plan/plan_draft.md

@@ -0,0 +1,335 @@
+# Adjective-Aware Object Embedding using CLIP, DINOv2, and Graph Neural Networks
+
+## 1. Project Objective
+
+Build a system that learns **adjectives associated with objects** (e.g.,
+chairs) and reshapes the embedding space so objects cluster based on
+attributes like:
+
+-   broken
+-   blue
+-   wooden
+-   plastic
+-   dirty
+-   metal
+
+Encoders used: - CLIP Vision Encoder - DINOv2 Vision Encoder - Graph
+Neural Network layer for attribute reasoning
+
+------------------------------------------------------------------------
+
+## 2. System Architecture
+
+Image │ ├── CLIP Encoder ├── DINOv2 Encoder │ └── Feature Fusion │ ▼
+Object Embedding │ ▼ Graph Construction │ ▼ Graph Neural Network │ ▼
+Adjective-aware embedding │ ▼ Attribute prediction
+
+------------------------------------------------------------------------
+
+## 3. Encoders
+
+### CLIP Vision Encoder
+
+Common models: - ViT-B/32 - ViT-L/14
+
+Output embedding size: 512--768
+
+Benefits: - aligned with language - semantic understanding
+
+### DINOv2 Encoder
+
+Common models: - vit-small - vit-base - vit-large
+
+Output embedding size: 768--1024
+
+Benefits: - strong visual structure representation - self-supervised
+learning
+
+------------------------------------------------------------------------
+
+## 4. Feature Fusion
+
+clip_embedding = CLIP(image)\
+dino_embedding = DINO(image)
+
+combined_embedding = concat(clip_embedding, dino_embedding)
+
+Typical dimension:
+
+1536
+
+------------------------------------------------------------------------
+
+## 5. Graph Design
+
+Graph nodes:
+
+Object nodes\
+Attribute nodes
+
+Example:
+
+Objects: chair_1 chair_2 chair_3
+
+Attributes: blue broken wooden plastic dirty metal
+
+Edges:
+
+chair → blue\
+chair → broken\
+chair → plastic
+
+Optional attribute relations:
+
+plastic ↔ material\
+blue ↔ color\
+broken ↔ damage
+
+------------------------------------------------------------------------
+
+## 6. Node Features
+
+### Object nodes
+
+Feature = fused embedding
+
+dimension = 1536
+
+### Attribute nodes
+
+Option 1 (recommended)
+
+Use CLIP text embeddings:
+
+"a blue chair"\
+"a broken chair"\
+"a wooden chair"
+
+Option 2
+
+Learn attribute embeddings (trainable vectors)
+
+------------------------------------------------------------------------
+
+## 7. GNN Model
+
+Suggested architecture:
+
+GATConv(input_dim,512)\
+ReLU\
+GATConv(512,256)\
+Linear(256,1)
+
+Prediction is computed from attribute nodes.
+
+------------------------------------------------------------------------
+
+## 8. Training Objectives
+
+### Attribute Prediction
+
+Multi‑label classification
+
+Loss:
+
+Binary Cross Entropy
+
+Example label:
+
+chair_1: broken=1\
+blue=0\
+plastic=1
+
+### Contrastive Embedding Loss
+
+Encourage clustering by adjectives.
+
+Example:
+
+Positive pair: blue chair A blue chair B
+
+Negative pair: blue chair vs broken chair
+
+Loss options: - Triplet loss - Contrastive loss - InfoNCE
+
+------------------------------------------------------------------------
+
+## 9. Datasets
+
+### Visual Genome
+
+\~108k images with objects, attributes, and relationships.
+
+Example annotation:
+
+chair → wooden\
+chair → old
+
+Download: https://visualgenome.org
+
+------------------------------------------------------------------------
+
+### Open Images V7
+
+\~9M images with detection and attribute labels.
+
+Example:
+
+chair → plastic\
+chair → old
+
+Download: https://storage.googleapis.com/openimages/web/index.html
+
+------------------------------------------------------------------------
+
+### MIT States Dataset
+
+Object‑state dataset.
+
+Examples:
+
+broken chair\
+painted chair\
+clean chair
+
+------------------------------------------------------------------------
+
+### SUN Attribute Dataset
+
+Dataset containing \~102 visual attributes.
+
+Examples:
+
+rusty\
+shiny\
+old\
+colorful
+
+------------------------------------------------------------------------
+
+### COCO Dataset
+
+Object detection dataset (\~330k images).
+
+Attributes must be manually annotated.
+
+Download: https://cocodataset.org
+
+------------------------------------------------------------------------
+
+### LAION‑400M
+
+Large image‑text dataset with captions.
+
+Example captions:
+
+"a broken plastic chair"\
+"a blue chair"
+
+Attributes can be extracted using NLP.
+
+Download: https://laion.ai
+
+------------------------------------------------------------------------
+
+### ImageNet‑Attribute variants
+
+Researchers often annotate ImageNet objects with attributes such as:
+
+metal\
+wooden\
+colorful
+
+Useful for transfer learning.
+
+------------------------------------------------------------------------
+
+## 10. Data Preparation
+
+Dataset pipeline:
+
+images ↓ object detection (chair) ↓ extract attributes ↓ build
+object‑attribute graph
+
+Example dataset:
+
+dataset/ chair_001.jpg chair_002.jpg
+
+labels:
+
+chair_001.jpg → \[blue,plastic\]\
+chair_002.jpg → \[broken,wooden\]
+
+------------------------------------------------------------------------
+
+## 11. Training Pipeline
+
+Image ↓ CLIP encoder ↓ DINOv2 encoder ↓ feature fusion ↓ graph
+construction ↓ GNN forward pass ↓ attribute prediction ↓ loss
+computation ↓ backpropagation
+
+------------------------------------------------------------------------
+
+## 12. Inference Pipeline
+
+New Image ↓ CLIP + DINO ↓ embedding fusion ↓ graph node creation ↓ GNN
+inference ↓ attribute scores
+
+Example output:
+
+broken: 0.91\
+blue: 0.12\
+plastic: 0.87\
+wooden: 0.04
+
+------------------------------------------------------------------------
+
+## 13. Evaluation
+
+Metrics:
+
+-   mean average precision
+-   F1 score
+-   multi‑label accuracy
+
+Embedding analysis:
+
+-   t‑SNE
+-   UMAP
+
+Goal: observe clusters based on adjectives.
+
+------------------------------------------------------------------------
+
+## 14. Future Extensions
+
+Graph Transformer instead of GNN.
+
+Scene graph learning:
+
+chair → broken → leg
+
+LLM integration to generate descriptions:
+
+"This is a broken blue plastic chair."
+
+------------------------------------------------------------------------
+
+## 15. Deliverables
+
+-   adjective aware embedding model
+-   attribute prediction model
+-   graph dataset
+-   clustering visualizations
+
+------------------------------------------------------------------------
+
+## 16. Summary
+
+The system combines:
+
+CLIP semantic features\
+DINOv2 visual features\
+Graph neural network reasoning
+
+to produce an **embedding space structured by object adjectives**.