first commit

plan/plan.md

@@ -0,0 +1,335 @@
+# Adjective-Aware Object Embedding using CLIP, DINOv2, and Graph Neural Networks
+
+## 1. Project Objective
+
+Build a system that learns **adjectives associated with objects** (e.g.,
+chairs) and reshapes the embedding space so objects cluster based on
+attributes like:
+
+-   broken
+-   blue
+-   wooden
+-   plastic
+-   dirty
+-   metal
+
+Encoders used: - CLIP Vision Encoder - DINOv2 Vision Encoder - Graph
+Neural Network layer for attribute reasoning
+
+------------------------------------------------------------------------
+
+## 2. System Architecture
+
+Image │ ├── CLIP Encoder ├── DINOv2 Encoder │ └── Feature Fusion │ ▼
+Object Embedding │ ▼ Graph Construction │ ▼ Graph Neural Network │ ▼
+Adjective-aware embedding │ ▼ Attribute prediction
+
+------------------------------------------------------------------------
+
+## 3. Encoders
+
+### CLIP Vision Encoder
+
+Common models: - ViT-B/32 - ViT-L/14
+
+Output embedding size: 512--768
+
+Benefits: - aligned with language - semantic understanding
+
+### DINOv2 Encoder
+
+Common models: - vit-small - vit-base - vit-large
+
+Output embedding size: 768--1024
+
+Benefits: - strong visual structure representation - self-supervised
+learning
+
+------------------------------------------------------------------------
+
+## 4. Feature Fusion
+
+clip_embedding = CLIP(image)\
+dino_embedding = DINO(image)
+
+combined_embedding = concat(clip_embedding, dino_embedding)
+
+Typical dimension:
+
+1536
+
+------------------------------------------------------------------------
+
+## 5. Graph Design
+
+Graph nodes:
+
+Object nodes\
+Attribute nodes
+
+Example:
+
+Objects: chair_1 chair_2 chair_3
+
+Attributes: blue broken wooden plastic dirty metal
+
+Edges:
+
+chair → blue\
+chair → broken\
+chair → plastic
+
+Optional attribute relations:
+
+plastic ↔ material\
+blue ↔ color\
+broken ↔ damage
+
+------------------------------------------------------------------------
+
+## 6. Node Features
+
+### Object nodes
+
+Feature = fused embedding
+
+dimension = 1536
+
+### Attribute nodes
+
+Option 1 (recommended)
+
+Use CLIP text embeddings:
+
+"a blue chair"\
+"a broken chair"\
+"a wooden chair"
+
+Option 2
+
+Learn attribute embeddings (trainable vectors)
+
+------------------------------------------------------------------------
+
+## 7. GNN Model
+
+Suggested architecture:
+
+GATConv(input_dim,512)\
+ReLU\
+GATConv(512,256)\
+Linear(256,1)
+
+Prediction is computed from attribute nodes.
+
+------------------------------------------------------------------------
+
+## 8. Training Objectives
+
+### Attribute Prediction
+
+Multi‑label classification
+
+Loss:
+
+Binary Cross Entropy
+
+Example label:
+
+chair_1: broken=1\
+blue=0\
+plastic=1
+
+### Contrastive Embedding Loss
+
+Encourage clustering by adjectives.
+
+Example:
+
+Positive pair: blue chair A blue chair B
+
+Negative pair: blue chair vs broken chair
+
+Loss options: - Triplet loss - Contrastive loss - InfoNCE
+
+------------------------------------------------------------------------
+
+## 9. Datasets
+
+### Visual Genome
+
+\~108k images with objects, attributes, and relationships.
+
+Example annotation:
+
+chair → wooden\
+chair → old
+
+Download: https://visualgenome.org
+
+------------------------------------------------------------------------
+
+### Open Images V7
+
+\~9M images with detection and attribute labels.
+
+Example:
+
+chair → plastic\
+chair → old
+
+Download: https://storage.googleapis.com/openimages/web/index.html
+
+------------------------------------------------------------------------
+
+### MIT States Dataset
+
+Object‑state dataset.
+
+Examples:
+
+broken chair\
+painted chair\
+clean chair
+
+------------------------------------------------------------------------
+
+### SUN Attribute Dataset
+
+Dataset containing \~102 visual attributes.
+
+Examples:
+
+rusty\
+shiny\
+old\
+colorful
+
+------------------------------------------------------------------------
+
+### COCO Dataset
+
+Object detection dataset (\~330k images).
+
+Attributes must be manually annotated.
+
+Download: https://cocodataset.org
+
+------------------------------------------------------------------------
+
+### LAION‑400M
+
+Large image‑text dataset with captions.
+
+Example captions:
+
+"a broken plastic chair"\
+"a blue chair"
+
+Attributes can be extracted using NLP.
+
+Download: https://laion.ai
+
+------------------------------------------------------------------------
+
+### ImageNet‑Attribute variants
+
+Researchers often annotate ImageNet objects with attributes such as:
+
+metal\
+wooden\
+colorful
+
+Useful for transfer learning.
+
+------------------------------------------------------------------------
+
+## 10. Data Preparation
+
+Dataset pipeline:
+
+images ↓ object detection (chair) ↓ extract attributes ↓ build
+object‑attribute graph
+
+Example dataset:
+
+dataset/ chair_001.jpg chair_002.jpg
+
+labels:
+
+chair_001.jpg → \[blue,plastic\]\
+chair_002.jpg → \[broken,wooden\]
+
+------------------------------------------------------------------------
+
+## 11. Training Pipeline
+
+Image ↓ CLIP encoder ↓ DINOv2 encoder ↓ feature fusion ↓ graph
+construction ↓ GNN forward pass ↓ attribute prediction ↓ loss
+computation ↓ backpropagation
+
+------------------------------------------------------------------------
+
+## 12. Inference Pipeline
+
+New Image ↓ CLIP + DINO ↓ embedding fusion ↓ graph node creation ↓ GNN
+inference ↓ attribute scores
+
+Example output:
+
+broken: 0.91\
+blue: 0.12\
+plastic: 0.87\
+wooden: 0.04
+
+------------------------------------------------------------------------
+
+## 13. Evaluation
+
+Metrics:
+
+-   mean average precision
+-   F1 score
+-   multi‑label accuracy
+
+Embedding analysis:
+
+-   t‑SNE
+-   UMAP
+
+Goal: observe clusters based on adjectives.
+
+------------------------------------------------------------------------
+
+## 14. Future Extensions
+
+Graph Transformer instead of GNN.
+
+Scene graph learning:
+
+chair → broken → leg
+
+LLM integration to generate descriptions:
+
+"This is a broken blue plastic chair."
+
+------------------------------------------------------------------------
+
+## 15. Deliverables
+
+-   adjective aware embedding model
+-   attribute prediction model
+-   graph dataset
+-   clustering visualizations
+
+------------------------------------------------------------------------
+
+## 16. Summary
+
+The system combines:
+
+CLIP semantic features\
+DINOv2 visual features\
+Graph neural network reasoning
+
+to produce an **embedding space structured by object adjectives**.