Blame for plan/plan_draft.md

# Adjective-Aware Object Embedding using CLIP, DINOv2, and Graph Neural Networks

## 1. Project Objective

Build a system that learns **adjectives associated with objects** (e.g.,
chairs) and reshapes the embedding space so objects cluster based on
attributes like:

-   broken
-   blue
-   wooden
-   plastic
-   dirty
-   metal

Encoders used: - CLIP Vision Encoder - DINOv2 Vision Encoder - Graph
Neural Network layer for attribute reasoning

------------------------------------------------------------------------

## 2. System Architecture

Image │ ├── CLIP Encoder ├── DINOv2 Encoder │ └── Feature Fusion │ ▼
Object Embedding │ ▼ Graph Construction │ ▼ Graph Neural Network │ ▼
Adjective-aware embedding │ ▼ Attribute prediction

------------------------------------------------------------------------

## 3. Encoders

### CLIP Vision Encoder

Common models: - ViT-B/32 - ViT-L/14

Output embedding size: 512--768

Benefits: - aligned with language - semantic understanding

### DINOv2 Encoder

Common models: - vit-small - vit-base - vit-large

Output embedding size: 768--1024

Benefits: - strong visual structure representation - self-supervised
learning

------------------------------------------------------------------------

## 4. Feature Fusion

clip_embedding = CLIP(image)\
dino_embedding = DINO(image)

combined_embedding = concat(clip_embedding, dino_embedding)

Typical dimension:

1536

------------------------------------------------------------------------

## 5. Graph Design

Graph nodes:

Object nodes\
Attribute nodes

Example:

Objects: chair_1 chair_2 chair_3

Attributes: blue broken wooden plastic dirty metal

Edges:

chair → blue\
chair → broken\
chair → plastic

Optional attribute relations:

plastic ↔ material\
blue ↔ color\
broken ↔ damage

------------------------------------------------------------------------

## 6. Node Features

### Object nodes

Feature = fused embedding

dimension = 1536

### Attribute nodes

Option 1 (recommended)

Use CLIP text embeddings:

"a blue chair"\
"a broken chair"\
"a wooden chair"

Option 2

Learn attribute embeddings (trainable vectors)

------------------------------------------------------------------------

## 7. GNN Model

Suggested architecture:

GATConv(input_dim,512)\
ReLU\
GATConv(512,256)\
Linear(256,1)

Prediction is computed from attribute nodes.

------------------------------------------------------------------------

## 8. Training Objectives

### Attribute Prediction

Multi‑label classification

Loss:

Binary Cross Entropy

Example label:

chair_1: broken=1\
blue=0\
plastic=1

### Contrastive Embedding Loss

Encourage clustering by adjectives.

Example:

Positive pair: blue chair A blue chair B

Negative pair: blue chair vs broken chair

Loss options: - Triplet loss - Contrastive loss - InfoNCE

------------------------------------------------------------------------

## 9. Datasets

### Visual Genome

\~108k images with objects, attributes, and relationships.

Example annotation:

chair → wooden\
chair → old

Download: https://visualgenome.org

------------------------------------------------------------------------

### Open Images V7

\~9M images with detection and attribute labels.

Example:

chair → plastic\
chair → old

Download: https://storage.googleapis.com/openimages/web/index.html

------------------------------------------------------------------------

### MIT States Dataset

Object‑state dataset.

Examples:

broken chair\
painted chair\
clean chair

------------------------------------------------------------------------

### SUN Attribute Dataset

Dataset containing \~102 visual attributes.

Examples:

rusty\
shiny\
old\
colorful

------------------------------------------------------------------------

### COCO Dataset

Object detection dataset (\~330k images).

Attributes must be manually annotated.

Download: https://cocodataset.org

------------------------------------------------------------------------

### LAION‑400M

Large image‑text dataset with captions.

Example captions:

"a broken plastic chair"\
"a blue chair"

Attributes can be extracted using NLP.

Download: https://laion.ai

------------------------------------------------------------------------

### ImageNet‑Attribute variants

Researchers often annotate ImageNet objects with attributes such as:

metal\
wooden\
colorful

Useful for transfer learning.

------------------------------------------------------------------------

## 10. Data Preparation

Dataset pipeline:

images ↓ object detection (chair) ↓ extract attributes ↓ build
object‑attribute graph

Example dataset:

dataset/ chair_001.jpg chair_002.jpg

labels:

chair_001.jpg → \[blue,plastic\]\
chair_002.jpg → \[broken,wooden\]

------------------------------------------------------------------------

## 11. Training Pipeline

Image ↓ CLIP encoder ↓ DINOv2 encoder ↓ feature fusion ↓ graph
construction ↓ GNN forward pass ↓ attribute prediction ↓ loss
computation ↓ backpropagation

------------------------------------------------------------------------

## 12. Inference Pipeline

New Image ↓ CLIP + DINO ↓ embedding fusion ↓ graph node creation ↓ GNN
inference ↓ attribute scores

Example output:

broken: 0.91\
blue: 0.12\
plastic: 0.87\
wooden: 0.04

------------------------------------------------------------------------

## 13. Evaluation

Metrics:

-   mean average precision
-   F1 score
-   multi‑label accuracy

Embedding analysis:

-   t‑SNE
-   UMAP

Goal: observe clusters based on adjectives.

------------------------------------------------------------------------

## 14. Future Extensions

Graph Transformer instead of GNN.

Scene graph learning:

chair → broken → leg

LLM integration to generate descriptions:

"This is a broken blue plastic chair."

------------------------------------------------------------------------

## 15. Deliverables

-   adjective aware embedding model
-   attribute prediction model
-   graph dataset
-   clustering visualizations

------------------------------------------------------------------------

## 16. Summary

The system combines:

CLIP semantic features\
DINOv2 visual features\
Graph neural network reasoning

to produce an **embedding space structured by object adjectives**.