Spaces:
Sleeping
Sleeping
File size: 5,438 Bytes
244b7a6 8e53dc6 4687353 8e53dc6 4687353 419fc11 8e53dc6 419fc11 8e53dc6 419fc11 8e53dc6 cca0e97 8e53dc6 4687353 8e53dc6 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 | ---
title: GraphAttributeLearning
emoji: 🪑
colorFrom: gray
colorTo: red
sdk: gradio
sdk_version: "6.9.0"
python_version: "3.10"
app_file: app.py
pinned: false
---
# Adjective-Aware Object Embedding
This project builds adjective-aware object embeddings using:
- CLIP image features
- DINOv2 image features
- Graph Neural Network (GNN) reasoning over object-attribute graphs
Current scope starts with `chair` and multi-label attributes like `broken`, `blue`, `wooden`, and `plastic`.
## Project Structure
- `plan/plan.md` - detailed execution strategy and technical plan
- `configs/dataset.yaml` - dataset, attributes, splits, normalization
- `configs/model.yaml` - encoder, fusion, graph, and head settings
- `configs/train.yaml` - optimization, staging, losses, checkpointing
- `configs/eval.yaml` - metrics, visualization, error analysis
- `configs/experiment.yaml` - baseline/main/ablation run definitions
- `scripts/data/` - data download, extraction, processing, split pipeline
- `src/data/` - reusable normalization and IO utilities for data pipeline
## Config Overview
### `dataset.yaml`
Defines:
- dataset root/cache/processed paths
- attribute taxonomy and synonym normalization
- split strategy and filtering rules
- basic augmentation
### `model.yaml`
Defines:
- CLIP + DINO encoder settings
- fusion type (`concat`, optional gated/projected ablations)
- graph variant (`bipartite` / `enriched`)
- attribute node initialization (`clip_text` / `trainable`)
- GNN layers and baseline MLP head
### `train.yaml`
Defines:
- two-stage training (frozen then partial fine-tuning)
- optimizer, scheduler, AMP, grad clipping
- weighted BCE + optional contrastive loss
- early stopping and checkpoint monitoring
### `eval.yaml`
Defines:
- classification metrics (mAP, macro/micro F1, per-attribute metrics)
- embedding metrics (retrieval@K, silhouette)
- UMAP/t-SNE options
- error analysis outputs
### `experiment.yaml`
Defines:
- baseline runs (A/B/C)
- main proposed run (CLIP+DINO+GNN)
- ablation runs (graph type, node init, loss, fine-tuning)
## Execution Strategy (No Timeline)
Follow this order:
1. Data prep + label normalization
2. Baselines (A/B/C)
3. Graph model training
4. Ablations and tuning
5. Robustness checks + packaging
6. Build a minimal Gradio demo interface (final step, after model is fully trained and selected)
Use `plan/plan.md` as the source of truth for quality gates and deliverables.
## How To Run
### 1) Run Chair-Only Data Pipeline (Visual Genome)
Default full pipeline:
`python scripts/data/run_data_pipeline.py --config configs/dataset.yaml`
Run individual stages:
- `python scripts/data/run_data_pipeline.py --download-only --config configs/dataset.yaml`
- `python scripts/data/run_data_pipeline.py --extract-only --config configs/dataset.yaml`
- `python scripts/data/run_data_pipeline.py --process-only --config configs/dataset.yaml`
- `python scripts/data/run_data_pipeline.py --split-only --config configs/dataset.yaml`
Direct scripts (if needed):
- `python scripts/data/download_visual_genome.py --config configs/dataset.yaml`
- `python scripts/data/extract_visual_genome.py --config configs/dataset.yaml`
- `python scripts/data/process_visual_genome.py --config configs/dataset.yaml`
- `python scripts/data/build_splits.py --config configs/dataset.yaml`
Important: the processing pipeline is strict chair-only (`primary_object_class: chair`).
### 2) Train/Evaluate Models
#### Baseline (CLIP/DINO + MLP)
- Smoke run (quick sanity check):
`python scripts/train_baseline.py --run-name smoke_baseline --mode smoke --device auto`
- Full run:
`python scripts/train_baseline.py --run-name baseline_full --mode full --device auto`
#### GNN (CLIP/DINO + bipartite native GNN)
- GNN smoke run:
`python scripts/train_gnn.py --run-name smoke_gnn --mode smoke --device auto`
- Compare baseline vs GNN (after both smoke runs complete):
`python scripts/compare_baseline_vs_gnn.py --baseline-run smoke_baseline --gnn-run smoke_gnn`
### 3) Inference and Gradio UI
#### CLI inference
- Baseline checkpoint:
`python scripts/infer.py --checkpoint outputs/smoke_baseline/best.pt --image path/to/chair.jpg --top-k 5 --threshold 0.5`
- GNN checkpoint:
`python scripts/infer.py --checkpoint outputs/smoke_gnn/best.pt --image path/to/chair.jpg --top-k 5 --threshold 0.5`
#### Gradio app (local)
- Launch:
`python app.py`
- Then:
- Open the printed URL.
- Upload a chair image.
- Choose **baseline** or **gnn**.
- Adjust threshold/top-k as needed.
## Expected Outputs
Data artifacts:
- `data/processed/visual_genome/samples.parquet` (or `samples.csv` fallback)
- `data/processed/visual_genome/label_vocab.json`
- `data/processed/visual_genome/label_frequencies.json`
- `data/processed/visual_genome/processing_report.json`
- `data/processed/visual_genome/splits/train.json`
- `data/processed/visual_genome/splits/val.json`
- `data/processed/visual_genome/splits/test.json`
- `data/processed/visual_genome/splits/split_report.json`
Training/eval artifacts:
- model checkpoints
- evaluation reports (mAP/F1/per-attribute)
- embedding analysis artifacts (UMAP/retrieval results)
- error analysis summaries
- minimal Gradio demo for interactive predictions
## Notes
- Keep experiments reproducible with fixed seeds.
- Track all runs with consistent config snapshots.
- Measure gains against **Baseline C** to isolate graph contribution.
|