Spaces:
Sleeping
Sleeping
| title: GraphAttributeLearning | |
| emoji: 🪑 | |
| colorFrom: gray | |
| colorTo: red | |
| sdk: gradio | |
| sdk_version: "6.9.0" | |
| python_version: "3.10" | |
| app_file: app.py | |
| pinned: false | |
| # Adjective-Aware Object Embedding | |
| This project builds adjective-aware object embeddings using: | |
| - CLIP image features | |
| - DINOv2 image features | |
| - Graph Neural Network (GNN) reasoning over object-attribute graphs | |
| Current scope starts with `chair` and multi-label attributes like `broken`, `blue`, `wooden`, and `plastic`. | |
| ## Project Structure | |
| - `plan/plan.md` - detailed execution strategy and technical plan | |
| - `configs/dataset.yaml` - dataset, attributes, splits, normalization | |
| - `configs/model.yaml` - encoder, fusion, graph, and head settings | |
| - `configs/train.yaml` - optimization, staging, losses, checkpointing | |
| - `configs/eval.yaml` - metrics, visualization, error analysis | |
| - `configs/experiment.yaml` - baseline/main/ablation run definitions | |
| - `scripts/data/` - data download, extraction, processing, split pipeline | |
| - `src/data/` - reusable normalization and IO utilities for data pipeline | |
| ## Config Overview | |
| ### `dataset.yaml` | |
| Defines: | |
| - dataset root/cache/processed paths | |
| - attribute taxonomy and synonym normalization | |
| - split strategy and filtering rules | |
| - basic augmentation | |
| ### `model.yaml` | |
| Defines: | |
| - CLIP + DINO encoder settings | |
| - fusion type (`concat`, optional gated/projected ablations) | |
| - graph variant (`bipartite` / `enriched`) | |
| - attribute node initialization (`clip_text` / `trainable`) | |
| - GNN layers and baseline MLP head | |
| ### `train.yaml` | |
| Defines: | |
| - two-stage training (frozen then partial fine-tuning) | |
| - optimizer, scheduler, AMP, grad clipping | |
| - weighted BCE + optional contrastive loss | |
| - early stopping and checkpoint monitoring | |
| ### `eval.yaml` | |
| Defines: | |
| - classification metrics (mAP, macro/micro F1, per-attribute metrics) | |
| - embedding metrics (retrieval@K, silhouette) | |
| - UMAP/t-SNE options | |
| - error analysis outputs | |
| ### `experiment.yaml` | |
| Defines: | |
| - baseline runs (A/B/C) | |
| - main proposed run (CLIP+DINO+GNN) | |
| - ablation runs (graph type, node init, loss, fine-tuning) | |
| ## Execution Strategy (No Timeline) | |
| Follow this order: | |
| 1. Data prep + label normalization | |
| 2. Baselines (A/B/C) | |
| 3. Graph model training | |
| 4. Ablations and tuning | |
| 5. Robustness checks + packaging | |
| 6. Build a minimal Gradio demo interface (final step, after model is fully trained and selected) | |
| Use `plan/plan.md` as the source of truth for quality gates and deliverables. | |
| ## How To Run | |
| ### 1) Run Chair-Only Data Pipeline (Visual Genome) | |
| Default full pipeline: | |
| `python scripts/data/run_data_pipeline.py --config configs/dataset.yaml` | |
| Run individual stages: | |
| - `python scripts/data/run_data_pipeline.py --download-only --config configs/dataset.yaml` | |
| - `python scripts/data/run_data_pipeline.py --extract-only --config configs/dataset.yaml` | |
| - `python scripts/data/run_data_pipeline.py --process-only --config configs/dataset.yaml` | |
| - `python scripts/data/run_data_pipeline.py --split-only --config configs/dataset.yaml` | |
| Direct scripts (if needed): | |
| - `python scripts/data/download_visual_genome.py --config configs/dataset.yaml` | |
| - `python scripts/data/extract_visual_genome.py --config configs/dataset.yaml` | |
| - `python scripts/data/process_visual_genome.py --config configs/dataset.yaml` | |
| - `python scripts/data/build_splits.py --config configs/dataset.yaml` | |
| Important: the processing pipeline is strict chair-only (`primary_object_class: chair`). | |
| ### 2) Train/Evaluate Models | |
| #### Baseline (CLIP/DINO + MLP) | |
| - Smoke run (quick sanity check): | |
| `python scripts/train_baseline.py --run-name smoke_baseline --mode smoke --device auto` | |
| - Full run: | |
| `python scripts/train_baseline.py --run-name baseline_full --mode full --device auto` | |
| #### GNN (CLIP/DINO + bipartite native GNN) | |
| - GNN smoke run: | |
| `python scripts/train_gnn.py --run-name smoke_gnn --mode smoke --device auto` | |
| - Compare baseline vs GNN (after both smoke runs complete): | |
| `python scripts/compare_baseline_vs_gnn.py --baseline-run smoke_baseline --gnn-run smoke_gnn` | |
| ### 3) Inference and Gradio UI | |
| #### CLI inference | |
| - Baseline checkpoint: | |
| `python scripts/infer.py --checkpoint outputs/smoke_baseline/best.pt --image path/to/chair.jpg --top-k 5 --threshold 0.5` | |
| - GNN checkpoint: | |
| `python scripts/infer.py --checkpoint outputs/smoke_gnn/best.pt --image path/to/chair.jpg --top-k 5 --threshold 0.5` | |
| #### Gradio app (local) | |
| - Launch: | |
| `python app.py` | |
| - Then: | |
| - Open the printed URL. | |
| - Upload a chair image. | |
| - Choose **baseline** or **gnn**. | |
| - Adjust threshold/top-k as needed. | |
| ## Expected Outputs | |
| Data artifacts: | |
| - `data/processed/visual_genome/samples.parquet` (or `samples.csv` fallback) | |
| - `data/processed/visual_genome/label_vocab.json` | |
| - `data/processed/visual_genome/label_frequencies.json` | |
| - `data/processed/visual_genome/processing_report.json` | |
| - `data/processed/visual_genome/splits/train.json` | |
| - `data/processed/visual_genome/splits/val.json` | |
| - `data/processed/visual_genome/splits/test.json` | |
| - `data/processed/visual_genome/splits/split_report.json` | |
| Training/eval artifacts: | |
| - model checkpoints | |
| - evaluation reports (mAP/F1/per-attribute) | |
| - embedding analysis artifacts (UMAP/retrieval results) | |
| - error analysis summaries | |
| - minimal Gradio demo for interactive predictions | |
| ## Notes | |
| - Keep experiments reproducible with fixed seeds. | |
| - Track all runs with consistent config snapshots. | |
| - Measure gains against **Baseline C** to isolate graph contribution. | |