Spaces:
Runtime error
Runtime error
| title: Lineage Graph Accelerator | |
| emoji: 🔥 | |
| colorFrom: purple | |
| colorTo: blue | |
| sdk: gradio | |
| sdk_version: 6.0.0 | |
| app_file: app.py | |
| pinned: true | |
| license: mit | |
| short_description: AI data lineage extraction & export to data catalogs | |
| tags: | |
| - data-lineage | |
| - mcp | |
| - gradio | |
| - data-governance | |
| - dbt | |
| - airflow | |
| - etl | |
| - mcp-in-action-track-productivity | |
| - hackathon | |
| # Lineage Graph Accelerator 🔥 | |
| **AI-powered data lineage extraction and visualization for modern data platforms** | |
| [](https://huggingface.co/spaces/aamanlamba/Lineage-graph-accelerator) | |
| [](https://opensource.org/licenses/MIT) | |
| [](https://gradio.app) | |
| > 🎉 **Built for the Gradio Agents & MCP Hackathon - Winter 2025** 🎉 | |
| > | |
| > Celebrating MCP's 1st Birthday! This project demonstrates the power of MCP integration for enterprise data governance. | |
| --- | |
| ## 🌟 What is Lineage Graph Accelerator? | |
| Lineage Graph Accelerator is an AI-powered tool that helps data teams: | |
| - **Extract** data lineage from dbt, Airflow, BigQuery, Snowflake, and more | |
| - **Visualize** complex data dependencies with interactive Mermaid diagrams | |
| - **Export** lineage to enterprise data catalogs (Collibra, Microsoft Purview, Alation) | |
| - **Integrate** with MCP servers for enhanced AI-powered processing | |
| ### Why Data Lineage Matters | |
| Understanding where your data comes from and where it goes is critical for: | |
| - **Data Quality**: Track data transformations and identify issues | |
| - **Compliance**: Document data flows for GDPR, CCPA, and other regulations | |
| - **Impact Analysis**: Understand downstream effects of schema changes | |
| - **Data Discovery**: Help analysts find and trust data assets | |
| --- | |
| ## 🎯 Key Features | |
| ### Multi-Source Support | |
| | Source | Status | Description | | |
| |--------|--------|-------------| | |
| | dbt Manifest | ✅ | Parse dbt's manifest.json for model dependencies | | |
| | Airflow DAG | ✅ | Extract task dependencies from DAG definitions | | |
| | SQL DDL | ✅ | Parse CREATE statements for table lineage | | |
| | BigQuery | ✅ | Query INFORMATION_SCHEMA for metadata | | |
| | Custom JSON | ✅ | Flexible node/edge format for any source | | |
| | Snowflake | 🔄 | Coming via MCP integration | | |
| ### Export to Data Catalogs | |
| | Catalog | Status | Format | | |
| |---------|--------|--------| | |
| | OpenLineage | ✅ | Universal open standard | | |
| | Collibra | ✅ | Data Intelligence Platform | | |
| | Microsoft Purview | ✅ | Azure Data Governance | | |
| | Alation | ✅ | Data Catalog | | |
| | Apache Atlas | 🔄 | Coming soon | | |
| ### Visualization Options | |
| - **Mermaid Diagrams**: Interactive, client-side rendering | |
| - **Subgraph Grouping**: Organize by data layer (raw, staging, marts) | |
| - **Color-Coded Nodes**: Distinguish sources, tables, models, reports | |
| - **Edge Labels**: Show transformation types | |
| --- | |
| ## 🚀 Quick Start | |
| ### Try Online (HuggingFace Space) | |
| 1. Visit [Lineage Graph Accelerator on HuggingFace](https://huggingface.co/spaces/YOUR_SPACE) | |
| 2. Click "Load Sample" to load example data | |
| 3. Click "Extract Lineage" to see the visualization | |
| 4. Explore the Demo Gallery for more examples | |
| ### Run Locally | |
| ```bash | |
| # Clone the repository | |
| git clone https://github.com/YOUR_REPO/lineage-graph-accelerator.git | |
| cd lineage-graph-accelerator | |
| # Create virtual environment | |
| python3 -m venv .venv | |
| source .venv/bin/activate | |
| # Install dependencies | |
| pip install -r requirements.txt | |
| # Run the app | |
| python app.py | |
| ``` | |
| Open http://127.0.0.1:7860 in your browser. | |
| --- | |
| ## 📖 Usage Guide | |
| ### 1. Text/File Metadata Tab | |
| Paste your metadata directly: | |
| ```json | |
| { | |
| "nodes": [ | |
| {"id": "source_db", "type": "source", "name": "Source Database"}, | |
| {"id": "staging", "type": "table", "name": "Staging Table"}, | |
| {"id": "analytics", "type": "table", "name": "Analytics Table"} | |
| ], | |
| "edges": [ | |
| {"from": "source_db", "to": "staging"}, | |
| {"from": "staging", "to": "analytics"} | |
| ] | |
| } | |
| ``` | |
| ### 2. Sample Data | |
| Load pre-built samples to explore different scenarios: | |
| - **Simple JSON**: Basic node/edge lineage | |
| - **dbt Manifest**: Full dbt project with 15+ models | |
| - **Airflow DAG**: ETL pipeline with 15 tasks | |
| - **Data Warehouse**: Snowflake-style multi-layer architecture | |
| - **ETL Pipeline**: Complex multi-source pipeline | |
| - **Complex Demo**: 50+ node e-commerce platform | |
| ### 3. Export to Data Catalogs | |
| 1. Extract lineage from your metadata | |
| 2. Expand "Export to Data Catalog" | |
| 3. Select format (OpenLineage, Collibra, Purview, Alation) | |
| 4. Click "Generate Export" | |
| 5. Copy the JSON for import into your catalog | |
| --- | |
| ## 🔌 MCP Integration | |
| Connect to MCP (Model Context Protocol) servers for enhanced processing: | |
| ``` | |
| ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ | |
| │ Lineage Graph │────▶│ MCP Server │────▶│ AI Model │ | |
| │ Accelerator │ │ (HuggingFace) │ │ (Claude) │ | |
| └─────────────────┘ └─────────────────┘ └─────────────────┘ | |
| ``` | |
| ### Configuration | |
| 1. Expand "MCP Server Configuration" in the UI | |
| 2. Enter your MCP server URL | |
| 3. Add API key (if required) | |
| 4. Click "Test Connection" | |
| ### Run Local MCP Server | |
| ```bash | |
| uvicorn mcp_example.server:app --reload --port 9000 | |
| ``` | |
| Then use `http://localhost:9000/mcp` as your server URL. | |
| --- | |
| ## 🏗️ Architecture | |
| ```mermaid | |
| flowchart TD | |
| A[User Interface - Gradio] --> B[Input Parser] | |
| B --> C{Source Type} | |
| C -->|dbt| D[dbt Parser] | |
| C -->|Airflow| E[Airflow Parser] | |
| C -->|SQL| F[SQL Parser] | |
| C -->|JSON| G[JSON Parser] | |
| D & E & F & G --> H[LineageGraph] | |
| H --> I[Mermaid Generator] | |
| H --> J[Export Engine] | |
| I --> K[Visualization] | |
| J --> L[OpenLineage] | |
| J --> M[Collibra] | |
| J --> N[Purview] | |
| J --> O[Alation] | |
| subgraph Optional | |
| P[MCP Server] --> H | |
| end | |
| ``` | |
| ### Project Structure | |
| ``` | |
| lineage-graph-accelerator/ | |
| ├── app.py # Main Gradio application | |
| ├── exporters/ # Data catalog exporters | |
| │ ├── __init__.py | |
| │ ├── base.py # Base classes | |
| │ ├── openlineage.py # OpenLineage format | |
| │ ├── collibra.py # Collibra format | |
| │ ├── purview.py # Microsoft Purview format | |
| │ └── alation.py # Alation format | |
| ├── samples/ # Sample data files | |
| │ ├── sample_metadata.json | |
| │ ├── dbt_manifest_sample.json | |
| │ ├── airflow_dag_sample.json | |
| │ ├── sql_ddl_sample.sql | |
| │ ├── warehouse_lineage_sample.json | |
| │ ├── etl_pipeline_sample.json | |
| │ └── complex_lineage_demo.json | |
| ├── mcp_example/ # Example MCP server | |
| │ └── server.py | |
| ├── tests/ # Unit tests | |
| │ └── test_app.py | |
| ├── memories/ # Agent configuration | |
| ├── USER_GUIDE.md # Comprehensive user guide | |
| ├── BUILD_PLAN.md # Development roadmap | |
| └── requirements.txt | |
| ``` | |
| --- | |
| ## 🧪 Testing | |
| ```bash | |
| # Activate virtual environment | |
| source .venv/bin/activate | |
| # Run unit tests | |
| python -m unittest tests.test_app -v | |
| # Run setup validation | |
| python test_setup.py | |
| ``` | |
| --- | |
| ## 📋 Requirements | |
| - Python 3.9+ | |
| - Gradio 5.49.1+ | |
| - See `requirements.txt` for full dependencies | |
| --- | |
| ## 🎖️ Competition Submission | |
| **Track**: Track 2 - MCP in Action (Productivity) | |
| **Team Members**: | |
| - [Aaman Lamba](https://aamanlamba.com) | [HuggingFace](https://huggingface.co/aamanlamba) | [GitHub](https://github.com/aamanlamba) | |
| ### Judging Criteria Alignment | |
| | Criteria | Implementation | | |
| |----------|----------------| | |
| | **UI/UX Design** | Clean, professional interface with tabs, accordions, and color-coded visualizations | | |
| | **Functionality** | Full MCP integration, multiple input formats, 5 export formats | | |
| | **Creativity** | Novel approach to data lineage visualization with AI-powered parsing | | |
| | **Documentation** | Comprehensive README, USER_GUIDE.md, inline comments | | |
| | **Real-world Impact** | Solves critical enterprise need for data governance and compliance | | |
| ### Demo Video | |
| 📺 **YouTube**: [Watch the Demo](https://youtu.be/U4Dfc7txa_0) | |
| 🎥 **Loom**: [Alternative Link](https://www.loom.com/share/3de27e88e01f4e97bfd13e4f0031f416) | |
| **Highlights**: | |
| - AI Assistant with Google Gemini generating lineage from natural language | |
| - MCP Integration with Local Demo server | |
| - Demo Gallery with 50+ node complex pipelines | |
| - Export to Collibra, Purview, and Apache Atlas | |
| - Interactive Mermaid visualizations with zoom and download | |
| ### Social Media Post | |
| 📱 **LinkedIn**: [View the announcement post](https://www.linkedin.com/posts/aamanlamba_lineage-graph-accelerator-a-hugging-face-activity-7400658296166297600-n9a6) | |
| --- | |
| ## 🔜 Roadmap | |
| - [x] Gradio 6 upgrade for enhanced UI components | |
| - [x] Agentic chatbot for natural language queries (Google Gemini) | |
| - [x] Apache Atlas export support | |
| - [ ] File upload functionality | |
| - [x] Graph export as PNG/SVG | |
| - [ ] Batch processing API | |
| - [ ] Column-level lineage | |
| --- | |
| ## 🤝 Contributing | |
| Contributions welcome! Please: | |
| 1. Fork the repository | |
| 2. Create a feature branch | |
| 3. Make your changes | |
| 4. Submit a pull request | |
| See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines. | |
| --- | |
| ## 📄 License | |
| MIT License - see [LICENSE](LICENSE) for details. | |
| --- | |
| ## 🙏 Acknowledgments | |
| - **Anthropic** - MCP Protocol and Claude | |
| - **Gradio Team** - Amazing UI framework | |
| - **HuggingFace** - Hosting and community | |
| - **dbt Labs** - Inspiration for metadata standards | |
| - **OpenLineage** - Open lineage specification | |
| --- | |
| ## 📞 Support | |
| - **Documentation**: [USER_GUIDE.md](USER_GUIDE.md) | |
| - **Author Website**: [aamanlamba.com](https://aamanlamba.com) | |
| - **Issues**: [GitHub Issues](https://github.com/aamanlamba/lineage-graph-accelerator/issues) | |
| - **Discussion**: [HuggingFace Community](https://huggingface.co/spaces/aamanlamba/Lineage-graph-accelerator/discussions) | |
| --- | |
| <p align="center"> | |
| Built with ❤️ by <a href="https://aamanlamba.com"><strong>Aaman Lamba</strong></a> for the <strong>Gradio Agents & MCP Hackathon - Winter 2025</strong> | |
| <br> | |
| Celebrating MCP's 1st Birthday! 🎂 | |
| </p> | |