AutoMLOps / README.md
mnoorchenar's picture
Sync from GitHub 56c7c35: README: reliable Star/Fork buttons
67e589d
|
Raw History Blame Contribute Delete
15.1 kB
---
title: AutoMLOps
colorFrom: purple
colorTo: blue
sdk: docker
app_port: 7860
pinned: false
---
<div align="center">
<h1>πŸ€– AutoMLOps</h1>
<img src="https://readme-typing-svg.demolab.com?font=Fira+Code&size=22&duration=3000&pause=1000&color=4F46E5&center=true&vCenter=true&width=700&lines=ML+Experiment+Tracking+%26+Pipeline+Platform;Visual+DAG+Orchestration+with+Apache+Airflow;50%2B+Algorithms+%C2%B7+AutoML+%C2%B7+Model+Registry" alt="Typing SVG"/>
<br/>
[![Python](https://img.shields.io/badge/Python-3.11-3b82f6?style=for-the-badge&logo=python&logoColor=white)](https://www.python.org/)
[![Flask](https://img.shields.io/badge/Flask-3.x-4f46e5?style=for-the-badge&logo=flask&logoColor=white)](https://flask.palletsprojects.com/)
[![Airflow](https://img.shields.io/badge/Apache_Airflow-2.10-017CEE?style=for-the-badge&logo=apacheairflow&logoColor=white)](https://airflow.apache.org/)
[![MLflow](https://img.shields.io/badge/MLflow-2.x-0194E2?style=for-the-badge&logo=mlflow&logoColor=white)](https://mlflow.org/)
[![Docker](https://img.shields.io/badge/Docker-Ready-3b82f6?style=for-the-badge&logo=docker&logoColor=white)](https://www.docker.com/)
[![Status](https://img.shields.io/badge/Status-Active-22c55e?style=for-the-badge)](#)
<br/>
[![Live Demo](https://img.shields.io/badge/%E2%96%B6_Live_Demo-Hugging_Face_Space-ffcc00?style=for-the-badge&logo=huggingface&logoColor=black)](https://huggingface.co/spaces/mnoorchenar/AutoMLOps)
[![Source Code](https://img.shields.io/badge/Source_Code-GitHub-181717?style=for-the-badge&logo=github&logoColor=white)](https://github.com/mnoorchenar/AutoMLOps)
<br/>
**πŸ€– AutoMLOps** β€” A full-stack ML experiment tracking and pipeline orchestration platform. Train 50+ algorithms, run visual DAG pipelines powered by Apache Airflow, and manage models β€” all in one Docker container deployed to HuggingFace Spaces.
<br/>
---
</div>
## Table of Contents
- [Features](#-features)
- [Architecture](#️-architecture)
- [Getting Started](#-getting-started)
- [Docker Deployment](#-docker-deployment)
- [Pages](#-pages)
- [Pipelines](#-pipelines)
- [ML Algorithms](#-ml-algorithms)
- [Project Structure](#-project-structure)
- [Author](#-author)
- [Contributing](#-contributing)
- [Disclaimer](#️-disclaimer)
- [License](#-license)
---
## ✨ Features
<table>
<tr>
<td>🎨 <b>Pipeline Studio</b></td>
<td>Interactive full-screen DAG canvas with clickable nodes, slide-in config panel, and live execution terminal</td>
</tr>
<tr>
<td>✈️ <b>Real Apache Airflow</b></td>
<td>Pipelines execute as genuine Airflow DAGs with XCom, TaskInstance tracking, and DagRun polling</td>
</tr>
<tr>
<td>πŸ€– <b>AutoML Engine</b></td>
<td>Automated hyperparameter search across all algorithm categories for classification and regression tasks</td>
</tr>
<tr>
<td>πŸ“ˆ <b>MLflow Tracking</b></td>
<td>Every training run logs parameters, metrics, and model artifacts to a persistent SQLite-backed MLflow store</td>
</tr>
<tr>
<td>πŸ“¦ <b>Model Registry</b></td>
<td>Register, version, and transition models through Staging β†’ Production β†’ Archived lifecycle stages</td>
</tr>
<tr>
<td>πŸŒ™ <b>Theme Toggle</b></td>
<td>Dark and light mode with instant CSS variable switching and localStorage persistence</td>
</tr>
<tr>
<td>🐳 <b>Single-Container Deployment</b></td>
<td>Flask + Airflow Scheduler + SQLite in one Docker image β€” no external services required</td>
</tr>
</table>
---
## πŸ—οΈ Architecture
```
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ AutoMLOps β”‚
β”‚ β”‚
β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚
β”‚ β”‚ Datasets │───▢│ MLOps Engine │───▢│ Flask API β”‚ β”‚
β”‚ β”‚ (sklearn + β”‚ β”‚ (sklearn / β”‚ β”‚ Backend β”‚ β”‚
β”‚ β”‚ custom) β”‚ β”‚ XGBoost / β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚
β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ LightGBM / β”‚ β”‚ β”‚
β”‚ β”‚ MLP) β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚
β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ Pipeline Studio β”‚ β”‚
β”‚ β”‚ AutoML Page β”‚ β”‚
β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ Model Registry β”‚ β”‚
β”‚ β”‚ MLflow DB │◀───│ Airflow β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚
β”‚ β”‚ (SQLite) β”‚ β”‚ Scheduler β”‚ β”‚
β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ (DAGs / XCom / β”‚ β”‚
β”‚ β”‚ TaskInstance) β”‚ β”‚
β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
```
---
## πŸš€ Getting Started
### Prerequisites
- Python 3.11+
- Docker (for containerised deployment)
- Git
### Local Installation
```bash
# 1. Clone the repository
git clone https://github.com/mnoorchenar/AutoMLOps.git
cd AutoMLOps
# 2. Create a virtual environment
python -m venv venv
source venv/bin/activate # Windows: venv\Scripts\activate
# 3. Install app dependencies
pip install -r requirements.txt
# 4. Install Apache Airflow with official constraints
AIRFLOW_VERSION=2.10.4
pip install "apache-airflow==${AIRFLOW_VERSION}" \
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-3.11.txt"
# 5. Initialise Airflow metadata DB
export AIRFLOW_HOME=$(pwd)/airflow_home
export AIRFLOW__CORE__DAGS_FOLDER=$(pwd)/dags
airflow db migrate
# 6. Start the Airflow scheduler (background)
airflow scheduler &
# 7. Run the Flask application
python app.py
```
Open your browser at `http://localhost:7860` πŸŽ‰
---
## 🐳 Docker Deployment
```bash
# Build and run
docker build -t automlops .
docker run -p 7860:7860 automlops
# Or deploy directly to HuggingFace Spaces
# Push to a Space with sdk: docker in README frontmatter
```
The Dockerfile builds a single image containing:
- Flask application (served via Gunicorn on port 7860)
- Apache Airflow Scheduler (started by `start.sh`)
- All Python dependencies with Airflow constraint-file pinning
- Pre-initialised Airflow SQLite metadata DB (`airflow db migrate`)
---
## πŸ“„ Pages
| Page | Route | Description | Status |
|------|-------|-------------|--------|
| 🎨 Pipeline Studio | `/` | Interactive DAG canvas β€” click nodes to configure and execute pipelines | βœ… Live |
| πŸ€– AutoML | `/automl` | Automated algorithm search across 50+ models for any dataset | βœ… Live |
| πŸ“¦ Model Registry | `/models` | Browse registered models, versions, and lifecycle stages | βœ… Live |
---
## πŸ”— Pipelines
Three production-quality pipelines are pre-built and immediately executable from the Pipeline Studio:
**Training Pipeline** (`training_pipeline`)
```
Load Data β†’ Validate β†’ Preprocess β†’ Feature Engineering β†’ Train Model β†’ Evaluate β†’ Report β†’ Register β†’ Deploy to Staging
```
**Retraining Pipeline** (`retraining_pipeline`)
```
Drift Detection β†’ Fetch New Data β†’ Merge Datasets β†’ Retrain Champion β†’ A/B Test β†’ Promote to Production
```
**Data Processing Pipeline** (`data_pipeline`)
```
Ingest Raw Data β†’ Clean Data β†’ Encode Features β†’ Scale Features β†’ Save to Feature Store
```
Each pipeline node is clickable in the UI β€” configurable nodes (dataset picker, algorithm picker) show a purple indicator dot. Execution logs stream live in the built-in terminal panel.
---
## 🧠 ML Algorithms
```python
# AutoMLOps Algorithm Registry β€” 50+ algorithms across 2 tasks
ALGORITHMS = {
"classification": {
"Linear Models": ["Logistic Regression", "Logistic Regression (L1)", "Ridge Classifier",
"SGD Classifier", "Passive Aggressive", "Linear Discriminant Analysis"],
"Tree-Based": ["Decision Tree", "Random Forest", "Extra Trees",
"Quadratic Discriminant Analysis"],
"Ensemble / Boosting": ["Gradient Boosting", "AdaBoost", "Bagging Classifier",
"XGBoost", "LightGBM"],
"Support Vector Machines":["SVC (RBF Kernel)", "SVC (Polynomial)", "SVC (Linear)", "LinearSVC"],
"Probabilistic": ["Gaussian Naive Bayes", "Bernoulli Naive Bayes", "Complement Naive Bayes"],
"Instance-Based (KNN)": ["KNN (k=3)", "KNN (k=5)", "KNN (k=9)"],
"Neural Networks": ["MLP (Small)", "MLP (Medium)", "MLP (Deep)"],
},
"regression": {
"Linear Models": ["Linear Regression", "Ridge Regression", "Lasso",
"ElasticNet", "Bayesian Ridge", "Huber Regressor"],
"Tree-Based": ["Decision Tree Regressor", "Random Forest Regressor",
"Extra Trees Regressor"],
"Ensemble / Boosting": ["Gradient Boosting Regressor", "AdaBoost Regressor",
"Bagging Regressor", "XGBoost Regressor", "LightGBM Regressor"],
"Support Vector Machines":["SVR (RBF)", "SVR (Linear)"],
"Instance-Based (KNN)": ["KNN Regressor (k=3)", "KNN Regressor (k=5)"],
"Neural Networks": ["MLP Regressor (Small)", "MLP Regressor (Medium)"],
}
}
```
**Built-in Datasets:**
| Dataset | Task | Samples | Features |
|---------|------|---------|---------|
| Iris Flowers | Classification | 150 | 4 |
| Wine Quality | Classification | 178 | 13 |
| Breast Cancer | Classification | 569 | 30 |
| Diabetes Progression | Regression | 442 | 10 |
| California Housing | Regression | 20,640 | 8 |
---
## πŸ“ Project Structure
```
AutoMLOps/
β”‚
β”œβ”€β”€ πŸ“‚ dags/ # Apache Airflow DAG files (parsed by scheduler)
β”‚ β”œβ”€β”€ __init__.py
β”‚ β”œβ”€β”€ data_pipeline.py
β”‚ β”œβ”€β”€ retraining_pipeline.py
β”‚ └── training_pipeline.py
β”œβ”€β”€ πŸ“‚ mlops/
β”‚ β”œβ”€β”€ __init__.py
β”‚ β”œβ”€β”€ airflow_runner.py
β”‚ β”œβ”€β”€ algorithms.py
β”‚ β”œβ”€β”€ datasets.py
β”‚ └── trainer.py
β”œβ”€β”€ πŸ“‚ pipelines/
β”‚ β”œβ”€β”€ __init__.py
β”‚ β”œβ”€β”€ dag_engine.py
β”‚ └── pipeline_defs.py
β”œβ”€β”€ πŸ“‚ static/ # CSS, JavaScript & assets
β”‚ β”œβ”€β”€ css/
β”‚ └── js/
β”œβ”€β”€ πŸ“‚ templates/ # HTML templates
β”‚ β”œβ”€β”€ automl.html
β”‚ β”œβ”€β”€ base.html
β”‚ β”œβ”€β”€ dashboard.html
β”‚ β”œβ”€β”€ experiments.html
β”‚ β”œβ”€β”€ models.html
β”‚ β”œβ”€β”€ pipeline.html
β”‚ └── … 1 more
β”œβ”€β”€ πŸ“„ Dockerfile # Single-container image (Flask + Airflow)
β”œβ”€β”€ πŸ“„ app.py # Flask application entry point + all API routes
β”œβ”€β”€ πŸ“„ docs-template.html
β”œβ”€β”€ πŸ“„ docs.html
β”œβ”€β”€ πŸ“„ readme-template.md
β”œβ”€β”€ πŸ“„ requirements.txt # Python dependencies
└── πŸ“„ start.sh # Startup: Airflow scheduler β†’ Gunicorn Flask
```
---
## πŸ‘¨β€πŸ’» Author
<div align="center">
<table>
<tr>
<td align="center" width="100%">
<img src="https://avatars.githubusercontent.com/mnoorchenar" width="120" style="border-radius:50%; border: 3px solid #4f46e5;" alt="Mohammad Noorchenarboo"/>
<h3>Mohammad Noorchenarboo</h3>
<code>Data Scientist</code> &nbsp;|&nbsp; <code>AI Researcher</code> &nbsp;|&nbsp; <code>Biostatistician</code>
πŸ“ &nbsp;Ontario, Canada &nbsp;&nbsp; πŸ“§ &nbsp;[mohammadnoorchenarboo@gmail.com](mailto:mohammadnoorchenarboo@gmail.com)
──────────────────────────────────────
[![LinkedIn](https://img.shields.io/badge/LinkedIn-0077B5?style=for-the-badge&logo=linkedin&logoColor=white)](https://www.linkedin.com/in/mnoorchenar)&nbsp;
[![Personal Site](https://img.shields.io/badge/Website-mnoorchenar.github.io-4f46e5?style=for-the-badge&logo=githubpages&logoColor=white)](https://mnoorchenar.github.io/)&nbsp;
[![HuggingFace](https://img.shields.io/badge/HuggingFace-ffcc00?style=for-the-badge&logo=huggingface&logoColor=black)](https://huggingface.co/mnoorchenar)&nbsp;
[![Google Scholar](https://img.shields.io/badge/Scholar-4285F4?style=for-the-badge&logo=googlescholar&logoColor=white)](https://scholar.google.ca/citations?user=nn_Toq0AAAAJ&hl=en)&nbsp;
[![GitHub](https://img.shields.io/badge/GitHub-181717?style=for-the-badge&logo=github&logoColor=white)](https://github.com/mnoorchenar)
</td>
</tr>
</table>
</div>
---
## 🀝 Contributing
Contributions are welcome! Please follow these steps:
1. **Fork** the repository
2. **Create** a feature branch: `git checkout -b feature/amazing-feature`
3. **Commit** your changes: `git commit -m 'Add amazing feature'`
4. **Push** to the branch: `git push origin feature/amazing-feature`
5. **Open** a Pull Request
---
## ⚠️ Disclaimer
> [!WARNING]
> This project is developed strictly for educational and research purposes and does not constitute professional advice of any kind. All datasets used are either synthetically generated or publicly available β€” no real user data is stored. This software is provided "as is" without warranty of any kind; use at your own risk.
---
## πŸ“œ License
Distributed under the **MIT License**.
---
<div align="center">
<img src="https://capsule-render.vercel.app/api?type=waving&color=0:3b82f6,100:4f46e5&height=120&section=footer&text=Made%20with%20%E2%9D%A4%EF%B8%8F%20by%20Mohammad%20Noorchenarboo&fontColor=ffffff&fontSize=18&fontAlignY=80" width="100%"/>
[![Star on GitHub](https://img.shields.io/badge/%E2%AD%90_Star-on_GitHub-181717?style=for-the-badge&logo=github&logoColor=white)](https://github.com/mnoorchenar/AutoMLOps/stargazers)
[![Fork on GitHub](https://img.shields.io/badge/%F0%9F%8D%B4_Fork-on_GitHub-181717?style=for-the-badge&logo=github&logoColor=white)](https://github.com/mnoorchenar/AutoMLOps/fork)
<sub>The name "AutoMLOps" is used purely for academic and research purposes. Any similarity to existing product names or trademarks is entirely coincidental. This project has no affiliation with any commercial entity.</sub>
</div>