Predictive_maintaimce / WHAT_TO_UPLOAD.md
pardhas133's picture
Upload 11 files
fc00054 verified
|
Raw History Blame Contribute Delete
8.12 kB
# What to Upload: GitHub vs Hugging Face
## Quick Summary
| Destination | What to Upload | How |
|------------|----------------|-----|
| **GitHub** | **Everything** (entire project folder) | `git push` |
| **HF Dataset** | Raw data + train/test splits | Scripts auto-upload |
| **HF Model** | Trained model file | Scripts auto-upload |
| **HF Space** | App files (code, Dockerfile, requirements) | Scripts auto-upload |
---
## πŸ“¦ GitHub Repository
### Upload: **Everything in the `mlops/` folder**
**What to include:**
```
mlops/
β”œβ”€β”€ data/
β”‚ β”œβ”€β”€ engine_data.csv βœ… Upload
β”‚ └── processed/
β”‚ β”œβ”€β”€ train.csv βœ… Upload
β”‚ └── test.csv βœ… Upload
β”œβ”€β”€ src/
β”‚ β”œβ”€β”€ app.py βœ… Upload
β”‚ β”œβ”€β”€ config.py βœ… Upload
β”‚ β”œβ”€β”€ data_prep.py βœ… Upload
β”‚ β”œβ”€β”€ data_register.py βœ… Upload
β”‚ β”œβ”€β”€ train.py βœ… Upload
β”‚ β”œβ”€β”€ inference.py βœ… Upload
β”‚ β”œβ”€β”€ deploy_to_hf.py βœ… Upload
β”‚ β”œβ”€β”€ eda.py βœ… Upload
β”‚ β”œβ”€β”€ hf_data_utils.py βœ… Upload
β”‚ └── hf_model_utils.py βœ… Upload
β”œβ”€β”€ notebooks/ βœ… Upload (if you have EDA notebooks)
β”œβ”€β”€ models/
β”‚ └── best_model.joblib ⚠️ Optional (large file)
β”œβ”€β”€ .github/
β”‚ └── workflows/
β”‚ └── pipeline.yml βœ… Upload (IMPORTANT!)
β”œβ”€β”€ requirements.txt βœ… Upload
β”œβ”€β”€ Dockerfile βœ… Upload
β”œβ”€β”€ README.md βœ… Upload
└── *.md files βœ… Upload (documentation)
❌ DON'T upload:
β”œβ”€β”€ .venv/ ❌ Skip (virtual environment)
β”œβ”€β”€ __pycache__/ ❌ Skip (Python cache)
β”œβ”€β”€ mlruns/ ❌ Skip (MLflow tracking - can be large)
└── .git/ ❌ Skip (git metadata)
```
### How to Upload to GitHub:
```bash
cd /Users/ananttripathi/Desktop/mlops
# Initialize git (if not already done)
git init
# Create .gitignore to exclude large/unnecessary files
cat > .gitignore << EOF
.venv/
__pycache__/
*.pyc
*.pyo
*.pyd
.Python
mlruns/
*.log
.DS_Store
EOF
# Add all files
git add .
# Commit
git commit -m "Initial commit: Predictive Maintenance MLOps Pipeline"
# Add remote (replace with your repo URL)
git remote add origin https://github.com/ananttripathi/engine-predictive-maintenance.git
# Push
git push -u origin main
```
---
## πŸ€— Hugging Face - Dataset Repo
**Repo**: `ananttripathiak/engine-maintenance-dataset`
### Upload: **Data files only**
**Files uploaded automatically by scripts:**
- `data/engine_data.csv` β†’ Uploaded as `data/engine_data.csv`
- `data/processed/train.csv` β†’ Uploaded as `data/train.csv`
- `data/processed/test.csv` β†’ Uploaded as `data/test.csv`
### How to Upload:
**Option 1: Run the scripts (automatic)**
```bash
# Step 1: Register raw data
python src/data_register.py
# Step 2: Prepare and upload train/test
python src/data_prep.py
```
**Option 2: Manual upload via HF Hub**
- Go to https://huggingface.co/datasets/ananttripathiak/engine-maintenance-dataset
- Click "Add file" β†’ Upload `data/engine_data.csv`
- Upload `data/processed/train.csv` and `test.csv`
---
## πŸ€— Hugging Face - Model Repo
**Repo**: `ananttripathiak/engine-maintenance-model`
### Upload: **Trained model file only**
**File uploaded automatically:**
- `models/best_model.joblib` β†’ Uploaded as `model.joblib`
### How to Upload:
**Option 1: Run the training script (automatic)**
```bash
python src/train.py
# This will:
# 1. Train the model
# 2. Save to models/best_model.joblib
# 3. Upload to HF Model Repo automatically
```
**Option 2: Manual upload via HF Hub**
- Go to https://huggingface.co/ananttripathiak/engine-maintenance-model
- Click "Add file" β†’ Upload `models/best_model.joblib`
- Rename it to `model.joblib` in the repo
---
## πŸ€— Hugging Face - Space (Streamlit App)
**Repo**: `ananttripathiak/engine-maintenance-space`
### Upload: **App deployment files**
**Files uploaded automatically by script:**
```
βœ… src/app.py (Main Streamlit app)
βœ… src/inference.py (Inference utilities)
βœ… src/config.py (Configuration)
βœ… Dockerfile (Container definition)
βœ… requirements.txt (Dependencies)
βœ… README.md (Documentation)
βœ… Other src/*.py files (If needed by app)
❌ NOT uploaded (ignored):
β”œβ”€β”€ data/ ❌ Too large
β”œβ”€β”€ mlruns/ ❌ MLflow tracking
β”œβ”€β”€ models/ ❌ Model is in Model Repo
β”œβ”€β”€ .github/ ❌ GitHub-specific
└── .venv/ ❌ Virtual environment
```
### How to Upload:
**Option 1: Run the deployment script (automatic)**
```bash
python src/deploy_to_hf.py
# This will:
# 1. Create/update the HF Space
# 2. Upload all deployment files
# 3. Configure it as a Streamlit app
```
**Option 2: Manual upload via HF Hub**
- Go to https://huggingface.co/spaces/ananttripathiak/engine-maintenance-space
- Upload files one by one or use HF CLI
---
## πŸ“‹ Complete Upload Checklist
### βœ… Step 1: GitHub (Manual - Do First)
- [ ] Create GitHub repository
- [ ] Add `.gitignore` file
- [ ] Push entire `mlops/` folder to GitHub
- [ ] Add GitHub Secrets (HF_TOKEN, HF_DATASET_REPO, HF_MODEL_REPO, HF_SPACE_REPO)
### βœ… Step 2: Hugging Face Dataset (Automatic)
- [ ] Set `HF_TOKEN` environment variable
- [ ] Run `python src/data_register.py` (uploads raw data)
- [ ] Run `python src/data_prep.py` (uploads train/test)
### βœ… Step 3: Hugging Face Model (Automatic)
- [ ] Run `python src/train.py` (trains model and uploads to HF)
### βœ… Step 4: Hugging Face Space (Automatic)
- [ ] Run `python src/deploy_to_hf.py` (deploys app to HF Space)
### βœ… Step 5: Verify
- [ ] Check GitHub repo: https://github.com/ananttripathi/engine-predictive-maintenance
- [ ] Check HF Dataset: https://huggingface.co/datasets/ananttripathiak/engine-maintenance-dataset
- [ ] Check HF Model: https://huggingface.co/ananttripathiak/engine-maintenance-model
- [ ] Check HF Space: https://huggingface.co/spaces/ananttripathiak/engine-maintenance-space
---
## 🎯 Key Differences
| Aspect | GitHub | Hugging Face |
|--------|--------|--------------|
| **Purpose** | Code repository & version control | Data/Model/App hosting |
| **What** | Entire project (code, data, docs) | Specific artifacts (data/model/app) |
| **How** | Manual `git push` | Automatic via scripts |
| **Size** | Can be large (includes everything) | Optimized (only what's needed) |
| **Access** | Public/Private repo | Public datasets/models/spaces |
| **CI/CD** | GitHub Actions workflow | HF Spaces auto-deploy |
---
## πŸ’‘ Pro Tips
1. **GitHub First**: Always push to GitHub first, then run HF scripts
2. **Use .gitignore**: Exclude large files like `mlruns/` and `.venv/` from GitHub
3. **HF Scripts are Smart**: They automatically create repos if they don't exist
4. **Check File Sizes**: HF has file size limits, so scripts exclude large files
5. **GitHub Secrets**: Store HF credentials in GitHub Secrets for CI/CD
---
## 🚨 Common Mistakes to Avoid
❌ **Don't upload to GitHub:**
- Virtual environment (`.venv/`)
- Large MLflow runs (`mlruns/`)
- Python cache files (`__pycache__/`)
❌ **Don't upload to HF Space:**
- Raw data files (too large)
- Model files (use Model Repo instead)
- MLflow tracking data
βœ… **Do upload to GitHub:**
- All source code
- Configuration files
- Documentation
- GitHub Actions workflow
βœ… **Do upload to HF:**
- Only what each repo type needs (data β†’ Dataset, model β†’ Model, app β†’ Space)
---
## πŸ“ž Need Help?
- **GitHub Issues**: Check your repo settings and secrets
- **HF Upload Errors**: Verify `HF_TOKEN` is set correctly
- **File Size Issues**: Check HF file size limits (usually 10GB for datasets)