Predictive_maintaimce / WHAT_TO_UPLOAD.md
pardhas133's picture
Upload 11 files
fc00054 verified
|
Raw History Blame Contribute Delete
8.12 kB

What to Upload: GitHub vs Hugging Face

Quick Summary

Destination What to Upload How
GitHub Everything (entire project folder) git push
HF Dataset Raw data + train/test splits Scripts auto-upload
HF Model Trained model file Scripts auto-upload
HF Space App files (code, Dockerfile, requirements) Scripts auto-upload

πŸ“¦ GitHub Repository

Upload: Everything in the mlops/ folder

What to include:

mlops/
β”œβ”€β”€ data/
β”‚   β”œβ”€β”€ engine_data.csv          βœ… Upload
β”‚   └── processed/
β”‚       β”œβ”€β”€ train.csv            βœ… Upload
β”‚       └── test.csv             βœ… Upload
β”œβ”€β”€ src/
β”‚   β”œβ”€β”€ app.py                   βœ… Upload
β”‚   β”œβ”€β”€ config.py                βœ… Upload
β”‚   β”œβ”€β”€ data_prep.py             βœ… Upload
β”‚   β”œβ”€β”€ data_register.py         βœ… Upload
β”‚   β”œβ”€β”€ train.py                 βœ… Upload
β”‚   β”œβ”€β”€ inference.py             βœ… Upload
β”‚   β”œβ”€β”€ deploy_to_hf.py         βœ… Upload
β”‚   β”œβ”€β”€ eda.py                  βœ… Upload
β”‚   β”œβ”€β”€ hf_data_utils.py        βœ… Upload
β”‚   └── hf_model_utils.py       βœ… Upload
β”œβ”€β”€ notebooks/                   βœ… Upload (if you have EDA notebooks)
β”œβ”€β”€ models/
β”‚   └── best_model.joblib        ⚠️ Optional (large file)
β”œβ”€β”€ .github/
β”‚   └── workflows/
β”‚       └── pipeline.yml         βœ… Upload (IMPORTANT!)
β”œβ”€β”€ requirements.txt             βœ… Upload
β”œβ”€β”€ Dockerfile                   βœ… Upload
β”œβ”€β”€ README.md                    βœ… Upload
└── *.md files                   βœ… Upload (documentation)

❌ DON'T upload:
β”œβ”€β”€ .venv/                       ❌ Skip (virtual environment)
β”œβ”€β”€ __pycache__/                 ❌ Skip (Python cache)
β”œβ”€β”€ mlruns/                      ❌ Skip (MLflow tracking - can be large)
└── .git/                        ❌ Skip (git metadata)

How to Upload to GitHub:

cd /Users/ananttripathi/Desktop/mlops

# Initialize git (if not already done)
git init

# Create .gitignore to exclude large/unnecessary files
cat > .gitignore << EOF
.venv/
__pycache__/
*.pyc
*.pyo
*.pyd
.Python
mlruns/
*.log
.DS_Store
EOF

# Add all files
git add .

# Commit
git commit -m "Initial commit: Predictive Maintenance MLOps Pipeline"

# Add remote (replace with your repo URL)
git remote add origin https://github.com/ananttripathi/engine-predictive-maintenance.git

# Push
git push -u origin main

πŸ€— Hugging Face - Dataset Repo

Repo: ananttripathiak/engine-maintenance-dataset

Upload: Data files only

Files uploaded automatically by scripts:

  • data/engine_data.csv β†’ Uploaded as data/engine_data.csv
  • data/processed/train.csv β†’ Uploaded as data/train.csv
  • data/processed/test.csv β†’ Uploaded as data/test.csv

How to Upload:

Option 1: Run the scripts (automatic)

# Step 1: Register raw data
python src/data_register.py

# Step 2: Prepare and upload train/test
python src/data_prep.py

Option 2: Manual upload via HF Hub


πŸ€— Hugging Face - Model Repo

Repo: ananttripathiak/engine-maintenance-model

Upload: Trained model file only

File uploaded automatically:

  • models/best_model.joblib β†’ Uploaded as model.joblib

How to Upload:

Option 1: Run the training script (automatic)

python src/train.py
# This will:
# 1. Train the model
# 2. Save to models/best_model.joblib
# 3. Upload to HF Model Repo automatically

Option 2: Manual upload via HF Hub


πŸ€— Hugging Face - Space (Streamlit App)

Repo: ananttripathiak/engine-maintenance-space

Upload: App deployment files

Files uploaded automatically by script:

βœ… src/app.py                    (Main Streamlit app)
βœ… src/inference.py             (Inference utilities)
βœ… src/config.py                (Configuration)
βœ… Dockerfile                   (Container definition)
βœ… requirements.txt             (Dependencies)
βœ… README.md                    (Documentation)
βœ… Other src/*.py files         (If needed by app)

❌ NOT uploaded (ignored):
β”œβ”€β”€ data/                       ❌ Too large
β”œβ”€β”€ mlruns/                     ❌ MLflow tracking
β”œβ”€β”€ models/                     ❌ Model is in Model Repo
β”œβ”€β”€ .github/                    ❌ GitHub-specific
└── .venv/                      ❌ Virtual environment

How to Upload:

Option 1: Run the deployment script (automatic)

python src/deploy_to_hf.py
# This will:
# 1. Create/update the HF Space
# 2. Upload all deployment files
# 3. Configure it as a Streamlit app

Option 2: Manual upload via HF Hub


πŸ“‹ Complete Upload Checklist

βœ… Step 1: GitHub (Manual - Do First)

  • Create GitHub repository
  • Add .gitignore file
  • Push entire mlops/ folder to GitHub
  • Add GitHub Secrets (HF_TOKEN, HF_DATASET_REPO, HF_MODEL_REPO, HF_SPACE_REPO)

βœ… Step 2: Hugging Face Dataset (Automatic)

  • Set HF_TOKEN environment variable
  • Run python src/data_register.py (uploads raw data)
  • Run python src/data_prep.py (uploads train/test)

βœ… Step 3: Hugging Face Model (Automatic)

  • Run python src/train.py (trains model and uploads to HF)

βœ… Step 4: Hugging Face Space (Automatic)

  • Run python src/deploy_to_hf.py (deploys app to HF Space)

βœ… Step 5: Verify


🎯 Key Differences

Aspect GitHub Hugging Face
Purpose Code repository & version control Data/Model/App hosting
What Entire project (code, data, docs) Specific artifacts (data/model/app)
How Manual git push Automatic via scripts
Size Can be large (includes everything) Optimized (only what's needed)
Access Public/Private repo Public datasets/models/spaces
CI/CD GitHub Actions workflow HF Spaces auto-deploy

πŸ’‘ Pro Tips

  1. GitHub First: Always push to GitHub first, then run HF scripts
  2. Use .gitignore: Exclude large files like mlruns/ and .venv/ from GitHub
  3. HF Scripts are Smart: They automatically create repos if they don't exist
  4. Check File Sizes: HF has file size limits, so scripts exclude large files
  5. GitHub Secrets: Store HF credentials in GitHub Secrets for CI/CD

🚨 Common Mistakes to Avoid

❌ Don't upload to GitHub:

  • Virtual environment (.venv/)
  • Large MLflow runs (mlruns/)
  • Python cache files (__pycache__/)

❌ Don't upload to HF Space:

  • Raw data files (too large)
  • Model files (use Model Repo instead)
  • MLflow tracking data

βœ… Do upload to GitHub:

  • All source code
  • Configuration files
  • Documentation
  • GitHub Actions workflow

βœ… Do upload to HF:

  • Only what each repo type needs (data β†’ Dataset, model β†’ Model, app β†’ Space)

πŸ“ž Need Help?

  • GitHub Issues: Check your repo settings and secrets
  • HF Upload Errors: Verify HF_TOKEN is set correctly
  • File Size Issues: Check HF file size limits (usually 10GB for datasets)