Spaces:
Running on Zero
Running on Zero
Commit History
-
Centralize hardware detection and optimize T4 GPU memory management
Unverified signature- Create new mosaic.hardware module for centralized GPU detection - Set T4 concurrency limit to 1 to prevent concurrent OOM errors - Update analysis.py, model_manager.py, and ui/app.py to use centralized detection - Simplify cleanup logic: only Paladin models cleared on T4, core models persist - Remove redundant GPU cleanup code (already handled by model_cache.cleanup()) Fixes T4 memory issues by enforcing sequential processing while maintaining efficient batch processing on high-memory GPUs. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>42a4892 -
Add aggressive GPU memory cleanup for T4 instances
Unverified signature- Force CUDA synchronization before cache clearing - Add explicit garbage collection after model cleanup - Log GPU memory usage after cleanup for debugging - Prevents OOM crashes when running multiple slides consecutively - Especially important for T4 GPUs with limited VRAM (~16GB)5549e2b -
Update AEON inference to require sex parameter
Unverified signature- Remove 'Unknown' from sex choices in CLI arguments - Update docstrings to indicate sex is required - Clarify that sex must be 'Male' or 'Female' for inference0d1b788
-
Add GitHub Actions workflows and comprehensive test suite
Unverified signature- Add CI/CD workflows for tests, code quality, and Docker builds - Add comprehensive Makefile with test, lint, format, and Docker targets - Add new test files for CLI, fixtures, UI components, and UI events - Refactor batch analysis into main analysis module - Update model manager and inference modules - Add .dockerignore for cleaner Docker builds - Update dependencies in uv.lock4780d8d
-
Fix remaining hardcoded data paths after rebase
Unverified signatureFound and fixed 3 hardcoded paths that weren't using get_data_directory(): - analysis.py:468 - Paladin model map path in batch processing - aeon.py:77 - Metadata path in run_with_model function - model_manager.py:157 - Data directory in load_all_models function All paths now use get_data_directory() to properly resolve HF cache or local data directory. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>c2c8715 -
Add comprehensive logging for batch processing verification
Unverified signatureImproves logging throughout the batch processing pipeline to clearly demonstrate that the optimization is working correctly. Model Manager (src/mosaic/model_manager.py): - Add detailed GPU detection and memory reporting - Show total GPU memory and memory before/after loading each model - Log memory management strategy (T4 aggressive vs A100 caching) - Clear banner indicating models loaded ONCE per batch - Show Paladin model cache hits vs new loads with ✓ indicators - Distinguish between aggressive mode (load/free) and caching mode Batch Analysis (src/mosaic/batch_analysis.py): - Add comprehensive timing for entire batch and per-slide processing - Log batch start with clear separator - Show per-slide progress with [n/total] indicators - Track and report slide processing times (avg, min, max, total) - Calculate and display batch overhead vs processing time - Add detailed summary at end with: * Success/failure counts * Model loading time (done once) * Total batch time * Per-slid…6e06a36 -
Fix segment_tissue unpacking in batch analysis
Unverified signatureThe analyze_slide_with_models function was incorrectly calling segment_tissue and unpacking its return value. segment_tissue returns 4 values (polygon, _, coords, attrs) but we were only unpacking 2. This fix: - Converts seg_config string to config object (BiopsySegConfig, etc.) - Calls segment_tissue with proper individual parameters - Correctly unpacks all 4 return values - Uses polygon for slide mask visualization Fixes: ValueError: too many values to unpack (expected 2) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>42bcf72 -
Add batch processing optimization for slide analysis
Unverified signatureImplements batch processing to reduce model loading overhead by ~90% when processing multiple slides. Models are now loaded once per batch instead of once per slide, providing 25-45% overall speedup for multi-slide batches. New features: - ModelCache class with adaptive memory management (T4 vs A100 GPUs) - Batch coordinator that loads models once and reuses across all slides - Automatic batch mode for >1 slide in both Gradio UI and CLI - GPU type detection for memory-optimized strategies - Comprehensive test suite with unit, integration, and regression tests Implementation: - src/mosaic/model_manager.py: Model loading and caching infrastructure - src/mosaic/batch_analysis.py: Batch processing coordinator - src/mosaic/analysis.py: Batch-optimized pipeline functions - src/mosaic/inference/aeon.py: Add run_with_model() for pre-loaded models - src/mosaic/inference/paladin.py: Add run_with_models() for batch mode - src/mosaic/ui/app.py: Integrate batch mode in Gradio UI - src/mosaic/grad…0234c58
-
Fix user directory creation and hide HF Spaces logs locally
Unverified signature- Fix create_user_directory return type inconsistency that caused TypeError - Only show GPU allocation and user auth logs when running on HF Spaces - Prevents confusing log messages when running the app locally 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>07d6e0e
-
Fix circular import by moving get_data_directory to separate module
Unverified signatureMoved get_data_directory() function from gradio_app.py to a new data_directory.py module to avoid circular imports. The circular import occurred because gradio_app.py imports from ui/utils.py, which was trying to import back from gradio_app.py. Changes: - Create src/mosaic/data_directory.py with get_data_directory() and set_data_directory() functions - Update gradio_app.py to import and use set_data_directory() - Update all other modules (aeon.py, data.py, utils.py, analysis.py) to import from data_directory module instead of gradio_app 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>52f8cb9 -
Fix model file location to use HuggingFace cache directory
Unverified signatureChanged all model file references from hardcoded local data/ directory to use HuggingFace cache directory. This fixes the issue where files couldn't be found after downloading from HuggingFace Hub. Changes: - Add get_data_directory() function to gradio_app.py that checks MOSAIC_DATA_DIR environment variable first, then falls back to local data/ for development - Update gradio_app.py to download to HF cache (removed local_dir parameter) and set MOSAIC_DATA_DIR environment variable - Update all model file references in aeon.py, data.py, utils.py, and analysis.py to use get_data_directory() instead of hardcoded paths 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>751062d
-
Update src/mosaic/analysis.py
Unverified signatureCo-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>9cc95d5 -
Add Aeon model test suite and reproducibility scripts
Unverified signature- Add comprehensive test summary (AEON_TEST_SUMMARY.md) - Add test results for 3 validation slides (100% accuracy, avg 99.10% confidence) - Add reproducibility scripts: - export_aeon_checkpoint.py: Convert PyTorch Lightning checkpoints to pickle - run_aeon_tests.sh: Automated test runner with validation - verify_aeon_results.py: Result verification against ground truth - Update analysis.py to use standard aeon_model.pkl path - Update .gitignore to exclude slide images and masks Test Results: - Slide 881837 (BLCA): 98.19% confidence ✓ - Slide 744547 (HCC): 99.49% confidence ✓ - Slide 755246 (HCC): 99.61% confidence ✓ 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>0506a57 -
Add comprehensive sex and tissue site parameter support
Unverified signatureThis commit enhances the Aeon inference pipeline with complete support for sex and tissue site parameters across the entire application: - Add sex and tissue site encoding functions with CSV-based mapping - Update Aeon model to load correct target mappings from metadata - Add CLI arguments for sex and tissue site in gradio_app.py - Add validation for sex and tissue site in settings - Update dependencies: add lightning, seaborn, and statsmodels - Fix tissue site map file path references 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>a2b6947
-
Complete implementation of sex and tissue site parameters
Unverified signature- Add UI dropdowns for Sex and Tissue Site selection - Update settings dataframe to include Sex and Tissue Site columns - Pass sex and tissue_site parameters through the analysis pipeline - Convert sex string to index (0=Male, 1=Female, None=Unknown) - Convert tissue site string to index using tissue site mapping - Add get_tissue_sites() helper function in ui/utils.py - Update all inference pipeline functions to handle these parameters - Pass sex and tissue_site_idx to Aeon model for improved inference49fbf68
-
Add sex and tissue site parameters to Aeon inference
Unverified signature- Add encoding functions to data.py (encode_sex, encode_tissue_site, tissue_site_to_one_hot) - Update TileFeatureTensorDataset to optionally include SEX and TISSUE_SITE fields - Add sex and tissue_site_idx parameters to Aeon run() function - Thread sex and tissue_site through analysis pipeline (Aeon only, Paladin unchanged) - Add Sex and Tissue Site dropdowns to UI - Add validation for Sex and Tissue Site in settings CSV - Update SETTINGS_COLUMNS to include new fieldsde40714
-
Fix user detection by decoding JWT token from referer
Unverified signature24b5de2 -
Add debug logging for request object
Unverified signature6d1bfd0
-
Improve user detection: check username and auth headers
Unverified signaturecd62763 -
Add request parameter to analyze_slide function signature
Unverified signature0524123 -
Add gr.Request parameter to analyze_slide signature
Unverified signature7f1cc8e -
Add dynamic GPU duration: 60s for anonymous, 300s for logged-in users
Unverified signature- Split pipeline into _free (60s) and _pro (300s) versions - Both call same _impl function for code reuse - analyze_slide now accepts gr.Request parameter - Logged-in users get 300s (can process 10k+ tiles) - Anonymous users get 60s (limited to ~2000 tiles) - Log user status for debugging This allows both free and PRO users to use the Space0e5928b
-
Fix T4 detection by checking actual GPU name
Unverified signature- No SPACE_HARDWARE env var on HF Spaces T4 - Detect T4 by checking torch.cuda.get_device_name() - Look for 'T4' in GPU name (Tesla T4) - Add hardware logging at module load - Should now correctly use 4 workers on T4cbb7db9 -
Add T4 GPU detection and optimized settings
Unverified signature- Detect T4 hardware via SPACE_HARDWARE env var - T4 settings: batch_size=64, num_workers=4 - ZeroGPU settings: batch_size=128, num_workers=0 - Standard GPU: batch_size=64, num_workers=8 - Auto-adapts to hardware for optimal performance - Ready to test on T4 small instance18b7b6e -
Actually apply simplification: remove individual decorators, add single pipeline decorator
Unverified signature91b0515 -
Simplify: single GPU decorator on pipeline instead of 4 separate calls
Unverified signature- Remove decorators from individual functions - Add single @spaces.GPU(duration=300) on _run_inference_pipeline - Same total GPU time, much simpler code - No overhead between GPU operations - All GPU work in one allocation949282a -
Reduce GPU durations to fit 300s total limit per request
Unverified signature- CTransPath: 180s b 60s - Optimus: 300s → 120s - Aeon: 90s → 60s - Paladin: 90s → 60s - TOTAL: 300s (was 660s) ZeroGPU likely has 300s TOTAL limit per request, not per call. Duration reserves GPU time, so total must fit within limit.4d7da11 -
Remove chunking completely - doesn't work with ZeroGPU
Unverified signature- Multi-call chunking pattern doesn't work (even 1551 tiles failed) - Back to simple: one GPU call per function - Process all tiles in single 180s/300s allocation - User should limit slide size if hitting time limits - Simple, proven pattern8d94a99 -
Increase chunk sizes for better performance
Unverified signature- CTransPath: 2,000 -> 10,000 tiles per chunk (5x increase) - Optimus: 1,500 -> 10,000 tiles per chunk (6.7x increase) - Each chunk still well within 180s/300s GPU token limits - Most slides will process in 1-2 chunks now - Fewer GPU allocations = better performance80e07ea -
Implement proper multi-GPU-call chunking for ZeroGPU
Unverified signature- Each chunk gets its own @spaces.GPU decorated function call - Each call gets fresh 300s GPU token (no token sharing/expiry) - Wrapper function (no decorator) splits work and calls chunk functions - CTransPath: 2000 tiles per chunk (180s each) - Optimus: 1500 tiles per chunk (300s each) - Non-ZeroGPU: processes all tiles at once (no chunking overhead) This is the correct pattern for ZeroGPU time limits.14f8f4b -
Drastically reduce chunk sizes for ZeroGPU reliability
Unverified signature- CTransPath: 15,000 b 2,000 tiles per chunk (87% reduction) - Optimus: 10,000 → 1,500 tiles per chunk (85% reduction) - Even 1881 tiles caused token expiry, so being very conservative - Each chunk now takes ~7-10s processing, leaves huge buffer - Should prevent any token expiry issues658b7b2 -
Make all GPU memory stats collection optional with try-except
Unverified signature- Wrap all max_memory_allocated and reset_peak_memory_stats calls in try-except - Prevents any CUDA-related errors from breaking execution - Logs timing info even if memory stats fail - Applied to CTransPath, Optimus, Aeon, and Paladin functions - Ensures robustness on ZeroGPU and other GPU environmentsc6bd865 -
Fix CUDA device error by checking availability before reset
Unverified signature- Add torch.cuda.is_available() check before reset_peak_memory_stats() - Prevents RuntimeError on ZeroGPU when GPU context is released - Applied to all 4 GPU functions: CTransPath, Optimus, Aeon, Paladinaafc601 -
Implement chunked processing for ZeroGPU to prevent token expiry
Unverified signature- Split CTransPath processing into 15k tile chunks - Split Optimus processing into 10k tile chunks - Each chunk gets fresh GPU token (180s/300s limit per chunk) - Multiple smaller GPU calls instead of one large call - Prevents timeout when processing large slides - Non-ZeroGPU environments process all tiles at once (no change) This allows processing of larger slides within ZeroGPU constraints445c0ed -
Fix ZeroGPU detection to use correct env var
Unverified signature- Change from ZERO_GPU=true to SPACES_ZERO_GPU=1 - This is the actual environment variable set by HF Spaces - Now num_workers will correctly be set to 0 on ZeroGPU - Also enables batch_size=128 optimization for H100d69d3b8 -
Optimize batch sizes for H100 ZeroGPU and reduce Optimus duration
Unverified signature- Reduce Optimus GPU duration from 600s to 300s to fit ZeroGPU limits - Increase batch_size to 128 on ZeroGPU (H100) for better throughput - Apply batch_size optimization to both CTransPath and Optimus - Keep batch_size=64 for non-ZeroGPU environments - Estimated throughput: 12,000-15,000 tiles in 300s with H-optimus-03f232ad -
Fix ZeroGPU detection and increase workers for non-ZeroGPU environments
Unverified signature- Check ZERO_GPU env var instead of just spaces package presence - Automatically increase num_workers to minimum of 8 when not on ZeroGPU - Apply to all GPU operations (CTransPath, Optimus, Aeon, Paladin) - Keep num_workers=0 on ZeroGPU to avoid multiprocessing issues875e616
-
Increase Optimus GPU duration to 600s (max)
Unverified signaturea1464f6 -
Separate GPU calls for each operation in HF Spaces
Unverified signature23a1b1e -
Import spaces before CUDA packages to fix initialization error
Unverified signature849bc8d -
fix: increase zerogpu duration
Unverified signaturec1f995e
-
Set num_workers=0 for Zero GPU compatibility
copilot-swe-agent[bot] +1: raylimCo-authored-by: raylim <3074310+raylim@users.noreply.github.com>641c24a -
Separate tissue segmentation from GPU-decorated function
copilot-swe-agent[bot] +1: raylimCo-authored-by: raylim <3074310+raylim@users.noreply.github.com>02bf3db -
Add Hugging Face Spaces Zero GPU support
copilot-swe-agent[bot] +1: raylimCo-authored-by: raylim <3074310+raylim@users.noreply.github.com>2a074d9
-
Add comprehensive documentation improvements
copilot-swe-agent[bot] +1: raylim- Fix installation instructions in README (correct repo URL) - Fix command name inconsistency (mosaic_app -> mosaic) - Add detailed examples section to README - Add CSV file format documentation - Add cancer subtypes reference - Add troubleshooting section - Add advanced usage examples - Create CONTRIBUTING.md with development guidelines - Add comprehensive docstrings to all modules - Add module-level docstrings to core modules Co-authored-by: raylim <3074310+raylim@users.noreply.github.com>71ae2f0
-
Refactor Gradio UI into separate module for better readability
copilot-swe-agent[bot] +1: raylimCo-authored-by: raylim <3074310+raylim@users.noreply.github.com>b955807