Optimize Cell_Cluster_Functional_Hypothesis_Tool: use batch PubMed queries instead of per-gene queries, reduce API calls by ~5x for multi-gene clusters
Add Cell_Cluster_Functional_Hypothesis_Tool: specialized tool for generating functional state hypotheses for cell clusters with evidence-based confidence scoring
- Move formatting logic to Pubmed_Search_Tool instead of app.py
- Add formatted_output field with readable citation format including PMID
- Improve error handling with detailed error messages and suggestions
- Add more fields: pmid, authors, publication_date, journal
- Update verification logic to require both search and synthesis steps for knowledge domain
- Keep app.py minimal by checking for formatted_output field
Prioritize search tools for knowledge domain tasks
- Update planner to require using search tools (Pubmed_Search_Tool, Google_Search_Tool) FIRST to retrieve real knowledge
- Then use Generalist_Solution_Generator_Tool to synthesize search results with proper citations
- Modify tool filtering to prioritize search tools over Generalist_Solution_Generator_Tool for knowledge domain
- Ensures system uses real, verifiable information from databases rather than only LLM knowledge
Simplify loop detection and require references for knowledge tasks
Changes:
1. Cell_State_Analyzer_Single_Tool and Cell_State_Analyzer_Multi_Tool:
- Removed loss threshold early stopping condition (early_stop_loss)
- Now only uses patience-based early stopping: 5 consecutive epochs without loss decrease
- Improved logging to clearly indicate early stopping reason
2. Remove dinov3_code directory:
- All dinov3_code files removed from repository
- Model loading now uses torch.hub.load from GitHub directly
- Weights loaded from Hugging Face Hub using hf_hub_download
Early stopping behavior:
- Training stops when loss does not decrease for 5 consecutive epochs
- No longer uses loss threshold (early_stop_loss) as stopping criterion
Update early stopping to only use patience (remove loss threshold)
Changes:
1. Removed loss threshold early stopping condition from both cell_state_analyzer tools
2. Early stopping now only uses: consecutive 5 epochs without loss decrease (patience=5)
3. Unified early stopping logic and logging messages in both tools
Both tools now use consistent early stopping:
- Single stopping criterion: no improvement for 5 consecutive epochs
- Clear logging: 'No improvement for 5 consecutive epochs (loss did not decrease)'
Organize .npy mask files into unified zip archive for Downloadable Files
Problem: .npy mask files were saved individually, making download cumbersome for multiple images.
Solution:
1. Created _create_unified_masks_zip() function similar to _create_unified_crops_zip():
- Collects all .npy mask files from segmentation tools
- Organizes masks by image_id in zip structure: {image_id}/{filename}.npy
- Auto-detects mask type (organoid/cell/nuclei) from filename
- Creates unified_{mask_type}.zip file
2. Enhanced _collect_visual_outputs() to:
- Collect .npy mask files from mask_path, mask_npy_path, organoid_mask_path, cell_mask_path, nuclei_mask_path fields
- Support both single image and per_image (multi-image) structures
- Automatically create unified zip for all collected masks
- Add unified zip to Downloadable Files (replacing individual .npy files)
Benefits:
- Users download single unified zip file instead of multiple individual .npy files
- Masks organized by image_id for easy navigation
- Consistent with crops zip organization patt…
Fix .npy mask files not appearing in Downloadable Files
Problem: .npy mask files were not being collected and added to Downloadable Files because:
1. _collect_visual_outputs excluded mask_path field (line 967, 981)
2. Filter logic skipped all files with 'mask' in filename (including .npy)
3. .npy files only exist in mask_path/mask_npy_path fields, not in visual_outputs
Solution:
1. Collect .npy files from mask_path and mask_npy_path fields (for both per_image and single image cases)
2. Modified filter logic: only skip .tif mask files (for display), allow .npy files (for download)
3. Added .npy to supported file formats list
Now .npy mask files will be:
- Collected from mask_path/mask_npy_path fields
- Allowed through filter (not skipped)
- Recognized as downloadable files
- Added to Downloadable Files list
- Not added to Visual Outputs (for display)
Simplify and optimize mask loading in Single_Cell_Cropper_Tool
Problem: Single_Cell_Cropper_Tool failed to load segmentation masks saved in .npy format after segmentation tools were updated to save masks in both .tif and .npy formats.
Solution:
- Added support for loading .npy format masks using np.load()
- Priority order: .npy (preferred, preserves original array data) > .tif/.tiff (16-bit TIFF) > other formats (PNG, etc.)
- Maintains backward compatibility with existing .tif and PNG mask formats
- Ensures proper dtype handling (uint16 for label masks supporting up to 65535 labels)
This allows Single_Cell_Cropper_Tool to work with the new dual-format mask saving system where:
- .npy files are used for analysis (downstream tools use mask_path which points to .npy)
- .tif files are used for visual display in Visual Outputs
Save segmentation masks in both .tif and .npy formats
Changes:
1. All Cellpose-based segmentation tools (organoid_segmenter, cell_segmenter, nuclei_segmenter) now save masks in two formats:
- .tif format (16-bit TIFF) 鈫?added to Visual Outputs for display
- .npy format (NumPy array) 鈫?automatically added to Downloadable Files for morphological analysis
2. app.py updated to recognize .npy files as downloadable files
3. Return values:
- mask_path: points to .npy file (for downstream tools and analysis)
- mask_tif_path: points to .tif file (for visual display)
- visual_outputs: includes [overlay_path, mask_tif_path]
This enables users to:
- View mask visualizations in Visual Outputs (.tif format)
- Download raw mask arrays from Downloadable Files (.npy format) for statistical analysis
Prevent re-running Cell_State_Analyzer after Analysis_Visualizer_Tool completes
Problem: After Analysis_Visualizer_Tool completes, the system was incorrectly trying to re-run Cell_State_Analyzer_Multi_Tool, causing unnecessary tool execution.
Solution:
1. Analysis_Visualizer_Tool now declares tool chain completion:
- required_next_tools: [] (no further tools needed)
- can_terminate_after_chain: True (pipeline complete)
- Applied to both main execute() and create_comparison_chart() methods
2. Enhanced verificate_memory prompt:
- Added explicit rule: If Analysis_Visualizer_Tool has been executed 鈫?STOP (pipeline complete)
- Clarified that Analysis_Visualizer_Tool is the final step in the analysis pipeline
- Added instruction: DO NOT re-run Cell_State_Analyzer after Analysis_Visualizer_Tool completes
3. Enhanced generate_next_step logic:
- Added check for can_terminate_after_chain when no required_next_tools
- Logs when tool chain is complete to prevent further tool selection
This ensures the analysis pipeline follows the correct sequence:…
Fix visual outputs display and implement intelligent tool dependency system
Visual Outputs Fixes:
- Fix indentation error in _collect_visual_outputs (image_id extraction and label generation)
- Replace ImageProcessor.load_for_display() with Image.open() for reliable image loading
- Add else block structure to match SHAPE_260113 reference implementation
- Ensures overlay images and Analysis_Visualizer_Tool results are properly displayed
Intelligent Tool Dependency System:
- Tools now declare required_next_tools in their return results
- Planner automatically enforces tool-declared dependencies
- Cell_State_Analyzer tools declare Analysis_Visualizer_Tool as required_next_tools
- Termination signals (can_terminate_after_chain) only checked after all required_next_tools complete
- Replaces hardcoded tool chain logic with intelligent dependency management
Summary Generation:
- Ensure Summary section is always generated in generate_final_output
- Include morphological features and statistical comparisons in summary when available
- Check for termination_recommended field in tool results before calling LLM
- If tool explicitly recommends termination, return stop_signal=True immediately
- Include detailed termination reason and tool summary in analysis
- Prevents planner from continuing when tool indicates results cannot answer query
- This is a normal termination signal, not an error
Add termination signal detection for insufficient analysis results
- Detect when analysis results cannot answer group comparison queries
- Add termination_recommended and termination_reason fields to tool results
- Include detailed limitations and solution suggestions in summary
- Mark termination recommendation when all groups are 'default' (cannot perform group comparison)
- Provide clear guidance in summary for planner to identify when to terminate (not as error)
Filter crop files: only read TIFF for multi-tool, exclude overlays for single-tool
- cell_state_analyzer_multi: filter to only include TIFF format files (multi-channel crops)
- cell_state_analyzer_single: exclude overlay/viz files, only include actual crop files
- Prevents reading PNG overlay files from segmentation tools
Update tools: unify model loading, fix visual outputs, improve group handling
- cell_state_analyzer_single: align model loading with multi version (torch.hub.load from GitHub)
- cell_state_analyzer_multi: remove rigid error detection for missing groups, intelligently handle single/multi-group analysis
- cell_segmenter: fix visual_outputs to only include overlay images (exclude mask files)
- analysis_visualizer: ensure output_path is included in visual_outputs for compatibility
Fix model architecture mismatch: change backbone from dinov3_vits16 to dinov3_vitb16 to match weights
Issue:
- Generated command string referenced 'tool_name' variable inside the string
- This variable doesn't exist in the execution scope, causing NameError
Fix:
- Pre-compute is_multi_tool boolean before generating command string
- Embed the boolean value directly in the command string instead of referencing tool_name
- This ensures the condition is evaluated at command generation time, not execution time
Fix segmentation tool outputs not appearing in Visual Outputs
Issues:
1. Collection logic used 'elif' for visual_outputs, so if deliverables existed, visual_outputs were ignored
2. Mask filtering was too aggressive, potentially skipping overlay images
Fixes:
1. Changed 'elif visual_outputs' to 'if visual_outputs' - now checks both deliverables and visual_outputs
2. Improved mask filtering: only skip TIFF mask files, not PNG overlay images
3. Added debug logging to track collected files
This ensures segmentation overlay images (cell_overlay, organoid_overlay, nuclei_overlay) appear in Visual Outputs and Downloadable Files.
Issue:
- Executor was passing in_channels and selected_channels to all Cell_State_Analyzer_*_Tool
- Cell_State_Analyzer_Single_Tool.execute() does NOT accept these parameters
- This caused 'unexpected keyword argument' error
Fix:
- Only pass in_channels/selected_channels to Cell_State_Analyzer_Multi_Tool
- Cell_State_Analyzer_Single_Tool receives standard parameters only
- Single_Tool internally converts single-channel to 3-channel RGB for DINOv3, so no channel selection needed
Add segmentation tool outputs to Visual Outputs and Downloadable Files
Changed approach:
- Removed hard-coded 'DO NOT use Image_Captioner_Tool' rules
- Added intelligent planning guidelines that help LLM reason about tool selection
- Emphasize efficiency vs. value trade-offs rather than strict prohibitions
- Let LLM intelligently decide when Image_Captioner_Tool adds value vs. direct segmentation
Philosophy:
- Trust LLM intelligence for tool selection
- Provide context and reasoning guidance, not hard rules
- Balance efficiency, quality, and user experience
- Allow flexibility for edge cases where Image_Captioner_Tool might be useful
Unify ImageProcessor to use skimage.io.imread globally
Global refactoring:
- Replace tifffile.imread and PIL Image.open with skimage.io.imread in ImageData.from_path
- Align channel order detection logic with reference implementation (img.shape[0] not in (1,2,3))
- Update documentation to reflect unified loading approach
- Keep tifffile and PIL only for saving operations (not loading)
Benefits:
- Consistent image loading across all tools (matches reference implementation)
- Better compatibility with multi-channel TIFF files
- Simplified codebase with single loading method
Align MultiChannelCellCropDataset with reference implementation
Root cause analysis:
- Error 'Expected 3 but got 2' occurs during model loading, not during metadata loading
- Transformers AutoModel.from_pretrained validates input channels during initialization
- Solution: Add ignore_mismatched_sizes=True to prevent validation errors during loading
- Channel adaptation happens AFTER model loading, so validation should be disabled
This prevents transformers from throwing channel mismatch errors before we can adapt the model.
Fix channel adaptation: update config BEFORE adapting patch embedding
Critical fix:
- Update backbone.config.num_channels IMMEDIATELY after loading model (before _adapt_patch_embedding)
- This prevents transformers from validating with wrong channel count during forward pass
- Add comprehensive logging for channel adaptation process
- Improve error handling and validation checks
Root cause: Transformers models validate input channels based on config.num_channels. If config is not updated before first forward pass (even during adaptation), it throws 'Expected 3 but got 2' error.
Fix Cell_State_Analyzer_*_Tool execution: add to merge-all tools list and improve channel detection
Critical fixes:
- Add Cell_State_Analyzer_Single_Tool and Cell_State_Analyzer_Multi_Tool to TOOLS_THAT_MERGE_ALL_IMAGES list (was missing, causing per_image execution)
- Update executor special handling to include Multi and Single tools (not just Cell_State_Analyzer_Tool)
- Improve channel detection error handling and logging
- Ensure in_channels is auto-detected from crop images (not hardcoded)
Root cause: Tools were being executed per_image instead of merge-all mode, causing channel mismatch errors and incorrect workflow.
Fix tool descriptions and channel adaptation logic
