| # Tool Priority System for LLM-Orchestrated Single-Cell Bioimage Analysis |
|
|
| ## Overview |
|
|
| This document describes the tool priority system implemented to ensure that the LLM-Orchestrated Single-Cell Bioimage Analysis System selects the most appropriate tools for each task, avoiding irrelevant tool calls and errors. |
|
|
| ## Architecture |
|
|
| ### 1. Tool Priority Configuration (`octotools/models/tool_priority.py`) |
| |
| A comprehensive priority system that categorizes tools based on their relevance to bioimage analysis tasks. |
| |
| #### Priority Levels |
| |
| - **HIGH (1)**: High priority tools for bioimage analysis |
| - Core image processing and segmentation: `Image_Preprocessor_Tool`, `Nuclei_Segmenter_Tool`, `Single_Cell_Cropper_Tool` |
| - Specialized analysis tools: `Cell_State_Analyzer_Tool`, `Fibroblast_Activation_Scorer_Tool`, `Analysis_Visualizer_Tool` |
|
|
| - **MEDIUM (2)**: Medium priority tools |
| - General-purpose image analysis tools: `Object_Detector_Tool`, `Advanced_Object_Detector_Tool`, `Image_Captioner_Tool` |
|
|
| - **LOW (3)**: Low priority tools (use sparingly) |
| - Utility tools: `Text_Detector_Tool` |
| - Code generation tools: `Python_Code_Generator_Tool`, `Generalist_Solution_Generator_Tool` |
|
|
| - **EXCLUDED (99)**: Tools not relevant for bioimage analysis |
| - All search tools (Google_Search_Tool, Pubmed_Search_Tool, etc.) |
| - Text extraction tools (URL_Text_Extractor_Tool) |
| - `Relevant_Patch_Zoomer_Tool` (not suitable for bioimages) |
|
|
| #### Key Features |
|
|
| 1. **Domain Detection**: Automatically detects task domain (bioimage, search, text_extraction, general) |
| 2. **Tool Filtering**: Filters out irrelevant tools based on detected domain |
| 3. **Dependency Management**: Tracks tool dependencies (e.g., Single_Cell_Cropper_Tool requires Nuclei_Segmenter_Tool) |
| 4. **Priority-Based Recommendations**: Suggests next tools based on priorities and dependencies |
|
|
| ### 2. Planner Integration (`octotools/models/planner.py`) |
|
|
| The Planner class has been enhanced to use the priority system: |
|
|
| #### Changes Made |
|
|
| 1. **Tool Priority Manager Integration**: |
| - Initializes `ToolPriorityManager` in `__init__` |
| - Tracks detected task domain |
| |
| 2. **Enhanced Query Analysis** (`analyze_query`): |
| - Detects task domain automatically |
| - Filters tools based on domain |
| - Excludes irrelevant tools from consideration |
|
|
| 3. **Improved Next Step Generation** (`generate_next_step`): |
| - Filters available tools based on domain |
| - Groups tools by priority in prompt |
| - Provides recommended next tools considering dependencies |
| - Enhanced prompt with explicit priority guidance |
|
|
| 4. **Tool Selection Validation** (`_validate_tool_selection`): |
| - Validates that selected tool is in available tools list |
| - Checks if tool is excluded for the domain |
| - Verifies tool dependencies are satisfied |
| - Warns if LOW priority code generation tools are selected when higher-priority alternatives exist |
|
|
| ## Usage |
|
|
| The system works automatically. When the Planner analyzes a query: |
|
|
| 1. **Domain Detection**: The system detects if the query is related to: |
| - Bioimage analysis (default for this system) |
| - Search tasks |
| - Text extraction |
| - General tasks |
|
|
| 2. **Tool Filtering**: Based on the detected domain: |
| - Bioimage tasks: Excludes search/text tools, includes all bioimage-relevant tools |
| - Other domains: Applies appropriate filtering |
|
|
| 3. **Priority-Based Selection**: The LLM receives: |
| - Tools organized by priority level |
| - Recommended next tools based on dependencies |
| - Clear guidance on which tools to prefer |
|
|
| 4. **Validation**: After tool selection: |
| - Validates the tool is appropriate |
| - Checks dependencies are satisfied |
| - Provides warnings for suboptimal selections |
|
|
| ## Benefits |
|
|
| 1. **Reduced Errors**: Prevents calling irrelevant tools (e.g., search tools for image analysis) |
| 2. **Better Tool Selection**: Prioritizes tools most relevant to the task |
| 3. **Dependency Management**: Ensures tools are called in the correct order |
| 4. **Extensibility**: Easy to add new tools and adjust priorities |
| 5. **Maintainability**: Centralized priority configuration |
|
|
| ## Configuration |
|
|
| To modify tool priorities or add new tools, edit `octotools/models/tool_priority.py`: |
|
|
| 1. Update `BIOIMAGE_TOOL_PRIORITIES` dictionary to set priorities |
| 2. Add tool dependencies to `TOOL_DEPENDENCIES` dictionary |
| 3. Update keywords in domain detection sets if needed |
|
|
| ## Example |
|
|
| For a bioimage analysis query like "segment the cells in this image": |
|
|
| 1. Domain detected: `bioimage` |
| 2. Tools filtered: Search/text tools excluded |
| 3. Available tools prioritized: |
| - HIGH: Image_Preprocessor_Tool, Nuclei_Segmenter_Tool, Cell_Segmenter_Tool, Organoid_Segmenter_Tool, Single_Cell_Cropper_Tool, Cell_State_Analyzer_Tool, Analysis_Visualizer_Tool |
| - MEDIUM: Object_Detector_Tool, Image_Captioner_Tool, etc. |
| - LOW: Text_Detector_Tool, Python_Code_Generator_Tool, Generalist_Solution_Generator_Tool |
| 4. LLM guided to select from HIGH priority tools first |
| 5. Tool selection validated before execution |
|
|
| ## Files Modified |
|
|
| - `octotools/models/tool_priority.py` (NEW): Priority system implementation |
| - `octotools/models/planner.py` (MODIFIED): Integrated priority system |
|
|
| ## Future Enhancements |
|
|
| - Dynamic priority adjustment based on query complexity |
| - Learning from successful tool sequences |
| - More sophisticated dependency resolution |
| - Integration with tool performance metrics |
