docs/TOOL_PRIORITY_SYSTEM.md

Tool Priority System for LLM-Orchestrated Single-Cell Bioimage Analysis

Overview

This document describes the tool priority system implemented to ensure that the LLM-Orchestrated Single-Cell Bioimage Analysis System selects the most appropriate tools for each task, avoiding irrelevant tool calls and errors.

Architecture

1. Tool Priority Configuration (octotools/models/tool_priority.py)

A comprehensive priority system that categorizes tools based on their relevance to bioimage analysis tasks.

Priority Levels

• HIGH (1): High priority tools for bioimage analysis

  • Core image processing and segmentation: Image_Preprocessor_Tool, Nuclei_Segmenter_Tool, Single_Cell_Cropper_Tool
  • Specialized analysis tools: Cell_State_Analyzer_Tool, Fibroblast_Activation_Scorer_Tool, Analysis_Visualizer_Tool

• MEDIUM (2): Medium priority tools

  • General-purpose image analysis tools: Object_Detector_Tool, Advanced_Object_Detector_Tool, Image_Captioner_Tool

• LOW (3): Low priority tools (use sparingly)

  • Utility tools: Text_Detector_Tool
  • Code generation tools: Python_Code_Generator_Tool, Generalist_Solution_Generator_Tool

• EXCLUDED (99): Tools not relevant for bioimage analysis

  • All search tools (Google_Search_Tool, Pubmed_Search_Tool, etc.)
  • Text extraction tools (URL_Text_Extractor_Tool)
  • Relevant_Patch_Zoomer_Tool (not suitable for bioimages)

Key Features

1. Domain Detection: Automatically detects task domain (bioimage, search, text_extraction, general)
2. Tool Filtering: Filters out irrelevant tools based on detected domain
3. Dependency Management: Tracks tool dependencies (e.g., Single_Cell_Cropper_Tool requires Nuclei_Segmenter_Tool)
4. Priority-Based Recommendations: Suggests next tools based on priorities and dependencies

2. Planner Integration (octotools/models/planner.py)

The Planner class has been enhanced to use the priority system:

Changes Made

1. Tool Priority Manager Integration:

   • Initializes ToolPriorityManager in __init__
   • Tracks detected task domain

2. Enhanced Query Analysis (analyze_query):

   • Detects task domain automatically
   • Filters tools based on domain
   • Excludes irrelevant tools from consideration

3. Improved Next Step Generation (generate_next_step):

   • Filters available tools based on domain
   • Groups tools by priority in prompt
   • Provides recommended next tools considering dependencies
   • Enhanced prompt with explicit priority guidance

4. Tool Selection Validation (_validate_tool_selection):

   • Validates that selected tool is in available tools list
   • Checks if tool is excluded for the domain
   • Verifies tool dependencies are satisfied
   • Warns if LOW priority code generation tools are selected when higher-priority alternatives exist

Usage

The system works automatically. When the Planner analyzes a query:

1. Domain Detection: The system detects if the query is related to:

   • Bioimage analysis (default for this system)
   • Search tasks
   • Text extraction
   • General tasks

2. Tool Filtering: Based on the detected domain:

   • Bioimage tasks: Excludes search/text tools, includes all bioimage-relevant tools
   • Other domains: Applies appropriate filtering

3. Priority-Based Selection: The LLM receives:

   • Tools organized by priority level
   • Recommended next tools based on dependencies
   • Clear guidance on which tools to prefer

4. Validation: After tool selection:

   • Validates the tool is appropriate
   • Checks dependencies are satisfied
   • Provides warnings for suboptimal selections

Benefits

1. Reduced Errors: Prevents calling irrelevant tools (e.g., search tools for image analysis)
2. Better Tool Selection: Prioritizes tools most relevant to the task
3. Dependency Management: Ensures tools are called in the correct order
4. Extensibility: Easy to add new tools and adjust priorities
5. Maintainability: Centralized priority configuration

Configuration

To modify tool priorities or add new tools, edit octotools/models/tool_priority.py:

1. Update BIOIMAGE_TOOL_PRIORITIES dictionary to set priorities
2. Add tool dependencies to TOOL_DEPENDENCIES dictionary
3. Update keywords in domain detection sets if needed

Example

For a bioimage analysis query like "segment the cells in this image":

1. Domain detected: bioimage
2. Tools filtered: Search/text tools excluded
3. Available tools prioritized:
   • HIGH: Image_Preprocessor_Tool, Nuclei_Segmenter_Tool, Cell_Segmenter_Tool, Organoid_Segmenter_Tool, Single_Cell_Cropper_Tool, Cell_State_Analyzer_Tool, Analysis_Visualizer_Tool
   • MEDIUM: Object_Detector_Tool, Image_Captioner_Tool, etc.
   • LOW: Text_Detector_Tool, Python_Code_Generator_Tool, Generalist_Solution_Generator_Tool
4. LLM guided to select from HIGH priority tools first
5. Tool selection validated before execution

Files Modified

• octotools/models/tool_priority.py (NEW): Priority system implementation
• octotools/models/planner.py (MODIFIED): Integrated priority system

Future Enhancements

• Dynamic priority adjustment based on query complexity
• Learning from successful tool sequences
• More sophisticated dependency resolution
• Integration with tool performance metrics