3v324v23 commited on
Commit
0258fef
Β·
1 Parent(s): 9019a9d

Refactor planner: Replace hard rules with intelligent planning guidance

Browse files

Changed approach:
- Removed hard-coded 'DO NOT use Image_Captioner_Tool' rules
- Added intelligent planning guidelines that help LLM reason about tool selection
- Emphasize efficiency vs. value trade-offs rather than strict prohibitions
- Let LLM intelligently decide when Image_Captioner_Tool adds value vs. direct segmentation

Philosophy:
- Trust LLM intelligence for tool selection
- Provide context and reasoning guidance, not hard rules
- Balance efficiency, quality, and user experience
- Allow flexibility for edge cases where Image_Captioner_Tool might be useful

Files changed (1) hide show
  1. octotools/models/planner.py +14 -6
octotools/models/planner.py CHANGED
@@ -204,8 +204,9 @@ CRITICAL INSTRUCTIONS:
204
  - "What cell states" or "analyze cell states" β†’ Full cell state analysis (preprocessing β†’ segmentation β†’ cropping β†’ analysis β†’ visualization)
205
  - "Compare" or "morphology" β†’ Full pipeline (analysis tool extracts features, visualizer displays)
206
  - Analysis_Visualizer_Tool requires pre-computed results from Cell_State_Analyzer_*_Tool
207
- 3. Select minimum necessary tools
208
  4. Do not skip analysis tool for morphology/comparison queries
 
209
 
210
  Provide your analysis in structured format:
211
  - Concise Summary: Brief summary of what the query asks
@@ -330,7 +331,7 @@ This system performs SINGLE-CELL level analysis. For queries involving cell stat
330
  - Analysis Tools: Cell_State_Analyzer_*_Tool analyzes individual cell/organoid crops - REQUIRES Single_Cell_Cropper_Tool output
331
  - Visualization Tool: Analysis_Visualizer_Tool ONLY visualizes pre-computed analysis results
332
 
333
- **Pipeline Flow (Cell/Organoid State Analysis):**
334
  1. Preprocessing (optional) β†’ Image_Preprocessor_Tool
335
  2. Segmentation β†’ [Cell/Nuclei/Organoid_Segmenter_Tool] β†’ produces masks
336
  3. Cropping β†’ Single_Cell_Cropper_Tool β†’ REQUIRED: produces individual crop images (one per cell/organoid)
@@ -350,14 +351,21 @@ This system performs SINGLE-CELL level analysis. For queries involving cell stat
350
  - Morphology/Comparison/States: Full pipeline (Segmentation β†’ Cropping β†’ Analysis β†’ Visualization)
351
  - Analysis tools analyze individual crops - cannot skip cropping step
352
 
 
 
 
 
 
 
353
  INSTRUCTIONS:
354
  1. Review the query analysis to understand what type of analysis is needed
355
  2. Check what has been done (PREVIOUS STEPS) and what is still needed
356
  3. Follow the bioimage analysis chain appropriate for the query type
357
- 4. Select ONE tool that is the logical next step in the pipeline
358
- 5. Ensure dependencies are satisfied (e.g., need segmentation before cropping, need cropping before analysis)
359
- 6. For Cell_State_Analyzer_*_Tool: Do NOT pass original images or masks - tool automatically loads crop images from Single_Cell_Cropper_Tool metadata (query_cache_dir parameter)
360
- 7. Formulate a clear sub-goal explaining what this tool will accomplish, referencing the correct input format
 
361
 
362
  Output format:
363
  <justification>: Why this tool is the best next step, referencing the analysis pipeline
 
204
  - "What cell states" or "analyze cell states" β†’ Full cell state analysis (preprocessing β†’ segmentation β†’ cropping β†’ analysis β†’ visualization)
205
  - "Compare" or "morphology" β†’ Full pipeline (analysis tool extracts features, visualizer displays)
206
  - Analysis_Visualizer_Tool requires pre-computed results from Cell_State_Analyzer_*_Tool
207
+ 3. Select minimum necessary tools based on query requirements
208
  4. Do not skip analysis tool for morphology/comparison queries
209
+ 5. Use intelligent planning: Consider whether additional context (e.g., image type analysis) would help, but prioritize efficiency - if the task can be accomplished directly with specialized tools, use them directly
210
 
211
  Provide your analysis in structured format:
212
  - Concise Summary: Brief summary of what the query asks
 
331
  - Analysis Tools: Cell_State_Analyzer_*_Tool analyzes individual cell/organoid crops - REQUIRES Single_Cell_Cropper_Tool output
332
  - Visualization Tool: Analysis_Visualizer_Tool ONLY visualizes pre-computed analysis results
333
 
334
+ **Pipeline Flow:**
335
  1. Preprocessing (optional) β†’ Image_Preprocessor_Tool
336
  2. Segmentation β†’ [Cell/Nuclei/Organoid_Segmenter_Tool] β†’ produces masks
337
  3. Cropping β†’ Single_Cell_Cropper_Tool β†’ REQUIRED: produces individual crop images (one per cell/organoid)
 
351
  - Morphology/Comparison/States: Full pipeline (Segmentation β†’ Cropping β†’ Analysis β†’ Visualization)
352
  - Analysis tools analyze individual crops - cannot skip cropping step
353
 
354
+ **Intelligent Planning Guidelines:**
355
+ - For counting queries: Segmentation tools (Nuclei_Segmenter_Tool, Cell_Segmenter_Tool, Organoid_Segmenter_Tool) are designed to handle counting directly. Consider whether additional image analysis would add value or if direct segmentation is more efficient.
356
+ - For ambiguous image types: Segmentation tools have built-in detection capabilities. Consider whether explicit image analysis would significantly improve results or if direct segmentation with auto-detection is sufficient.
357
+ - Tool selection should balance: (1) Task efficiency - can the task be accomplished directly? (2) Result quality - would additional context improve outcomes? (3) User experience - minimize unnecessary steps while ensuring accuracy.
358
+ - Image_Captioner_Tool can provide valuable context for complex or ambiguous cases, but for standard bioimage workflows, specialized tools often work more efficiently.
359
+
360
  INSTRUCTIONS:
361
  1. Review the query analysis to understand what type of analysis is needed
362
  2. Check what has been done (PREVIOUS STEPS) and what is still needed
363
  3. Follow the bioimage analysis chain appropriate for the query type
364
+ 4. Use intelligent reasoning: Evaluate whether each step adds value or if a more direct approach would be better
365
+ 5. Select ONE tool that is the logical next step in the pipeline
366
+ 6. Ensure dependencies are satisfied (e.g., need segmentation before cropping, need cropping before analysis)
367
+ 7. For Cell_State_Analyzer_*_Tool: Do NOT pass original images or masks - tool automatically loads crop images from Single_Cell_Cropper_Tool metadata (query_cache_dir parameter)
368
+ 8. Formulate a clear sub-goal explaining what this tool will accomplish, referencing the correct input format
369
 
370
  Output format:
371
  <justification>: Why this tool is the best next step, referencing the analysis pipeline