Refactor planner: Replace hard rules with intelligent planning guidance
Browse filesChanged approach:
- Removed hard-coded 'DO NOT use Image_Captioner_Tool' rules
- Added intelligent planning guidelines that help LLM reason about tool selection
- Emphasize efficiency vs. value trade-offs rather than strict prohibitions
- Let LLM intelligently decide when Image_Captioner_Tool adds value vs. direct segmentation
Philosophy:
- Trust LLM intelligence for tool selection
- Provide context and reasoning guidance, not hard rules
- Balance efficiency, quality, and user experience
- Allow flexibility for edge cases where Image_Captioner_Tool might be useful
- octotools/models/planner.py +14 -6
octotools/models/planner.py
CHANGED
|
@@ -204,8 +204,9 @@ CRITICAL INSTRUCTIONS:
|
|
| 204 |
- "What cell states" or "analyze cell states" β Full cell state analysis (preprocessing β segmentation β cropping β analysis β visualization)
|
| 205 |
- "Compare" or "morphology" β Full pipeline (analysis tool extracts features, visualizer displays)
|
| 206 |
- Analysis_Visualizer_Tool requires pre-computed results from Cell_State_Analyzer_*_Tool
|
| 207 |
-
3. Select minimum necessary tools
|
| 208 |
4. Do not skip analysis tool for morphology/comparison queries
|
|
|
|
| 209 |
|
| 210 |
Provide your analysis in structured format:
|
| 211 |
- Concise Summary: Brief summary of what the query asks
|
|
@@ -330,7 +331,7 @@ This system performs SINGLE-CELL level analysis. For queries involving cell stat
|
|
| 330 |
- Analysis Tools: Cell_State_Analyzer_*_Tool analyzes individual cell/organoid crops - REQUIRES Single_Cell_Cropper_Tool output
|
| 331 |
- Visualization Tool: Analysis_Visualizer_Tool ONLY visualizes pre-computed analysis results
|
| 332 |
|
| 333 |
-
**Pipeline Flow
|
| 334 |
1. Preprocessing (optional) β Image_Preprocessor_Tool
|
| 335 |
2. Segmentation β [Cell/Nuclei/Organoid_Segmenter_Tool] β produces masks
|
| 336 |
3. Cropping β Single_Cell_Cropper_Tool β REQUIRED: produces individual crop images (one per cell/organoid)
|
|
@@ -350,14 +351,21 @@ This system performs SINGLE-CELL level analysis. For queries involving cell stat
|
|
| 350 |
- Morphology/Comparison/States: Full pipeline (Segmentation β Cropping β Analysis β Visualization)
|
| 351 |
- Analysis tools analyze individual crops - cannot skip cropping step
|
| 352 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 353 |
INSTRUCTIONS:
|
| 354 |
1. Review the query analysis to understand what type of analysis is needed
|
| 355 |
2. Check what has been done (PREVIOUS STEPS) and what is still needed
|
| 356 |
3. Follow the bioimage analysis chain appropriate for the query type
|
| 357 |
-
4.
|
| 358 |
-
5.
|
| 359 |
-
6.
|
| 360 |
-
7.
|
|
|
|
| 361 |
|
| 362 |
Output format:
|
| 363 |
<justification>: Why this tool is the best next step, referencing the analysis pipeline
|
|
|
|
| 204 |
- "What cell states" or "analyze cell states" β Full cell state analysis (preprocessing β segmentation β cropping β analysis β visualization)
|
| 205 |
- "Compare" or "morphology" β Full pipeline (analysis tool extracts features, visualizer displays)
|
| 206 |
- Analysis_Visualizer_Tool requires pre-computed results from Cell_State_Analyzer_*_Tool
|
| 207 |
+
3. Select minimum necessary tools based on query requirements
|
| 208 |
4. Do not skip analysis tool for morphology/comparison queries
|
| 209 |
+
5. Use intelligent planning: Consider whether additional context (e.g., image type analysis) would help, but prioritize efficiency - if the task can be accomplished directly with specialized tools, use them directly
|
| 210 |
|
| 211 |
Provide your analysis in structured format:
|
| 212 |
- Concise Summary: Brief summary of what the query asks
|
|
|
|
| 331 |
- Analysis Tools: Cell_State_Analyzer_*_Tool analyzes individual cell/organoid crops - REQUIRES Single_Cell_Cropper_Tool output
|
| 332 |
- Visualization Tool: Analysis_Visualizer_Tool ONLY visualizes pre-computed analysis results
|
| 333 |
|
| 334 |
+
**Pipeline Flow:**
|
| 335 |
1. Preprocessing (optional) β Image_Preprocessor_Tool
|
| 336 |
2. Segmentation β [Cell/Nuclei/Organoid_Segmenter_Tool] β produces masks
|
| 337 |
3. Cropping β Single_Cell_Cropper_Tool β REQUIRED: produces individual crop images (one per cell/organoid)
|
|
|
|
| 351 |
- Morphology/Comparison/States: Full pipeline (Segmentation β Cropping β Analysis β Visualization)
|
| 352 |
- Analysis tools analyze individual crops - cannot skip cropping step
|
| 353 |
|
| 354 |
+
**Intelligent Planning Guidelines:**
|
| 355 |
+
- For counting queries: Segmentation tools (Nuclei_Segmenter_Tool, Cell_Segmenter_Tool, Organoid_Segmenter_Tool) are designed to handle counting directly. Consider whether additional image analysis would add value or if direct segmentation is more efficient.
|
| 356 |
+
- For ambiguous image types: Segmentation tools have built-in detection capabilities. Consider whether explicit image analysis would significantly improve results or if direct segmentation with auto-detection is sufficient.
|
| 357 |
+
- Tool selection should balance: (1) Task efficiency - can the task be accomplished directly? (2) Result quality - would additional context improve outcomes? (3) User experience - minimize unnecessary steps while ensuring accuracy.
|
| 358 |
+
- Image_Captioner_Tool can provide valuable context for complex or ambiguous cases, but for standard bioimage workflows, specialized tools often work more efficiently.
|
| 359 |
+
|
| 360 |
INSTRUCTIONS:
|
| 361 |
1. Review the query analysis to understand what type of analysis is needed
|
| 362 |
2. Check what has been done (PREVIOUS STEPS) and what is still needed
|
| 363 |
3. Follow the bioimage analysis chain appropriate for the query type
|
| 364 |
+
4. Use intelligent reasoning: Evaluate whether each step adds value or if a more direct approach would be better
|
| 365 |
+
5. Select ONE tool that is the logical next step in the pipeline
|
| 366 |
+
6. Ensure dependencies are satisfied (e.g., need segmentation before cropping, need cropping before analysis)
|
| 367 |
+
7. For Cell_State_Analyzer_*_Tool: Do NOT pass original images or masks - tool automatically loads crop images from Single_Cell_Cropper_Tool metadata (query_cache_dir parameter)
|
| 368 |
+
8. Formulate a clear sub-goal explaining what this tool will accomplish, referencing the correct input format
|
| 369 |
|
| 370 |
Output format:
|
| 371 |
<justification>: Why this tool is the best next step, referencing the analysis pipeline
|