docs/FILE_STRUCTURE_DOCUMENTATION.md

Project File Structure and Naming Convention Documentation

Overview

This document describes the file naming and storage structure used throughout the project to ensure consistent file organization and prevent path-related errors.

Directory Structure

Base Structure

{solver_cache}/
└── {query_id}/                    # Query-specific cache directory
    └── tool_cache/                # All tool outputs go here
        ├── {group}/               # Group subdirectories (created by some tools)
        │   └── {files}            # Group-specific files
        └── {files}                # Direct tool outputs (metadata, etc.)

Query Cache Directory

• Location: {solver_cache}/{query_id}/
• Purpose: Root directory for all files related to a specific query
• Example: /tmp/solver_cache/2d1d6a184495417fa2294fb620d903cb/

Tool Cache Directory

• Location: {query_cache_dir}/tool_cache/
• Purpose: All tool outputs are stored here
• Example: /tmp/solver_cache/2d1d6a184495417fa2294fb620d903cb/tool_cache/

Tool-Specific File Storage

1. Image_Preprocessor_Tool

• Output Location: {tool_cache_dir}/{group}/
• File Naming: {image_identifier}_{group}_processed.png
• Example: tool_cache/default/A1_02_1_1_Phase Contrast_001_default_processed.png
• Note: Creates group subdirectories

2. Nuclei_Segmenter_Tool / Cell_Segmenter_Tool

• Output Location: {tool_cache_dir}/
• File Naming:
  • nuclei_mask_{image_identifier}.tif (for Nuclei_Segmenter_Tool)
  • cell_mask_{image_identifier}.tif (for Cell_Segmenter_Tool)
• Example: tool_cache/nuclei_mask_c3e4dac8a1a740baa9cf843ce5873570.tif

3. Single_Cell_Cropper_Tool

• Output Location: {tool_cache_dir}/
• File Naming:
  • Cell crops: cell_{idx:04d}_crop.{format}
  • Cell masks: cell_{idx:04d}_mask.{format}
  • Metadata: cell_crops_metadata_{uuid8}.json ⚠️ CRITICAL
• Metadata Location: {tool_cache_dir}/cell_crops_metadata_*.json
• Example: tool_cache/cell_crops_metadata_a1b2c3d4.json
• Note:
  • Metadata is saved directly in tool_cache_dir, NOT in group subdirectories
  • Cell crops are also saved directly in tool_cache_dir
  • The group parameter is stored in metadata, not used for directory structure

4. Cell_State_Analyzer_Tool

• Input: Reads metadata from {tool_cache_dir}/cell_crops_metadata_*.json
• Search Strategy:
  1. First checks {tool_cache_dir}/ directly
  2. Then checks all subdirectories in {tool_cache_dir}/ (including group subdirectories)
  3. Merges all found metadata files for multi-image processing
• Output Location: {tool_cache_dir}/
• File Naming: adata_{timestamp}.h5ad, cell_metadata_{timestamp}.csv

Critical Path Handling Rules

Executor Path Management

1. query_cache_dir: Always the parent directory (without 'tool_cache')

   • Example: /tmp/solver_cache/2d1d6a184495417fa2294fb620d903cb
   • Normalized in set_query_cache_dir() to remove 'tool_cache' suffix if present

2. tool_cache_dir: Always {query_cache_dir}/tool_cache

   • Example: /tmp/solver_cache/2d1d6a184495417fa2294fb620d903cb/tool_cache
   • Set via tool.set_custom_output_dir(self.tool_cache_dir) in executor

Single_Cell_Cropper_Tool Path Handling

1. Receives query_cache_dir parameter (parent directory)
2. Creates tool_cache_dir = os.path.join(query_cache_dir, "tool_cache")
3. Saves metadata to: {tool_cache_dir}/cell_crops_metadata_{uuid}.json
4. Does NOT create group subdirectories for metadata

Cell_State_Analyzer_Tool Path Handling

1. Receives query_cache_dir parameter (parent directory)
2. Searches in:
   • {query_cache_dir}/tool_cache/ (primary location)
   • All subdirectories of {query_cache_dir}/tool_cache/ (including group subdirectories)
3. Merges all found metadata files

Common Issues and Solutions

Issue 1: Metadata Not Found

Symptom: No metadata files found. Searched in: {path}

Possible Causes:

1. Single_Cell_Cropper_Tool not executed before Cell_State_Analyzer_Tool
2. Metadata saved in group subdirectory but Cell_State_Analyzer_Tool only checking root
3. Path mismatch between save and load locations

Solution:

• Updated _load_cell_data_from_metadata() to search in all subdirectories
• Added comprehensive logging to track file locations

Issue 2: Path Normalization

Symptom: Path contains 'tool_cache' when it shouldn't

Solution:

• set_query_cache_dir() normalizes paths to ensure query_cache_dir is always parent directory
• Command generation uses normalized paths

Best Practices

1. Always use query_cache_dir parameter (parent directory) when calling tools
2. Never hardcode paths - use os.path.join() for cross-platform compatibility
3. Normalize paths before comparison or storage
4. Log paths during save and load operations for debugging
5. Check subdirectories when searching for files that might be in group subdirectories

File Naming Conventions

Metadata Files

• Pattern: {tool_name}_metadata_{uuid8}.json
• Example: cell_crops_metadata_a1b2c3d4.json
• UUID: 8-character hexadecimal string

Image Files

• Pattern: {prefix}_{identifier}_{suffix}.{ext}
• Examples:
  • nuclei_mask_c3e4dac8a1a740baa9cf843ce5873570.tif
  • A1_02_1_1_Phase Contrast_001_default_processed.png
  • cell_0001_crop.png

Data Files

• Pattern: {type}_{timestamp}.{ext}
• Examples:
  • adata_20240104_123456.h5ad
  • cell_metadata_20240104_123456.csv

Summary

Key Points:

1. All tool outputs go to {query_cache_dir}/tool_cache/
2. Some tools (Image_Preprocessor_Tool) create group subdirectories
3. Single_Cell_Cropper_Tool saves metadata directly in tool_cache_dir, NOT in subdirectories
4. Cell_State_Analyzer_Tool searches both root and subdirectories to find metadata
5. Path normalization ensures consistent directory structure
6. Comprehensive logging helps debug path-related issues