Spaces:

Text-to-Document-Generation
/

Docgenie-API

Paused

Docgenie-API / ARCHITECTURE.md

Ahadhassan-2003

deploy: update HF Space

dc4e6da 6 months ago

11 kB

# 🏗️ DocGenie Architecture & Dependency Resolution

## 📦 Package Structure

```
docgenie/                          ← Root monorepo
├── docgenie/                      ← Core package (importable)
│   ├── __init__.py               
│   ├── generation/               ← Used by API
│   │   ├── pipeline_01/
│   │   │   └── claude_batching.py  ← ClaudeBatchedClient
│   │   ├── pipeline_03/
│   │   ├── pipeline_04/
│   │   └── utils/
│   ├── evaluation/
│   └── utils/
│
├── api/                          ← API Service (imports docgenie.*)
│   ├── main.py                   from docgenie import ENV
│   ├── worker.py                 from docgenie.generation.pipeline_01...
│   ├── utils.py                  from docgenie.generation...
│   └── requirements.txt          Extra: Redis, Supabase, Google
│
├── handwriting_service/          ← GPU Service (NO docgenie imports!)
│   ├── main.py                   ✓ Self-contained
│   ├── inference.py              ✓ No external deps
│   └── models.py
│
└── WordStylist/                  ← Model code (used by handwriting)
```

## 🔗 Dependency Graph

```
┌─────────────────────────────────────────────────────────────┐
│                     API Service                              │
│  ┌──────────────────────────────────────────────────────┐   │
│  │ api/main.py                                          │   │
│  │ ↓ imports                                             │   │
│  │ api/utils.py (call_claude_api_direct)               │   │
│  └──────────────────────────────────────────────────────┘   │
│                                                              │
│  ┌──────────────────────────────────────────────────────┐   │
│  │ api/worker.py                                        │   │
│  │ ↓ imports                                             │   │
│  │ from docgenie.generation.pipeline_01.claude_batching │   │
│  │ from docgenie.generation.constants                   │   │
│  │ from docgenie.generation.pipeline_03_process_response│   │
│  │ from docgenie.generation.pipeline_04_render_pdf...   │   │
│  │ from docgenie import ENV                             │   │
│  └──────────────────────────────────────────────────────┘   │
│                           ↓                                  │
│                    REQUIRES                                  │
│  ┌──────────────────────────────────────────────────────┐   │
│  │          docgenie/ package                           │   │
│  │          (entire generation module)                   │   │
│  └──────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────┐
│              Handwriting Service                             │
│  ┌──────────────────────────────────────────────────────┐   │
│  │ handwriting_service/main.py                          │   │
│  │ ↓ imports                                             │   │
│  │ from handwriting_service.inference import ...        │   │
│  │ from handwriting_service.models import ...           │   │
│  └──────────────────────────────────────────────────────┘   │
│                           ↓                                  │
│                    REQUIRES                                  │
│  ┌──────────────────────────────────────────────────────┐   │
│  │          WordStylist/ model                          │   │
│  │          (diffusion model code)                       │   │
│  └──────────────────────────────────────────────────────┘   │
│                                                              │
│  ✓ NO docgenie imports - completely independent!            │
└─────────────────────────────────────────────────────────────┘
```

## 🐳 Docker Build Strategy

### ❌ What Doesn't Work

```dockerfile
# ❌ WRONG: Can't copy just api/ folder
FROM python:3.11
COPY api/ /app/api/              # Missing docgenie package!
RUN pip install -r requirements.txt
CMD ["uvicorn", "main:app"]     # ImportError: No module named 'docgenie'
```

### ✅ What Works

```dockerfile
# ✅ CORRECT: Copy entire monorepo
FROM python:3.11
WORKDIR /app
# Copy everything
COPY . .
# Install docgenie as package
RUN pip install -e .             # Makes docgenie.* importable
# Install API requirements
RUN pip install -r api/requirements.txt
WORKDIR /app/api
CMD ["uvicorn", "main:app"]     # ✓ docgenie imports work!
```

## 🚢 Deployment Strategy Comparison

### Option 1: Separate Deployments (❌ Won't Work)

```
API Deployment:
├── api/ folder only
└── ❌ Missing docgenie package → ImportError
Handwriting Deployment:
├── handwriting_service/ folder
└── WordStylist/
```

**Problem:** API can't find docgenie imports!

### Option 2: Monorepo Deployment (✅ Works)

```
API Deployment:
├── docgenie/ package (core)
├── api/ service (imports docgenie)
├── setup.py
└── requirements.txt
Handwriting Deployment:
├── handwriting_service/
└── WordStylist/
```

**Solution:** Deploy entire repo for API, standalone for handwriting!

## 📁 File Structure in Containers

### API Container (Railway/EC2)
```
/app/
├── docgenie/              ← Installed as Python package
│   ├── __init__.py
│   ├── generation/
│   └── utils/
├── api/                   ← Working directory
│   ├── main.py
│   ├── worker.py
│   └── utils.py
├── setup.py
└── pyproject.toml
Python can import:
✓ from docgenie.generation.pipeline_01 import ...
✓ from docgenie import ENV
```

### Handwriting Container (RunPod)
```
/app/
├── handwriting_service/
│   ├── main.py           ← No docgenie imports!
│   ├── inference.py
│   └── models.py
└── WordStylist/          ← Model code
    ├── ldm/
    └── wordstylist_inference.py
Python can import:
✓ from handwriting_service.inference import ...
✓ No docgenie dependencies needed!
```

## 🎯 Import Resolution Flow

### API Service Import Chain

1. **FastAPI starts:**
   ```python
   uvicorn main:app
   ```

2. **main.py imports utils:**
   ```python
   from api.utils import call_claude_api_direct
   ```

3. **utils.py imports docgenie:**
   ```python
   from docgenie.generation.pipeline_01.claude_batching import ClaudeBatchedClient
   ```

4. **Python looks for docgenie:**
   - Checks sys.path
   - Finds `/app` (where `pip install -e .` installed it)
   - Loads `docgenie/__init__.py`
   - ✓ Import succeeds!

### Handwriting Service Import Chain

1. **FastAPI starts:**
   ```python
   uvicorn main:app
   ```

2. **main.py imports local modules:**
   ```python
   from handwriting_service.inference import HandwritingGenerator
   ```

3. **inference.py imports WordStylist:**
   ```python
   sys.path.insert(0, str(Path(__file__).parent.parent / "WordStylist"))
   from ldm.models.diffusion.ddpm import LatentDiffusion
   ```

4. **Python loads local modules:**
   - No external package dependencies
   - ✓ Completely self-contained!

## 🔍 Verifying Imports

### Test API Imports
```bash
# Inside API container
python3 -c "from docgenie.generation.pipeline_01.claude_batching import ClaudeBatchedClient; print('✓ Import works!')"
```

### Test Handwriting Imports
```bash
# Inside handwriting container
python3 -c "from handwriting_service.inference import HandwritingGenerator; print('✓ Import works!')"
```

## 💡 Key Insights

1. **API needs monorepo:** Must deploy entire `docgenie/` folder structure
2. **Handwriting is independent:** Can deploy just `handwriting_service/` + `WordStylist/`
3. **Docker layer caching:** Install docgenie package first, then API requirements
4. **Working directory matters:** Set WORKDIR to /app/api for API service
5. **Python package installation:** `pip install -e .` makes docgenie importable globally

## 📊 Deployment Size Comparison

| Deployment | Size | Contents |
|------------|------|----------|
| API (Railway) | ~2GB | Python 3.11 + docgenie + API deps + Playwright |
| Worker (Railway) | ~2GB | Same as API (shares image) |
| Handwriting (RunPod) | ~8GB | CUDA 11.8 + PyTorch + Diffusers + WordStylist |

**Total:** ~12GB (but cached independently)

## ✅ Checklist for Successful Deployment

- [ ] Dockerfile copies **entire monorepo** for API
- [ ] `pip install -e .` runs before API requirements
- [ ] WORKDIR set to /app/api for runtime
- [ ] Handwriting Dockerfile copies only handwriting_service/ + WordStylist/
- [ ] .dockerignore excludes data/ folders (too large)
- [ ] Environment variables set in Railway/EC2
- [ ] Redis URL points to Upstash
- [ ] HANDWRITING_SERVICE_URL points to RunPod endpoint
## 🎉 Result
```
✓ API can import from docgenie package
✓ Worker can use ClaudeBatchedClient
✓ Handwriting service runs independently
✓ All services communicate via HTTP
✓ No more ImportError!
```

	# 🏗️ DocGenie Architecture & Dependency Resolution

	## 📦 Package Structure

	```
	docgenie/ ← Root monorepo
	├── docgenie/ ← Core package (importable)
	│ ├── __init__.py
	│ ├── generation/ ← Used by API
	│ │ ├── pipeline_01/
	│ │ │ └── claude_batching.py ← ClaudeBatchedClient
	│ │ ├── pipeline_03/
	│ │ ├── pipeline_04/
	│ │ └── utils/
	│ ├── evaluation/
	│ └── utils/
	│
	├── api/ ← API Service (imports docgenie.*)
	│ ├── main.py from docgenie import ENV
	│ ├── worker.py from docgenie.generation.pipeline_01...
	│ ├── utils.py from docgenie.generation...
	│ └── requirements.txt Extra: Redis, Supabase, Google
	│
	├── handwriting_service/ ← GPU Service (NO docgenie imports!)
	│ ├── main.py ✓ Self-contained
	│ ├── inference.py ✓ No external deps
	│ └── models.py
	│
	└── WordStylist/ ← Model code (used by handwriting)
	```

	## 🔗 Dependency Graph

	```
	┌─────────────────────────────────────────────────────────────┐
	│ API Service │
	│ ┌──────────────────────────────────────────────────────┐ │
	│ │ api/main.py │ │
	│ │ ↓ imports │ │
	│ │ api/utils.py (call_claude_api_direct) │ │
	│ └──────────────────────────────────────────────────────┘ │
	│ │
	│ ┌──────────────────────────────────────────────────────┐ │
	│ │ api/worker.py │ │
	│ │ ↓ imports │ │
	│ │ from docgenie.generation.pipeline_01.claude_batching │ │
	│ │ from docgenie.generation.constants │ │
	│ │ from docgenie.generation.pipeline_03_process_response│ │
	│ │ from docgenie.generation.pipeline_04_render_pdf... │ │
	│ │ from docgenie import ENV │ │
	│ └──────────────────────────────────────────────────────┘ │
	│ ↓ │
	│ REQUIRES │
	│ ┌──────────────────────────────────────────────────────┐ │
	│ │ docgenie/ package │ │
	│ │ (entire generation module) │ │
	│ └──────────────────────────────────────────────────────┘ │
	└─────────────────────────────────────────────────────────────┘

	┌─────────────────────────────────────────────────────────────┐
	│ Handwriting Service │
	│ ┌──────────────────────────────────────────────────────┐ │
	│ │ handwriting_service/main.py │ │
	│ │ ↓ imports │ │
	│ │ from handwriting_service.inference import ... │ │
	│ │ from handwriting_service.models import ... │ │
	│ └──────────────────────────────────────────────────────┘ │
	│ ↓ │
	│ REQUIRES │
	│ ┌──────────────────────────────────────────────────────┐ │
	│ │ WordStylist/ model │ │
	│ │ (diffusion model code) │ │
	│ └──────────────────────────────────────────────────────┘ │
	│ │
	│ ✓ NO docgenie imports - completely independent! │
	└─────────────────────────────────────────────────────────────┘
	```

	## 🐳 Docker Build Strategy

	### ❌ What Doesn't Work

	```dockerfile
	# ❌ WRONG: Can't copy just api/ folder
	FROM python:3.11
	COPY api/ /app/api/ # Missing docgenie package!
	RUN pip install -r requirements.txt
	CMD ["uvicorn", "main:app"] # ImportError: No module named 'docgenie'
	```

	### ✅ What Works

	```dockerfile
	# ✅ CORRECT: Copy entire monorepo
	FROM python:3.11
	WORKDIR /app

	# Copy everything
	COPY . .

	# Install docgenie as package
	RUN pip install -e . # Makes docgenie.* importable

	# Install API requirements
	RUN pip install -r api/requirements.txt

	WORKDIR /app/api
	CMD ["uvicorn", "main:app"] # ✓ docgenie imports work!
	```

	## 🚢 Deployment Strategy Comparison

	### Option 1: Separate Deployments (❌ Won't Work)

	```
	API Deployment:
	├── api/ folder only
	└── ❌ Missing docgenie package → ImportError

	Handwriting Deployment:
	├── handwriting_service/ folder
	└── WordStylist/
	```

	Problem: API can't find docgenie imports!

	### Option 2: Monorepo Deployment (✅ Works)

	```
	API Deployment:
	├── docgenie/ package (core)
	├── api/ service (imports docgenie)
	├── setup.py
	└── requirements.txt

	Handwriting Deployment:
	├── handwriting_service/
	└── WordStylist/
	```

	Solution: Deploy entire repo for API, standalone for handwriting!

	## 📁 File Structure in Containers

	### API Container (Railway/EC2)
	```
	/app/
	├── docgenie/ ← Installed as Python package
	│ ├── __init__.py
	│ ├── generation/
	│ └── utils/
	├── api/ ← Working directory
	│ ├── main.py
	│ ├── worker.py
	│ └── utils.py
	├── setup.py
	└── pyproject.toml

	Python can import:
	✓ from docgenie.generation.pipeline_01 import ...
	✓ from docgenie import ENV
	```

	### Handwriting Container (RunPod)
	```
	/app/
	├── handwriting_service/
	│ ├── main.py ← No docgenie imports!
	│ ├── inference.py
	│ └── models.py
	└── WordStylist/ ← Model code
	├── ldm/
	└── wordstylist_inference.py

	Python can import:
	✓ from handwriting_service.inference import ...
	✓ No docgenie dependencies needed!
	```

	## 🎯 Import Resolution Flow

	### API Service Import Chain

	1. FastAPI starts:
	```python
	uvicorn main:app
	```

	2. main.py imports utils:
	```python
	from api.utils import call_claude_api_direct
	```

	3. utils.py imports docgenie:
	```python
	from docgenie.generation.pipeline_01.claude_batching import ClaudeBatchedClient
	```

	4. Python looks for docgenie:
	- Checks sys.path
	- Finds `/app` (where `pip install -e .` installed it)
	- Loads `docgenie/__init__.py`
	- ✓ Import succeeds!

	### Handwriting Service Import Chain

	1. FastAPI starts:
	```python
	uvicorn main:app
	```

	2. main.py imports local modules:
	```python
	from handwriting_service.inference import HandwritingGenerator
	```

	3. inference.py imports WordStylist:
	```python
	sys.path.insert(0, str(Path(__file__).parent.parent / "WordStylist"))
	from ldm.models.diffusion.ddpm import LatentDiffusion
	```

	4. Python loads local modules:
	- No external package dependencies
	- ✓ Completely self-contained!

	## 🔍 Verifying Imports

	### Test API Imports
	```bash
	# Inside API container
	python3 -c "from docgenie.generation.pipeline_01.claude_batching import ClaudeBatchedClient; print('✓ Import works!')"
	```

	### Test Handwriting Imports
	```bash
	# Inside handwriting container
	python3 -c "from handwriting_service.inference import HandwritingGenerator; print('✓ Import works!')"
	```

	## 💡 Key Insights

	1. API needs monorepo: Must deploy entire `docgenie/` folder structure
	2. Handwriting is independent: Can deploy just `handwriting_service/` + `WordStylist/`
	3. Docker layer caching: Install docgenie package first, then API requirements
	4. Working directory matters: Set WORKDIR to /app/api for API service
	5. Python package installation: `pip install -e .` makes docgenie importable globally

	## 📊 Deployment Size Comparison

	\| Deployment \| Size \| Contents \|
	\|------------\|------\|----------\|
	\| API (Railway) \| ~2GB \| Python 3.11 + docgenie + API deps + Playwright \|
	\| Worker (Railway) \| ~2GB \| Same as API (shares image) \|
	\| Handwriting (RunPod) \| ~8GB \| CUDA 11.8 + PyTorch + Diffusers + WordStylist \|

	Total: ~12GB (but cached independently)

	## ✅ Checklist for Successful Deployment

	- [ ] Dockerfile copies entire monorepo for API
	- [ ] `pip install -e .` runs before API requirements
	- [ ] WORKDIR set to /app/api for runtime
	- [ ] Handwriting Dockerfile copies only handwriting_service/ + WordStylist/
	- [ ] .dockerignore excludes data/ folders (too large)
	- [ ] Environment variables set in Railway/EC2
	- [ ] Redis URL points to Upstash
	- [ ] HANDWRITING_SERVICE_URL points to RunPod endpoint

	## 🎉 Result

	```
	✓ API can import from docgenie package
	✓ Worker can use ClaudeBatchedClient
	✓ Handwriting service runs independently
	✓ All services communicate via HTTP
	✓ No more ImportError!
	```