RAG-Documind-AI / docs /explanations /phase5 /async_processing.md
AIenthusSP's picture
feat: add async/batch processing utilities
dcb67f2
|
Raw History Blame Contribute Delete
3.11 kB

πŸš€ Phase 5: Async & Batch Processing

🧠 Why Async/Batch Processing?

Processing documents and embeddings one at a time is slow. Async/batch processing enables:

Benefit Explanation
Parallelism Process multiple items simultaneously
Efficiency Batch API calls to reduce overhead
Resilience Retry failed operations automatically
Control Rate limiting prevents API throttling

πŸ“ File: src/rag/async_utils.py

Classes Overview

Class Purpose
BatchResult Dataclass for batch operation results
AsyncBatchProcessor Generic batch processor with concurrency
AsyncDocumentProcessor Document pipeline processor
RateLimiter Token bucket rate limiter
AsyncRetry Retry decorator with backoff

πŸ”§ Class: BatchResult

@dataclass
class BatchResult(Generic[T]):
    results: List[T]                    # All results (may include None)
    errors: List[Optional[Exception]]   # Errors for each item
    success_count: int
    error_count: int
    
    @property
    def all_succeeded(self) -> bool: ...
    
    def get_successful(self) -> List[T]: ...

πŸ”§ Class: AsyncBatchProcessor

Constructor

def __init__(
    self,
    process_fn: Callable[[T], R],  # Sync or async function
    batch_size: int = 32,
    max_concurrency: int = 4,
    retry_count: int = 3,
    retry_delay: float = 1.0,
):

Method: process()

async def process(
    self,
    items: List[T],
    progress_callback: Optional[Callable[[int, int], None]] = None,
) -> BatchResult[R]:

Usage:

processor = AsyncBatchProcessor(
    process_fn=embed_text,
    batch_size=32,
    max_concurrency=4,
)

# Async usage
results = await processor.process(texts)

# Sync usage
results = processor.process_sync(texts)

print(f"Processed {results.success_count} items")

πŸ”§ Class: RateLimiter

Token bucket algorithm for smooth rate limiting.

limiter = RateLimiter(
    requests_per_second=10.0,
    burst_size=20,
)

async with limiter:
    await make_api_call()

πŸ”§ Class: AsyncRetry

Decorator for automatic retries with exponential backoff.

@AsyncRetry(max_retries=3, base_delay=1.0)
async def fetch_data():
    response = await api.get("/data")
    return response

πŸ”§ Helper Functions

gather_with_concurrency()

results = await gather_with_concurrency(
    coros=[fetch(url) for url in urls],
    max_concurrency=10,
)

run_async()

# Run async code from sync context
result = run_async(some_coroutine())

πŸ“‹ Summary

Component Purpose
BatchResult Structured batch output
AsyncBatchProcessor Concurrent item processing
AsyncDocumentProcessor Full document pipeline
RateLimiter API rate control
AsyncRetry Automatic retry with backoff
gather_with_concurrency Limited parallel execution
run_async Sync-async bridge