Spaces:
Build error
Build error
File size: 3,113 Bytes
dcb67f2 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 | # π Phase 5: Async & Batch Processing
## π§ Why Async/Batch Processing?
Processing documents and embeddings one at a time is slow. Async/batch processing enables:
| Benefit | Explanation |
|---------|-------------|
| **Parallelism** | Process multiple items simultaneously |
| **Efficiency** | Batch API calls to reduce overhead |
| **Resilience** | Retry failed operations automatically |
| **Control** | Rate limiting prevents API throttling |
---
## π File: `src/rag/async_utils.py`
### Classes Overview
| Class | Purpose |
|-------|---------|
| `BatchResult` | Dataclass for batch operation results |
| `AsyncBatchProcessor` | Generic batch processor with concurrency |
| `AsyncDocumentProcessor` | Document pipeline processor |
| `RateLimiter` | Token bucket rate limiter |
| `AsyncRetry` | Retry decorator with backoff |
---
## π§ Class: `BatchResult`
```python
@dataclass
class BatchResult(Generic[T]):
results: List[T] # All results (may include None)
errors: List[Optional[Exception]] # Errors for each item
success_count: int
error_count: int
@property
def all_succeeded(self) -> bool: ...
def get_successful(self) -> List[T]: ...
```
---
## π§ Class: `AsyncBatchProcessor`
### Constructor
```python
def __init__(
self,
process_fn: Callable[[T], R], # Sync or async function
batch_size: int = 32,
max_concurrency: int = 4,
retry_count: int = 3,
retry_delay: float = 1.0,
):
```
### Method: `process()`
```python
async def process(
self,
items: List[T],
progress_callback: Optional[Callable[[int, int], None]] = None,
) -> BatchResult[R]:
```
**Usage:**
```python
processor = AsyncBatchProcessor(
process_fn=embed_text,
batch_size=32,
max_concurrency=4,
)
# Async usage
results = await processor.process(texts)
# Sync usage
results = processor.process_sync(texts)
print(f"Processed {results.success_count} items")
```
---
## π§ Class: `RateLimiter`
Token bucket algorithm for smooth rate limiting.
```python
limiter = RateLimiter(
requests_per_second=10.0,
burst_size=20,
)
async with limiter:
await make_api_call()
```
---
## π§ Class: `AsyncRetry`
Decorator for automatic retries with exponential backoff.
```python
@AsyncRetry(max_retries=3, base_delay=1.0)
async def fetch_data():
response = await api.get("/data")
return response
```
---
## π§ Helper Functions
### `gather_with_concurrency()`
```python
results = await gather_with_concurrency(
coros=[fetch(url) for url in urls],
max_concurrency=10,
)
```
### `run_async()`
```python
# Run async code from sync context
result = run_async(some_coroutine())
```
---
## π Summary
| Component | Purpose |
|-----------|---------|
| `BatchResult` | Structured batch output |
| `AsyncBatchProcessor` | Concurrent item processing |
| `AsyncDocumentProcessor` | Full document pipeline |
| `RateLimiter` | API rate control |
| `AsyncRetry` | Automatic retry with backoff |
| `gather_with_concurrency` | Limited parallel execution |
| `run_async` | Sync-async bridge |
|