Spaces:
Build error
Build error
docs: refine inference engine spec — fix prompt format, image pipeline, EnginePool factory
Browse files
docs/superpowers/specs/2026-06-09-local-inference-engine-design.md
CHANGED
|
@@ -97,8 +97,20 @@ class TextEngine:
|
|
| 97 |
def generate(self, texts: list[str], scenario: str, cefr_level: CEFRLevel) -> TextResult: ...
|
| 98 |
```
|
| 99 |
|
| 100 |
-
**Invocation:** `llama-cli -m <
|
| 101 |
-
The prompt
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 102 |
|
| 103 |
**No load/unload needed.** llama-cli is a stateless subprocess — it exits after generating output. No VRAM persists between calls.
|
| 104 |
|
|
@@ -120,17 +132,19 @@ class TTSEngine:
|
|
| 120 |
|
| 121 |
#### `ImageGenEngine`
|
| 122 |
|
| 123 |
-
Uses `diffusers.Flux2KleinPipeline
|
| 124 |
|
| 125 |
```python
|
| 126 |
class ImageGenEngine:
|
| 127 |
-
def __init__(self,
|
| 128 |
def _load_pipeline(self) -> None: ... # Lazy load on first use
|
| 129 |
def generate(self, prompts: list[str], output_dir: Path) -> ImageResult: ...
|
| 130 |
def unload(self) -> None: ... # del pipeline + torch.cuda.empty_cache()
|
| 131 |
```
|
| 132 |
|
| 133 |
-
**Invocation:** `pipe(prompt=prompt, num_inference_steps=.
|
|
|
|
|
|
|
| 134 |
|
| 135 |
**Lifecycle:** Same pattern as TTSEngine — lazy load, batch inference, unload.
|
| 136 |
|
|
@@ -140,22 +154,35 @@ Manages mutual exclusion between engines.
|
|
| 140 |
|
| 141 |
```python
|
| 142 |
class EnginePool:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 143 |
_instance: ClassVar[EnginePool | None] = None
|
| 144 |
|
| 145 |
-
|
|
|
|
| 146 |
|
| 147 |
def get_text_engine(self) -> TextEngine: ...
|
| 148 |
def get_tts_engine(self) -> TTSEngine: ...
|
| 149 |
def get_image_engine(self) -> ImageGenEngine: ...
|
| 150 |
|
| 151 |
def _ensure_exclusive(self, target: str) -> None:
|
| 152 |
-
"""Unload any loaded engine except the target."""
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 153 |
```
|
| 154 |
|
| 155 |
**Behavior:**
|
| 156 |
-
- `get_text_engine()` —
|
| 157 |
-
- `get_tts_engine()` — unloads
|
| 158 |
-
- `get_image_engine()` — unloads
|
| 159 |
|
| 160 |
**Thread safety:** Not required. Gradio handlers run sequentially in a single thread.
|
| 161 |
|
|
|
|
| 97 |
def generate(self, texts: list[str], scenario: str, cefr_level: CEFRLevel) -> TextResult: ...
|
| 98 |
```
|
| 99 |
|
| 100 |
+
**Invocation:** `llama-cli -m <tildeopen_gguf> -p <prompt> -n 512 --temp 0.7 -ngl 99`
|
| 101 |
+
The prompt is a structured template:
|
| 102 |
+
|
| 103 |
+
```
|
| 104 |
+
You are a translator. Translate the following {cefr_level} text into the target language.
|
| 105 |
+
Scenario: {scenario}
|
| 106 |
+
Translate these sentences, one per line, in order:
|
| 107 |
+
{text_lines}
|
| 108 |
+
Output ONLY the translations, one per line. No explanations.
|
| 109 |
+
```
|
| 110 |
+
|
| 111 |
+
llama-cli outputs plain text (one translation per line). Output is split by newlines and wrapped in `TextResult`.
|
| 112 |
+
|
| 113 |
+
**Note:** `-ngl 99` offloads all layers to GPU. TildeOpen GGUF path comes from settings.yaml (`models.tildeopen.file`).
|
| 114 |
|
| 115 |
**No load/unload needed.** llama-cli is a stateless subprocess — it exits after generating output. No VRAM persists between calls.
|
| 116 |
|
|
|
|
| 132 |
|
| 133 |
#### `ImageGenEngine`
|
| 134 |
|
| 135 |
+
Uses `diffusers.Flux2KleinPipeline.from_pretrained()` loading the base model repo (`black-forest-labs/FLUX.2-klein-4B`) from HF Hub on first access. Unloads after generation.
|
| 136 |
|
| 137 |
```python
|
| 138 |
class ImageGenEngine:
|
| 139 |
+
def __init__(self, device: str = "cuda"): ...
|
| 140 |
def _load_pipeline(self) -> None: ... # Lazy load on first use
|
| 141 |
def generate(self, prompts: list[str], output_dir: Path) -> ImageResult: ...
|
| 142 |
def unload(self) -> None: ... # del pipeline + torch.cuda.empty_cache()
|
| 143 |
```
|
| 144 |
|
| 145 |
+
**Invocation:** `pipe(prompt=prompt, num_inference_steps=28, guidance_scale=3.5)` → returns list of PIL images. Each saved as `.png`.
|
| 146 |
+
|
| 147 |
+
**Prompt generation:** Each card's prompt is constructed from its English text + translation: `{translation}. Scene: {text}. Illustrative, educational style.`
|
| 148 |
|
| 149 |
**Lifecycle:** Same pattern as TTSEngine — lazy load, batch inference, unload.
|
| 150 |
|
|
|
|
| 154 |
|
| 155 |
```python
|
| 156 |
class EnginePool:
|
| 157 |
+
"""Singleton managing mutual exclusion between inference engines.
|
| 158 |
+
|
| 159 |
+
Factory pattern via classmethod — callers use EnginePool.get(config) to
|
| 160 |
+
obtain (or create) the singleton instance.
|
| 161 |
+
"""
|
| 162 |
_instance: ClassVar[EnginePool | None] = None
|
| 163 |
|
| 164 |
+
@classmethod
|
| 165 |
+
def get(cls, config: EngineConfig) -> "EnginePool": ...
|
| 166 |
|
| 167 |
def get_text_engine(self) -> TextEngine: ...
|
| 168 |
def get_tts_engine(self) -> TTSEngine: ...
|
| 169 |
def get_image_engine(self) -> ImageGenEngine: ...
|
| 170 |
|
| 171 |
def _ensure_exclusive(self, target: str) -> None:
|
| 172 |
+
"""Unload any loaded engine except the target. Called before each get_*()."""
|
| 173 |
+
if target == "text":
|
| 174 |
+
self._unload_tts()
|
| 175 |
+
self._unload_image()
|
| 176 |
+
elif target == "tts":
|
| 177 |
+
self._unload_image()
|
| 178 |
+
elif target == "image":
|
| 179 |
+
self._unload_tts()
|
| 180 |
```
|
| 181 |
|
| 182 |
**Behavior:**
|
| 183 |
+
- `get_text_engine()` — returns a new TextEngine each time (no VRAM to free). Clears TTS and image engines.
|
| 184 |
+
- `get_tts_engine()` — unloads ImageGenEngine if active, then lazy-loads TTSEngine.
|
| 185 |
+
- `get_image_engine()` — unloads TTSEngine if active, then lazy-loads ImageGenEngine.
|
| 186 |
|
| 187 |
**Thread safety:** Not required. Gradio handlers run sequentially in a single thread.
|
| 188 |
|