Takosaga commited on
Commit
2e7cabe
·
1 Parent(s): 873e40d

docs: refine inference engine spec — fix prompt format, image pipeline, EnginePool factory

Browse files
docs/superpowers/specs/2026-06-09-local-inference-engine-design.md CHANGED
@@ -97,8 +97,20 @@ class TextEngine:
97
  def generate(self, texts: list[str], scenario: str, cefr_level: CEFRLevel) -> TextResult: ...
98
  ```
99
 
100
- **Invocation:** `llama-cli -m <model> -p <prompt> -n <tokens> --temp 0.7`
101
- The prompt includes the scenario, CEFR level, and requested translations in a structured format (e.g., JSON array). Output is parsed into `TextResult`.
 
 
 
 
 
 
 
 
 
 
 
 
102
 
103
  **No load/unload needed.** llama-cli is a stateless subprocess — it exits after generating output. No VRAM persists between calls.
104
 
@@ -120,17 +132,19 @@ class TTSEngine:
120
 
121
  #### `ImageGenEngine`
122
 
123
- Uses `diffusers.Flux2KleinPipeline` with `from_single_file()` for GGUF loading. Loads model on first access, unloads after generation.
124
 
125
  ```python
126
  class ImageGenEngine:
127
- def __init__(self, gguf_path: str, device: str = "cuda"): ...
128
  def _load_pipeline(self) -> None: ... # Lazy load on first use
129
  def generate(self, prompts: list[str], output_dir: Path) -> ImageResult: ...
130
  def unload(self) -> None: ... # del pipeline + torch.cuda.empty_cache()
131
  ```
132
 
133
- **Invocation:** `pipe(prompt=prompt, num_inference_steps=...)` → returns PIL images. Each saved as `.png`.
 
 
134
 
135
  **Lifecycle:** Same pattern as TTSEngine — lazy load, batch inference, unload.
136
 
@@ -140,22 +154,35 @@ Manages mutual exclusion between engines.
140
 
141
  ```python
142
  class EnginePool:
 
 
 
 
 
143
  _instance: ClassVar[EnginePool | None] = None
144
 
145
- def __init__(self, config: EngineConfig): ...
 
146
 
147
  def get_text_engine(self) -> TextEngine: ...
148
  def get_tts_engine(self) -> TTSEngine: ...
149
  def get_image_engine(self) -> ImageGenEngine: ...
150
 
151
  def _ensure_exclusive(self, target: str) -> None:
152
- """Unload any loaded engine except the target."""
 
 
 
 
 
 
 
153
  ```
154
 
155
  **Behavior:**
156
- - `get_text_engine()` — always returns a new TextEngine (no VRAM to free). Clears TTS and image engines if active.
157
- - `get_tts_engine()` — unloads TextEngine (N/A, it's subprocess), unloads ImageGenEngine if active, then loads TTSEngine.
158
- - `get_image_engine()` — unloads TextEngine (N/A), unloads TTSEngine if active, then loads ImageGenEngine.
159
 
160
  **Thread safety:** Not required. Gradio handlers run sequentially in a single thread.
161
 
 
97
  def generate(self, texts: list[str], scenario: str, cefr_level: CEFRLevel) -> TextResult: ...
98
  ```
99
 
100
+ **Invocation:** `llama-cli -m <tildeopen_gguf> -p <prompt> -n 512 --temp 0.7 -ngl 99`
101
+ The prompt is a structured template:
102
+
103
+ ```
104
+ You are a translator. Translate the following {cefr_level} text into the target language.
105
+ Scenario: {scenario}
106
+ Translate these sentences, one per line, in order:
107
+ {text_lines}
108
+ Output ONLY the translations, one per line. No explanations.
109
+ ```
110
+
111
+ llama-cli outputs plain text (one translation per line). Output is split by newlines and wrapped in `TextResult`.
112
+
113
+ **Note:** `-ngl 99` offloads all layers to GPU. TildeOpen GGUF path comes from settings.yaml (`models.tildeopen.file`).
114
 
115
  **No load/unload needed.** llama-cli is a stateless subprocess — it exits after generating output. No VRAM persists between calls.
116
 
 
132
 
133
  #### `ImageGenEngine`
134
 
135
+ Uses `diffusers.Flux2KleinPipeline.from_pretrained()` loading the base model repo (`black-forest-labs/FLUX.2-klein-4B`) from HF Hub on first access. Unloads after generation.
136
 
137
  ```python
138
  class ImageGenEngine:
139
+ def __init__(self, device: str = "cuda"): ...
140
  def _load_pipeline(self) -> None: ... # Lazy load on first use
141
  def generate(self, prompts: list[str], output_dir: Path) -> ImageResult: ...
142
  def unload(self) -> None: ... # del pipeline + torch.cuda.empty_cache()
143
  ```
144
 
145
+ **Invocation:** `pipe(prompt=prompt, num_inference_steps=28, guidance_scale=3.5)` → returns list of PIL images. Each saved as `.png`.
146
+
147
+ **Prompt generation:** Each card's prompt is constructed from its English text + translation: `{translation}. Scene: {text}. Illustrative, educational style.`
148
 
149
  **Lifecycle:** Same pattern as TTSEngine — lazy load, batch inference, unload.
150
 
 
154
 
155
  ```python
156
  class EnginePool:
157
+ """Singleton managing mutual exclusion between inference engines.
158
+
159
+ Factory pattern via classmethod — callers use EnginePool.get(config) to
160
+ obtain (or create) the singleton instance.
161
+ """
162
  _instance: ClassVar[EnginePool | None] = None
163
 
164
+ @classmethod
165
+ def get(cls, config: EngineConfig) -> "EnginePool": ...
166
 
167
  def get_text_engine(self) -> TextEngine: ...
168
  def get_tts_engine(self) -> TTSEngine: ...
169
  def get_image_engine(self) -> ImageGenEngine: ...
170
 
171
  def _ensure_exclusive(self, target: str) -> None:
172
+ """Unload any loaded engine except the target. Called before each get_*()."""
173
+ if target == "text":
174
+ self._unload_tts()
175
+ self._unload_image()
176
+ elif target == "tts":
177
+ self._unload_image()
178
+ elif target == "image":
179
+ self._unload_tts()
180
  ```
181
 
182
  **Behavior:**
183
+ - `get_text_engine()` — returns a new TextEngine each time (no VRAM to free). Clears TTS and image engines.
184
+ - `get_tts_engine()` — unloads ImageGenEngine if active, then lazy-loads TTSEngine.
185
+ - `get_image_engine()` — unloads TTSEngine if active, then lazy-loads ImageGenEngine.
186
 
187
  **Thread safety:** Not required. Gradio handlers run sequentially in a single thread.
188