Upload folder using huggingface_hub (part 2)

backend/pipeline/vector_client.py

@@ -334,32 +334,45 @@ class PGVectorCollectionWrapper:
                                     model_name,
                                 ],
                             )
+                        elif embeddings is not None and embeddings[i]:
+                            # No `document`, but the caller supplied a
+                            # precomputed vector (exactly `VisualIndexService.
+                            # index`'s shape: it hands over vectors it computed
+                            # itself, never raw text). The `embedding(model,
+                            # doc)` derivation does not apply here -- but
+                            # writing NULL would silently discard the caller's
+                            # vector. Persist the supplied vector directly, as a
+                            # pgvector literal. Dormant on Neon/Cloud SQL today
+                            # (is_alloydb_ai_supported() is False there); this is
+                            # what makes the visual index buildable the day
+                            # someone points this at a real AlloyDB instance.
+                            vec_literal = (
+                                "["
+                                + ",".join(str(float(x)) for x in embeddings[i])
+                                + "]"
+                            )
+                            sql = """
+                                INSERT INTO animetix_vectorrecord (collection_name, item_id, embedding, metadata, document, created_at)
+                                VALUES (%s, %s, %s::vector, %s, NULL, NOW())
+                                ON CONFLICT (collection_name, item_id)
+                                DO UPDATE SET
+                                    embedding = EXCLUDED.embedding,
+                                    metadata = EXCLUDED.metadata,
+                                    document = EXCLUDED.document;
+                            """
+                            cursor.execute(
+                                sql,
+                                [
+                                    self.name,
+                                    str(item_id),
+                                    vec_literal,
+                                    json.dumps(clean_metas[i]),
+                                ],
+                            )
                         else:
-                            # Sans `document`, cette branche calcule NULL --
-                            # elle ne lit JAMAIS `embeddings` (l'embedding
-                            # vient de `embedding(model, doc)`, pas d'un
-                            # vecteur déjà calculé). Si l'appelant A fourni un
-                            # vecteur (ex. `VisualIndexService.index`, qui
-                            # n'appelle jamais `upsert_items` avec des
-                            # `documents`), l'écrire quand même en NULL le
-                            # jetterait en silence : `strict=True` ne se
-                            # déclenche jamais (aucune exception ne remonte),
-                            # et l'appelant annonce un succès pour une ligne
-                            # sans vecteur. Dormant sur Neon/Cloud SQL
-                            # aujourd'hui ; armé le jour où quelqu'un pointe
-                            # ceci sur une vraie instance AlloyDB.
-                            if embeddings is not None and embeddings[i]:
-                                raise ValueError(
-                                    f"AlloyDB AI branch: item {item_id!r} in "
-                                    f"collection {self.name!r} was given a "
-                                    "computed embedding but no `document` -- "
-                                    "this branch derives the embedding FROM "
-                                    "`documents` (`embedding(model, doc)`) and "
-                                    "silently writes NULL otherwise, discarding "
-                                    "the vector the caller already computed. "
-                                    "Refusing rather than writing a vectorless "
-                                    "row that looks like a successful upsert."
-                                )
+                            # No document AND no vector: a metadata-only row
+                            # (embedding NULL) is the honest answer -- nothing
+                            # is being discarded.
                             sql = """
                                 INSERT INTO animetix_vectorrecord (collection_name, item_id, embedding, metadata, document, created_at)
                                 VALUES (%s, %s, NULL, %s, NULL, NOW())

docs/HISTORY.md

@@ -2,6 +2,19 @@
 
 This document archives the major milestones of the project's technical evolution.
 
+## [2026-07-16] Session: Dockerfile hardening, Covertest→Postgres, coverage & fetch hygiene, script archive, BlindtestPage split, CCIP integration guard
+
+- **Dockerfile hardening (safe subset).** `Dockerfile.brain` runtime → `python:3.12-slim` (bake stages left on 3.11 to preserve the ~90-min merge cache); `Dockerfile.dataflow` drops root to a non-root `appuser` after apt/pip. Brain stays root by design (Ollama installs as root, the GPU path is untestable and CI never rebuilds the brain); the dataflow non-root drop is **unverified against a real Dataflow submission** — revert the `USER` block if the runner rejects it. The `:latest` launcher is kept (Google's recommended form, with pin-by-digest instructions in-file).
+- **Covertest — covers out of the JSON blob.** `manga_covers.json` (0.5 → ~25 MB, loaded entirely into RAM by the pgvector singleton at first call) moved to a Postgres `MangaCover` model with on-demand lookups (`sync_covers` command + migration).
+- **`finetuning_dataset` coverage 65 → 85%.** Hermetic integration tests over the language×tier matrix (anime/manga/character), language omission, Alpaca-base load exceptions, out-of-bounds noise, and multi-turn noise.
+- **Raw-`fetch()` toast hygiene.** Failure toasts harmonized on `MangaVoicePage`, `offlineLibrary` (via a `useChapterDownload` hook), and the OpenData download (`OpenDataPage` / `labService`); the three keep raw `fetch` on purpose (binary / cross-origin) but now surface an `error` toast.
+- **Frontend coverage gate raised** to match reality and block regressions: 29→38 statements / 22→28 branches / 28→34 functions / 29→39 lines.
+- **`scripts/` cleaned.** 7 unused one-offs → `scripts/archive/`, stale `test_*.pyc` purged, and a `scripts/README.md` documenting the active and archived scripts.
+- **`BlindtestPage.tsx` 753 → 324 lines**, extracting a `useBlindtestGame` hook plus `BlindtestPlayer` / `BlindtestSecretReveal` / `BlindtestBonusRecap` components. The other >500-line components (TransparencyPage, AudioLabPage, LabHubPage, SpeechToSpeechLabPage) remain and stay tracked as open.
+- **CCIP integration guard.** A new `integration`-marked test (`tests/adapters/test_ccip_vision_integration.py`) drives the real `deepghs/ccip_onnx` graph over real catalog portraits and pins the end-to-end property — 768-d, non-zero, deterministic, discriminating (mean cross-character ~0.33 ≪ self ~1.0), size-invariant, query-ranks-self — that a subtly-wrong preprocessing change would break while passing the mocked units. De-risks the phase-2 character build (which remains a prod job).
+- **AlloyDB upsert persists a supplied vector.** `PGVectorCollectionWrapper.upsert`'s AlloyDB-AI branch derives embeddings from `documents` via `embedding(model, doc)`; handed a precomputed vector with no document — exactly `VisualIndexService.index`'s shape — it used to *raise* (a guard against silently writing NULL). It now persists the supplied vector directly as a pgvector literal (`%s::vector`), so the visual index is buildable the day this points at a real AlloyDB instance. The genuine no-vector/no-document case still writes NULL. Dormant on Neon/Cloud SQL (`is_alloydb_ai_supported()` is False there).
+- **`reconcile_db` no longer issues a clean bill of health for a store it skipped.** When Neo4j was offline (the dead local URI → `driver is None`) the command skipped every graph check but still printed "[SUCCESS] All sampled items are synchronized across Django, pgvector, **and Neo4j**" — the skip went unnoticed behind a green verdict. The verdict now lists the stores actually checked and, when any was skipped, warns their synchronization is `UNVERIFIED` rather than confirmed; full SUCCESS only when every store was reached. The dead local URI itself remains an operator config action (point `NEO4J_URI` at a live instance).
+
 ## [2026-07-15] Session: Technical-debt sweep — hardcoded config & data externalized, duplicated logic unified
 
 Closure of five 🟡 items from the *audit dette 2026-07-11*.