Spaces:

---

pimcore

/

local-inference-service

Like 0

Running

Adjustments to the local inference service for use from Pimcore BPT: a batched,
OpenAI-shaped embedding API, a correctness fix to the embedding vectors themselves,
resilience and resource limits, and a lint/type/test gate.

Fast-forward on top of main (3 commits, 19 files, +1031/-723).

Embedding API (breaking change)

The embedding endpoints move to a single OpenAI-compatible shape, shared by text and images:

• POST /text-embedding and POST /image-embedding — the model moves out of the URL path and
  into the body.
• Request: {"model": "...", "input": "..."|[...]} — input is one value or a list (for images,
  each value is a base64-encoded image). A list is embedded in a single forward pass.
• Response: {"object": "list", "data": [{"index": n, "embedding": [...]}], "model": "..."}.

Callers of the old endpoints must be updated:

POST /image-embedding-upload/{model_name} (a Swagger-only development helper) is removed.
GET /embedding-vector-size/{model_name} is unchanged.

Embedding correctness

• All embeddings are L2-normalized now, on every code path. Normalization previously only
  happened in the CLS-token fallback, so CLIP/SigLIP models served through get_image_features /
  get_text_features returned raw features. A kNN index scoring with the default L2 metric then
  ranks partly by vector norm rather than by similarity — small-norm images float to the top of
  every query. Observed symptom: unrelated screenshots ranked #1 for "a blue sports car", with
  every hit's score collapsing to the same 1/(1+|q|^2) ~ 0.002. Normalizing is idempotent, so
  paths that already normalized are unaffected.
• SigLIP text is canonicalized (lowercase, punctuation dropped, whitespace collapsed) and
  padded to fixed max_length, matching what SigLIP's own training pipeline
  (big_vision.canonicalize_text) did to every caption. Raw-cased text is out of distribution for
  it, and dynamic padding degrades its embeddings.
• Attention-masked mean pooling for BERT-style sentence encoders (e.g. Nomic). The plain
  last_hidden_state.mean(dim=1) fallback averaged over padding tokens too.
• Nomic-vision / CLS-pooled ViTs take the CLS token.

Resilience

• A base64 input that cannot be decoded is skipped, not fatal: it is omitted from data, so
  its index is simply absent and the remaining indices do not shift.
• If the batched forward pass fails (e.g. a channel/normalization mismatch raising
  mean must have 1 elements if it is an iterable, got 3), the batch falls back to per-image
  forwards, dropping only the images that actually fail instead of returning 500 for all of them.

Resource limits

The service still has no authentication by design and is meant for a trusted network, but two
inputs could previously exhaust memory on their own:

• MaxBodySizeMiddleware rejects oversized bodies with 413 before a handler allocates for them.
  A declared Content-Length over the limit is refused up front, and the streamed bytes are
  counted as well, so an absent or understated Content-Length (chunked transfer) still trips it.
  Default 100 MB, MAX_BODY_SIZE_BYTES.
• Image.MAX_IMAGE_PIXELS caps decoding at 50 MP (MAX_IMAGE_PIXELS), with Pillow's
  decompression-bomb warning promoted to an error so the cap is exact. A tiny compressed file can
  declare enormous dimensions; decoding allocates against the declared pixel count.
• docker-compose.yaml publishes on 127.0.0.1:7860 instead of all interfaces, and sets
  HF_HUB_OFFLINE=1 / TRANSFORMERS_OFFLINE=1 so models load from the pre-populated cache volume
  only — a request naming an arbitrary model cannot trigger a download, and a slow/unreachable HF
  network no longer stalls cold starts.

README gains a "Deployment & network" section stating the trust boundary explicitly, plus the
embedding contract note.

Tooling and tests

• requirements.txt pinned to the validated versions; requirements-dev.txt adds the tooling.
• pyproject.toml configures ruff (lint + format, line length 120), mypy (files = ["src"]), and
  pytest. .pre-commit-config.yaml runs ruff + mypy per commit.
• 20 tests across 4 files, using stubbed models so they need no weights: middleware limits,
  base64 decoding and the bomb guard, per-image failure isolation, SigLIP canonicalization,
  unit-norm output from every branch, and the API envelope.
• Singleton service instances for all tasks — previously translation, classification and
  image-to-text constructed a new service object per request. Note this is consistency only, not
  a caching win for those three: ClassificationTaskService.classify() and
  TextToImageTaskService.extract() still call pipeline(...) per request, so their models are
  still loaded per request. Only the embedding services memoize, via _model_cache /
  _processor_cache.
• Dead code removed: src/image_classification.py was fully superseded by ClassificationTaskService.
• The StreamToLogger stdout/stderr hijack is gone in favour of ordinary logging at INFO.

Note on CI

.github/workflows/ci.yml runs ruff, mypy and pytest, but HF Spaces do not execute GitHub
Actions — this gate only fires if the repository is also mirrored to GitHub. Until then the same
commands are the local gate:

pip install -r requirements-dev.txt
ruff check . && ruff format --check . && mypy src && pytest

The test suite has not been run in the environment that opened this PR (it needs the torch /
transformers runtime deps), so please run it as part of review.