README.md

metadata

title: Vectorization API
emoji: 🧠
colorFrom: blue
colorTo: indigo
sdk: docker
app_port: 7860
pinned: false

Vectorization API — configurable resident models

CPU-only FastAPI inference service. Only selected models load during startup, then remain resident. Inference never loads model weights. No search, Qdrant, Manticore, reranking or Entity Linking.

Task	Model	Max tokens	Output
Standard	intfloat/multilingual-e5-small	512	384D
Literature	Qwen/Qwen3-Embedding-0.6B	32768	1024D
ColBERT	lightonai/mLateOn	8192	128D per token
Triple retrieval	BAAI/bge-m3	8192 native, 2048 server default	dense 1024D + sparse lexical + ColBERT 1024D/token

Runtime model selection

Set ENABLED_MODELS in Hugging Face Space Settings → Variables and secrets (or as a container environment variable). Comma-separated names:

ENABLED_MODELS=e5,bge-m3

Supported: e5 (384D), qwen (1024D), colbert (mLateOn 128D), bge-m3 (1024D dense/sparse/ColBERT). For only BGE-M3 use ENABLED_MODELS=bge-m3; for all four, use ENABLED_MODELS=e5,qwen,colbert,bge-m3.

When omitted, all four load (previous default). Old BGE_M3_ENABLED=0 is still honored only if ENABLED_MODELS is unset. With an explicit ENABLED_MODELS, the list takes precedence. Empty, duplicate, or unknown names fail startup with an explanatory error; at least one model is required. Changing the environment requires a Space restart.

Disabled models are not instantiated or kept in RAM. Endpoints remain defined for compatibility but return HTTP 503 if their model is disabled. GET /models and GET /openai/v1/models show only loaded models; GET /health includes loaded_models. The legacy GET /model is E5-only and returns 503 if E5 is not loaded, as does an /embed request without a model (which defaults to E5).

Scope: this controls runtime loading and memory. The Docker build still downloads all four checkpoints and performs the mLateOn build preflight, even if a model is disabled for runtime. Build-time selection is separate.

BGE-M3: triple-vector embedding

The official FlagEmbedding.BGEM3FlagModel supports dense, sparse and ColBERT projections in one forward pass. By default the API computes only dense + sparse, skipping the ColBERT projection and its large output. Set return_colbert: true in the request to compute and return all three. These BGE-M3 ColBERT vectors are 1024D, unrelated to the existing 128D lightonai/mLateOn vectors. Do not store them in the same named Qdrant vector field.

POST /bge-m3/encode requires Authorization: Bearer <API_KEY>:

curl -sS https://smartdataalex-jupyter.hf.space/bge-m3/encode \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"input":["What is RAG?", "מהי שליפה סמנטית?"],"input_type":"document","return_colbert":false}'

Response contains model: "BAAI/bge-m3", dense_dimensions: 1024, colbert_dimensions: 1024, data[] entries with index, dense (1024 floats), sparse.indices (integer tokenizer IDs), sparse.values (learned positive lexical weights) and tokens_count. Only when return_colbert: true, each item additionally contains colbert: a variable-length array of 1024D token embeddings. By default return_colbert: false, and the projection is not computed. The sparse indices/values structure is directly suitable for Qdrant sparse vectors. A single request accepts 1–8 texts; inference is serial within the shared CPU inference gate. Query/document modes require no manual instruction prefix for BGE-M3.

BGE-M3 loading is controlled by ENABLED_MODELS (with legacy BGE_M3_ENABLED fallback when the new variable is absent). Model checkpoint download is pinned. BGE_M3_MAX_TOKENS=2048 is the initial operational cap on our shared 16-GiB CPU server; the underlying model supports 8192. To use larger contexts, increase that variable (maximum 8192) after checking CPU inference memory and latency. Overlong texts produce HTTP 422, not silent truncation. Max total tokens per request: 8192. No change to existing OpenAI-compatible/Voyage/ColBERT response contracts.

BGE-M3 itself does not compress or quantize its ColBERT output. Qdrant supports storage compression for multivectors through datatype: turbo4 (approximately 8x smaller vs float32, requires a compatible Qdrant version). Set hnsw_config.m=0 for ColBERT reranking fields. These are Qdrant collection settings, not parameters for the BGE-M3 inference endpoint; transfer of raw ColBERT JSON is still large.

API

Legacy endpoints require X-API-Key; Voyage-compatible /v1/embeddings uses Authorization: Bearer <API_KEY>. Public endpoints: /health, /docs and /openapi.json.

• POST /embed: {"text":"Your text","type":"query"} defaults to E5. The optional model accepts e5, qwen, or the corresponding full model name. The existing E5 response fields are unchanged: text, type, tokens_count, embedding_model, dimensions, embedding.
• POST /colbert/encode: {"text":"Your text","type":"passage"} returns embedding_model, tokens_count, dimensions and token_embeddings (matrix of individual 128D vectors).
• GET /models: readiness, dimensions, limits, normalization method, current/peak RSS and startup RAM readings for all models.
• GET /model: original E5-only metadata contract.
• GET /health: returns 200 when all configured models have completed startup.

Example (replace URL and key):

curl -sS -X POST 'http://localhost:7860/embed' \
  -H 'X-API-Key: YOUR_API_KEY' -H 'Content-Type: application/json' \
  -d '{"text":"What does the author mean?","type":"query","model":"qwen"}'

curl -sS -X POST 'http://localhost:7860/colbert/encode' \
  -H 'X-API-Key: YOUR_API_KEY' -H 'Content-Type: application/json' \
  -d '{"text":"שלום עולם","type":"passage"}'

E5 keeps query: / passage: prefixes, attention-mask-aware mean pooling, no extra L2 normalization. Qwen uses its official query instruction + last-token pooling + L2 normalization; documents have no instruction. PyLate mLateOn handles query and passage separately, retaining token vectors and their model-native normalization. Token limits include model markers and special tokens. The legacy API rejects overlong inputs (HTTP 422), while E5 Voyage and OpenAI-compatible endpoints truncate inputs exceeding 512 tokens, logging affected batch indices. A single CPU inference runs at a time; up to 16 additional HTTP requests wait for a free inference slot (40 seconds by default). A full queue or expired wait returns HTTP 429 with Retry-After. Configure INFERENCE_MAX_WAITERS and INFERENCE_WAIT_SECONDS as needed. Dify's OpenAI-compatible embedding provider uses a 60-second HTTP timeout, so keep the server's wait timeout below that.

OpenAI-compatible API (separate from Voyage)

OpenAI SDK-compatible base URL: https://YOUR-SPACE.hf.space/openai/v1. This separate namespace is necessary because the existing Voyage-compatible /v1/embeddings endpoint is already in production and must retain its default input_type handling. Point OpenAI-compatible custom-provider integrations to the base URL ending in /openai/v1, not to the Voyage /v1 URL.

• POST /openai/v1/embeddings: accepts model, input (string or string array), optional dimensions, encoding_format (float or base64) and user. Also accepts and ignores the non-standard boolean drop_params sent by some OpenAI-compatible connection checks; other unknown fields remain errors. Responds in the OpenAI data[].embedding / usage.prompt_tokens / usage.total_tokens format. Only fixed native dimensions are supported.
• GET /openai/v1/models: dense model aliases for provider selection.
• Bearer authentication uses the same API_KEY as Voyage; no new secrets.
• Dense E5, Qwen3 and BGE-M3 (bge-m3 or BAAI/bge-m3). BGE-M3 returns only its 1024D dense embedding; sparse and ColBERT are not computed on this route. Use /bge-m3/encode for the hybrid API. mLateOn ColBERT remains on /colbert/encode and Voyage.
• Crucial: this endpoint forwards raw input to the tokenizer without adding any query/document instructions. Supply prefixes in the client configuration. For E5 only, inputs exceeding 512 tokens are truncated from the right to 512 tokens, including the prefix and special tokens. Short inputs are unchanged; truncation is logged with the input index and original token count. Qwen and BGE-M3 inputs are not truncated (BGE-M3 uses BGE_M3_MAX_TOKENS=2048 by default). Unsupported dimensions return an error.

For BGE-M3 in Dify: use model bge-m3, embedding dimensions 1024, and leave query/document prefixes empty. Only dense vectors are returned to standard OpenAI-compatible clients; the native BGE-M3 endpoint keeps dense + sparse and optional ColBERT.

For E5 configure in Dify:

• Document prefix: passage:
• Query prefix: query:
• Model max chunks per batch: start with 16 (Dify's default is 1, which is not this server's limit). Dify inserts a space after the prefix.

The OpenAI endpoint accepts up to 1000 strings per request, subject to 65,536 total processed input tokens (after E5 truncation) and internal padding-aware microbatches. Large requests may still exceed the client's timeout; prefer batches of 8–16 on CPU.

For Qwen3:

• Document prefix: empty

• Query prefix (with a newline between the lines):

  Instruct: Given a question, retrieve relevant passages that answer the question
  Query:
  

The Qwen query prefix contains an actual newline before Query:; ensure the client inserts a newline character rather than the two characters backslash+n. Use Query: immediately before the question, matching the model's original instruction template.

Example with the standard Python OpenAI SDK:

from openai import OpenAI

client = OpenAI(
    base_url="https://YOUR-SPACE.hf.space/openai/v1",
    api_key="YOUR_API_KEY",
)
# The calling application has to apply the configured prefix.
embedding = client.embeddings.create(
    model="e5",
    input=["passage: Первое предложение", "passage: Второе предложение"],
)
print(len(embedding.data[0].embedding))  # 384

Official Voyage AI SDK compatibility

The unmodified Voyage Python client can talk directly to our service. The base URL includes /v1 (not /v1/embeddings):

import voyageai

client = voyageai.Client(
    api_key="YOUR_API_KEY",
    base_url="https://YOUR-SPACE.hf.space/v1",
)

documents = client.embed(
    ["Первый чанк", "Второй чанк"],
    model="e5",               # or "qwen"
    input_type="document",
    truncation=False,
)
print(documents.embeddings)   # list of 384D or 1024D vectors
print(documents.total_tokens)

query = client.embed(
    ["Что происходило в книге?"],
    model="qwen",
    input_type="query",
)

The official client sends POST /v1/embeddings with Bearer authorization, input, model, input_type, truncation, and optional output settings. For Voyage E5 only, chunks longer than 512 tokens (including the injected query: or passage: prefix and special tokens) are automatically truncated from the right, identically to the OpenAI E5 endpoint. This fallback applies even with truncation=False; short texts are unchanged. Qwen and ColBERT still reject overlong inputs. Our response implements Voyage's standard object/data/model/usage envelope; the SDK's EmbeddingsObject.embeddings and total_tokens work.

Model names: e5, qwen, colbert (or the corresponding full Hugging Face IDs). input_type="query" selects query instructions, input_type="document" selects document/passage encoding; an omitted type defaults to document. A single string or a list of 1–1000 strings is accepted.

model="colbert" is a non-standard multi-vector extension: the standard Voyage client will expose embeddings as a 3-D array [document][token][128]. This is not interchangeable with regular Voyage dense embeddings or dense-only vector database integrations.

All three models are already loaded in RAM. Batch calls use actual model batch inference where possible: E5 and Qwen run padded tensor batches, and mLateOn uses PyLate's batch encoding. The server partitions requests by padded token cost to protect 16-GiB CPU memory. Requests are limited to 65,536 aggregate tokens for dense vectors or 32,768 for ColBERT. The request-wide token budget is enforced after E5 truncation; over-budget requests and overlong Qwen/ColBERT inputs still return HTTP 422. Only float output and the fixed model dimensions are supported; quantization, changing dimensions, and Voyage's other API families are not implemented.

Unmodified-SDK integration smoke test:

pip install voyageai==0.5.0
SPACE_URL=http://localhost:7860 API_TEST_KEY='your-strong-api-key' \
  python scripts/smoke_voyage_sdk.py

Deployment, RAM and rollback

Docker downloads immutable checkpoints at image build, and runtime operates offline. Startup fails if any of the three models cannot be loaded or exceeds the configured resident memory budget. The default MEMORY_BUDGET_MB=15360 reserves memory within a nominal 16-GiB container. Run exactly one Uvicorn worker and one concurrent inference.

docker build -t vectorization:three-models .
docker run --rm -p 7860:7860 --memory 16g \
  -e API_KEY='your-strong-api-key' vectorization:three-models

16 GiB RAM is a target, not verified sufficient for 32768-token Qwen inference. Validate longest contexts on the target server, including peak RSS, CPU time and stability. Do not lower the advertised token limits if tests fail.

Tests

python -m pip install torch==2.8.0 --index-url https://download.pytorch.org/whl/cpu
python -m pip install -r requirements-dev.txt
pytest -q tests/test_app.py --junitxml=unit-results.xml

SPACE_URL=http://localhost:7860 API_TEST_KEY='your-strong-api-key' \
  python scripts/smoke_live.py

Unit tests stub model weights. Only a real Space smoke test demonstrates end-to-end inference. For separate 16-GiB RAM/CPU benchmark runs using the downloaded Docker model weights:

docker run --rm --memory 16g --entrypoint python -e BENCH_LONG=1 \
  -v "$PWD/reports:/service/reports" \
  -e BENCH_REPORT=/service/reports/bench-result.json \
  vectorization:three-models scripts/bench_local.py

The benchmark loads all three models at startup and saves timings plus process RSS after every case.

Preserve the existing Space

GitHub Actions runs unit tests on PRs and pushes. Automatic production deployment from main is opt-in: set the GitHub repository variable HF_AUTO_DEPLOY_ENABLED=true and secrets GIT_TOKEN plus API_TEST_KEY. After a successful test run on each new commit to main, Actions backs up the current Space, deploys with commit/revision protection, and runs the old API and official Voyage SDK smoke tests. If a post-deployment smoke check fails, the workflow attempts a revision-guarded automatic restore from the backup. Manual deployment is also available from main through workflow_dispatch (deploy=true) with HF_DEPLOY_ENABLED=true. Commits to feature branches and PRs never deploy the production Space. Production runs and PR runs use separate concurrency groups, so a PR update cannot interrupt a deployment. The existing workflow first makes a complete backup of SmartDataAlex/Jupyter, uploads it as a CI artifact, deploys only if the Space revision matches the backup and runs real authenticated smoke tests. If the backup fails, deployment is blocked.

Manual rollback using a saved backup:

HF_TOKEN='space-scoped-write-token' python scripts/restore_space.py \
  SmartDataAlex-Jupyter-<revision>.tar.gz \
  --confirm-space SmartDataAlex/Jupyter

Store a permanent backup separately; GitHub Actions artifacts expire.