Spaces:
Running
Running
| title: Vectorization API | |
| emoji: 🧠 | |
| colorFrom: blue | |
| colorTo: indigo | |
| sdk: docker | |
| app_port: 7860 | |
| pinned: false | |
| # Vectorization API — configurable resident models | |
| CPU-only FastAPI inference service. **Only selected models load during startup**, then remain resident. Inference never loads model weights. No search, Qdrant, Manticore, reranking or Entity Linking. | |
| | Task | Model | Max tokens | Output | | |
| | --- | --- | ---: | --- | | |
| | Standard | `intfloat/multilingual-e5-small` | 512 | 384D | | |
| | Literature | `Qwen/Qwen3-Embedding-0.6B` | 32768 | 1024D | | |
| | ColBERT | `lightonai/mLateOn` | 8192 | 128D per token | | |
| | Triple retrieval | `BAAI/bge-m3` | 8192 native, 2048 server default | dense 1024D + sparse lexical + ColBERT 1024D/token | | |
| ## Runtime model selection | |
| Set `ENABLED_MODELS` in Hugging Face Space **Settings → Variables and secrets** | |
| (or as a container environment variable). Comma-separated names: | |
| ```dotenv | |
| ENABLED_MODELS=e5,bge-m3 | |
| ``` | |
| Supported: `e5` (384D), `qwen` (1024D), `colbert` (mLateOn 128D), | |
| `bge-m3` (1024D dense/sparse/ColBERT). For only BGE-M3 use | |
| `ENABLED_MODELS=bge-m3`; for all four, use | |
| `ENABLED_MODELS=e5,qwen,colbert,bge-m3`. | |
| When omitted, all four load (previous default). Old `BGE_M3_ENABLED=0` | |
| is still honored **only if `ENABLED_MODELS` is unset**. With an explicit | |
| `ENABLED_MODELS`, the list takes precedence. Empty, duplicate, or unknown | |
| names fail startup with an explanatory error; at least one model is required. | |
| Changing the environment requires a Space restart. | |
| Disabled models are not instantiated or kept in RAM. Endpoints remain | |
| defined for compatibility but return HTTP 503 if their model is disabled. | |
| `GET /models` and `GET /openai/v1/models` show only loaded models; | |
| `GET /health` includes `loaded_models`. The legacy `GET /model` is | |
| E5-only and returns 503 if E5 is not loaded, as does an `/embed` request | |
| without a model (which defaults to E5). | |
| **Scope:** this controls *runtime loading and memory*. The Docker build | |
| still downloads all four checkpoints and performs the mLateOn build preflight, | |
| even if a model is disabled for runtime. Build-time selection is separate. | |
| ## BGE-M3: triple-vector embedding | |
| The official `FlagEmbedding.BGEM3FlagModel` supports dense, sparse and | |
| ColBERT projections in one forward pass. By default the API computes only | |
| **dense + sparse**, skipping the ColBERT projection and its large output. | |
| Set `return_colbert: true` in the request to compute and return all three. | |
| These BGE-M3 ColBERT | |
| vectors are 1024D, unrelated to the existing 128D `lightonai/mLateOn` | |
| vectors. Do not store them in the same named Qdrant vector field. | |
| `POST /bge-m3/encode` requires `Authorization: Bearer <API_KEY>`: | |
| ```bash | |
| curl -sS https://smartdataalex-jupyter.hf.space/bge-m3/encode \ | |
| -H "Authorization: Bearer YOUR_API_KEY" \ | |
| -H "Content-Type: application/json" \ | |
| -d '{"input":["What is RAG?", "מהי שליפה סמנטית?"],"input_type":"document","return_colbert":false}' | |
| ``` | |
| Response contains `model: "BAAI/bge-m3"`, `dense_dimensions: 1024`, | |
| `colbert_dimensions: 1024`, `data[]` entries with `index`, `dense` | |
| (1024 floats), `sparse.indices` (integer tokenizer IDs), | |
| `sparse.values` (learned positive lexical weights) and `tokens_count`. | |
| Only when `return_colbert: true`, each item additionally contains | |
| `colbert`: a variable-length array of 1024D token embeddings. | |
| By default `return_colbert: false`, and the projection is not computed. | |
| The sparse `indices`/`values` structure is directly suitable for Qdrant | |
| sparse vectors. A single request accepts 1–8 texts; inference is serial | |
| within the shared CPU inference gate. Query/document modes require no | |
| manual instruction prefix for BGE-M3. | |
| BGE-M3 loading is controlled by `ENABLED_MODELS` (with legacy | |
| `BGE_M3_ENABLED` fallback when the new variable is absent). Model | |
| checkpoint download is pinned. `BGE_M3_MAX_TOKENS=2048` is the initial **operational** cap on | |
| our shared 16-GiB CPU server; the underlying model supports 8192. To use | |
| larger contexts, increase that variable (maximum 8192) after checking | |
| CPU inference memory and latency. Overlong texts produce HTTP 422, not | |
| silent truncation. Max total tokens per request: 8192. No change to | |
| existing OpenAI-compatible/Voyage/ColBERT response contracts. | |
| BGE-M3 itself does not compress or quantize its ColBERT output. Qdrant | |
| supports storage compression for multivectors through `datatype: turbo4` | |
| (approximately 8x smaller vs float32, requires a compatible Qdrant version). | |
| Set `hnsw_config.m=0` for ColBERT reranking fields. These are Qdrant | |
| collection settings, not parameters for the BGE-M3 inference endpoint; | |
| transfer of raw ColBERT JSON is still large. | |
| ## API | |
| Legacy endpoints require `X-API-Key`; Voyage-compatible `/v1/embeddings` uses `Authorization: Bearer <API_KEY>`. Public endpoints: `/health`, `/docs` and `/openapi.json`. | |
| - `POST /embed`: `{"text":"Your text","type":"query"}` defaults to E5. The optional `model` accepts `e5`, `qwen`, or the corresponding full model name. The existing E5 response fields are unchanged: `text`, `type`, `tokens_count`, `embedding_model`, `dimensions`, `embedding`. | |
| - `POST /colbert/encode`: `{"text":"Your text","type":"passage"}` returns `embedding_model`, `tokens_count`, `dimensions` and `token_embeddings` (matrix of individual 128D vectors). | |
| - `GET /models`: readiness, dimensions, limits, normalization method, current/peak RSS and startup RAM readings for all models. | |
| - `GET /model`: original E5-only metadata contract. | |
| - `GET /health`: returns 200 when all configured models have completed startup. | |
| Example (replace URL and key): | |
| ```bash | |
| curl -sS -X POST 'http://localhost:7860/embed' \ | |
| -H 'X-API-Key: YOUR_API_KEY' -H 'Content-Type: application/json' \ | |
| -d '{"text":"What does the author mean?","type":"query","model":"qwen"}' | |
| curl -sS -X POST 'http://localhost:7860/colbert/encode' \ | |
| -H 'X-API-Key: YOUR_API_KEY' -H 'Content-Type: application/json' \ | |
| -d '{"text":"שלום עולם","type":"passage"}' | |
| ``` | |
| E5 keeps `query: ` / `passage: ` prefixes, attention-mask-aware mean pooling, **no extra L2 normalization**. Qwen uses its official query instruction + last-token pooling + L2 normalization; documents have no instruction. PyLate mLateOn handles query and passage separately, retaining token vectors and their model-native normalization. Token limits include model markers and special tokens. The legacy API | |
| rejects overlong inputs (HTTP 422), while E5 Voyage and OpenAI-compatible | |
| endpoints truncate inputs exceeding 512 tokens, logging affected batch indices. A single CPU inference runs at a time; up to 16 additional HTTP requests wait | |
| for a free inference slot (40 seconds by default). A full queue or expired wait | |
| returns HTTP 429 with `Retry-After`. Configure `INFERENCE_MAX_WAITERS` and | |
| `INFERENCE_WAIT_SECONDS` as needed. Dify's OpenAI-compatible embedding provider | |
| uses a 60-second HTTP timeout, so keep the server's wait timeout below that. | |
| ## OpenAI-compatible API (separate from Voyage) | |
| **OpenAI SDK-compatible base URL:** `https://YOUR-SPACE.hf.space/openai/v1`. | |
| This separate namespace is necessary because the existing Voyage-compatible | |
| `/v1/embeddings` endpoint is already in production and must retain its default | |
| `input_type` handling. Point OpenAI-compatible custom-provider integrations | |
| to the base URL ending in `/openai/v1`, not to the Voyage `/v1` URL. | |
| - `POST /openai/v1/embeddings`: accepts `model`, `input` (string or string array), | |
| optional `dimensions`, `encoding_format` (`float` or `base64`) and `user`. | |
| Also accepts and ignores the non-standard boolean `drop_params` sent by | |
| some OpenAI-compatible connection checks; other unknown fields remain errors. | |
| Responds in the OpenAI `data[].embedding` / `usage.prompt_tokens` / | |
| `usage.total_tokens` format. Only fixed native dimensions are supported. | |
| - `GET /openai/v1/models`: dense model aliases for provider selection. | |
| - Bearer authentication uses the **same** `API_KEY` as Voyage; no new secrets. | |
| - Dense E5, Qwen3 and BGE-M3 (`bge-m3` or `BAAI/bge-m3`). BGE-M3 | |
| returns only its 1024D dense embedding; sparse and ColBERT are not | |
| computed on this route. Use `/bge-m3/encode` for the hybrid API. | |
| mLateOn ColBERT remains on `/colbert/encode` and Voyage. | |
| - **Crucial:** this endpoint forwards raw input to the tokenizer **without adding any | |
| query/document instructions**. Supply prefixes in the client configuration. | |
| For **E5 only**, inputs exceeding 512 tokens are truncated from the right | |
| to 512 tokens, including the prefix and special tokens. Short inputs are | |
| unchanged; truncation is logged with the input index and original token count. | |
| Qwen and BGE-M3 inputs are not truncated (BGE-M3 uses | |
| `BGE_M3_MAX_TOKENS=2048` by default). Unsupported dimensions return an error. | |
| For BGE-M3 in Dify: use model `bge-m3`, embedding dimensions `1024`, | |
| and leave query/document prefixes empty. Only dense vectors are returned | |
| to standard OpenAI-compatible clients; the native BGE-M3 endpoint keeps | |
| dense + sparse and optional ColBERT. | |
| For E5 configure in Dify: | |
| - Document prefix: `passage:` | |
| - Query prefix: `query:` | |
| - Model max chunks per batch: start with `16` (Dify's default is `1`, | |
| which is not this server's limit). Dify inserts a space after the prefix. | |
| The OpenAI endpoint accepts up to 1000 strings per request, subject to | |
| 65,536 total **processed** input tokens (after E5 truncation) and internal | |
| padding-aware microbatches. Large | |
| requests may still exceed the client's timeout; prefer batches of 8–16 on CPU. | |
| For Qwen3: | |
| - Document prefix: *empty* | |
| - Query prefix (with a newline between the lines): | |
| ```text | |
| Instruct: Given a question, retrieve relevant passages that answer the question | |
| Query: | |
| ``` | |
| The Qwen query prefix contains an actual newline before `Query:`; ensure the | |
| client inserts a newline character rather than the two characters backslash+n. | |
| Use `Query:` immediately before the question, matching the model's original | |
| instruction template. | |
| Example with the standard Python OpenAI SDK: | |
| ```python | |
| from openai import OpenAI | |
| client = OpenAI( | |
| base_url="https://YOUR-SPACE.hf.space/openai/v1", | |
| api_key="YOUR_API_KEY", | |
| ) | |
| # The calling application has to apply the configured prefix. | |
| embedding = client.embeddings.create( | |
| model="e5", | |
| input=["passage: Первое предложение", "passage: Второе предложение"], | |
| ) | |
| print(len(embedding.data[0].embedding)) # 384 | |
| ``` | |
| ## Official Voyage AI SDK compatibility | |
| The **unmodified Voyage Python client** can talk directly to our service. | |
| The base URL includes `/v1` (not `/v1/embeddings`): | |
| ```python | |
| import voyageai | |
| client = voyageai.Client( | |
| api_key="YOUR_API_KEY", | |
| base_url="https://YOUR-SPACE.hf.space/v1", | |
| ) | |
| documents = client.embed( | |
| ["Первый чанк", "Второй чанк"], | |
| model="e5", # or "qwen" | |
| input_type="document", | |
| truncation=False, | |
| ) | |
| print(documents.embeddings) # list of 384D or 1024D vectors | |
| print(documents.total_tokens) | |
| query = client.embed( | |
| ["Что происходило в книге?"], | |
| model="qwen", | |
| input_type="query", | |
| ) | |
| ``` | |
| The official client sends `POST /v1/embeddings` with Bearer authorization, | |
| `input`, `model`, `input_type`, `truncation`, and optional output settings. | |
| For Voyage **E5 only**, chunks longer than 512 tokens (including the injected | |
| `query: ` or `passage: ` prefix and special tokens) are automatically truncated | |
| from the right, identically to the OpenAI E5 endpoint. This fallback applies | |
| even with `truncation=False`; short texts are unchanged. Qwen and ColBERT | |
| still reject overlong inputs. | |
| Our response implements Voyage's standard `object`/`data`/`model`/`usage` | |
| envelope; the SDK's `EmbeddingsObject.embeddings` and `total_tokens` work. | |
| Model names: `e5`, `qwen`, `colbert` (or the corresponding full Hugging Face IDs). | |
| `input_type="query"` selects query instructions, `input_type="document"` selects | |
| document/passage encoding; an omitted type defaults to document. | |
| A single string or a list of 1–1000 strings is accepted. | |
| `model="colbert"` is a **non-standard multi-vector extension**: | |
| the standard Voyage client will expose `embeddings` as a 3-D array | |
| `[document][token][128]`. This is not interchangeable with regular | |
| Voyage dense embeddings or dense-only vector database integrations. | |
| All three models are already loaded in RAM. Batch calls use **actual model | |
| batch inference** where possible: E5 and Qwen run padded tensor batches, | |
| and mLateOn uses PyLate's batch encoding. The server partitions requests | |
| by padded token cost to protect 16-GiB CPU memory. Requests are limited to | |
| 65,536 aggregate tokens for dense vectors or 32,768 for ColBERT. | |
| The request-wide token budget is enforced after E5 truncation; over-budget | |
| requests and overlong Qwen/ColBERT inputs still return HTTP 422. Only float | |
| output and the fixed model | |
| dimensions are supported; quantization, changing dimensions, and Voyage's | |
| other API families are not implemented. | |
| Unmodified-SDK integration smoke test: | |
| ```bash | |
| pip install voyageai==0.5.0 | |
| SPACE_URL=http://localhost:7860 API_TEST_KEY='your-strong-api-key' \ | |
| python scripts/smoke_voyage_sdk.py | |
| ``` | |
| ## Deployment, RAM and rollback | |
| Docker downloads immutable checkpoints at image build, and runtime operates offline. Startup fails if any of the three models cannot be loaded or exceeds the configured resident memory budget. The default `MEMORY_BUDGET_MB=15360` reserves memory within a nominal 16-GiB container. Run exactly **one** Uvicorn worker and one concurrent inference. | |
| ```bash | |
| docker build -t vectorization:three-models . | |
| docker run --rm -p 7860:7860 --memory 16g \ | |
| -e API_KEY='your-strong-api-key' vectorization:three-models | |
| ``` | |
| 16 GiB RAM is a target, **not verified sufficient** for 32768-token Qwen inference. Validate longest contexts on the target server, including peak RSS, CPU time and stability. Do not lower the advertised token limits if tests fail. | |
| ### Tests | |
| ```bash | |
| python -m pip install torch==2.8.0 --index-url https://download.pytorch.org/whl/cpu | |
| python -m pip install -r requirements-dev.txt | |
| pytest -q tests/test_app.py --junitxml=unit-results.xml | |
| SPACE_URL=http://localhost:7860 API_TEST_KEY='your-strong-api-key' \ | |
| python scripts/smoke_live.py | |
| ``` | |
| Unit tests stub model weights. Only a real Space smoke test demonstrates end-to-end inference. For separate 16-GiB RAM/CPU benchmark runs using the downloaded Docker model weights: | |
| ```bash | |
| docker run --rm --memory 16g --entrypoint python -e BENCH_LONG=1 \ | |
| -v "$PWD/reports:/service/reports" \ | |
| -e BENCH_REPORT=/service/reports/bench-result.json \ | |
| vectorization:three-models scripts/bench_local.py | |
| ``` | |
| The benchmark loads all three models at startup and saves timings plus process RSS after every case. | |
| ### Preserve the existing Space | |
| GitHub Actions runs unit tests on PRs and pushes. **Automatic production deployment from `main` is opt-in:** set the GitHub repository variable `HF_AUTO_DEPLOY_ENABLED=true` and secrets `GIT_TOKEN` plus `API_TEST_KEY`. After a successful test run on each new commit to `main`, Actions backs up the current Space, deploys with commit/revision protection, and runs the old API and official Voyage SDK smoke tests. If a post-deployment smoke check fails, the workflow attempts a revision-guarded automatic restore from the backup. Manual deployment is also available from `main` through `workflow_dispatch` (`deploy=true`) with `HF_DEPLOY_ENABLED=true`. Commits to feature branches and PRs never deploy the production Space. Production runs and PR runs use separate concurrency groups, so a PR update cannot interrupt a deployment. The existing workflow first makes a complete backup of `SmartDataAlex/Jupyter`, uploads it as a CI artifact, deploys only if the Space revision matches the backup and runs real authenticated smoke tests. If the backup fails, deployment is blocked. | |
| Manual rollback using a saved backup: | |
| ```bash | |
| HF_TOKEN='space-scoped-write-token' python scripts/restore_space.py \ | |
| SmartDataAlex-Jupyter-<revision>.tar.gz \ | |
| --confirm-space SmartDataAlex/Jupyter | |
| ``` | |
| Store a permanent backup separately; GitHub Actions artifacts expire. | |