Blame for README.md

---
title: Vectorization API
emoji: 🧠
colorFrom: blue
colorTo: indigo
sdk: docker
app_port: 7860
pinned: false
---

# Vectorization API — configurable resident models

CPU-only FastAPI inference service. **Only selected models load during startup**, then remain resident. Inference never loads model weights. No search, Qdrant, Manticore, reranking or Entity Linking.

| Task | Model | Max tokens | Output |
| --- | --- | ---: | --- |
| Standard | `intfloat/multilingual-e5-small` | 512 | 384D |
| Literature | `Qwen/Qwen3-Embedding-0.6B` | 32768 | 1024D |
| ColBERT | `lightonai/mLateOn` | 8192 | 128D per token |
| Triple retrieval | `BAAI/bge-m3` | 8192 native, 2048 server default | dense 1024D + sparse lexical + ColBERT 1024D/token |

## Runtime model selection

Set `ENABLED_MODELS` in Hugging Face Space **Settings → Variables and secrets**
(or as a container environment variable). Comma-separated names:

```dotenv
ENABLED_MODELS=e5,bge-m3
```

Supported: `e5` (384D), `qwen` (1024D), `colbert` (mLateOn 128D),
`bge-m3` (1024D dense/sparse/ColBERT). For only BGE-M3 use
`ENABLED_MODELS=bge-m3`; for all four, use
`ENABLED_MODELS=e5,qwen,colbert,bge-m3`.

When omitted, all four load (previous default). Old `BGE_M3_ENABLED=0`
is still honored **only if `ENABLED_MODELS` is unset**. With an explicit
`ENABLED_MODELS`, the list takes precedence. Empty, duplicate, or unknown
names fail startup with an explanatory error; at least one model is required.
Changing the environment requires a Space restart.

Disabled models are not instantiated or kept in RAM. Endpoints remain
defined for compatibility but return HTTP 503 if their model is disabled.
`GET /models` and `GET /openai/v1/models` show only loaded models;
`GET /health` includes `loaded_models`. The legacy `GET /model` is
E5-only and returns 503 if E5 is not loaded, as does an `/embed` request
without a model (which defaults to E5).

**Scope:** this controls *runtime loading and memory*. The Docker build
still downloads all four checkpoints and performs the mLateOn build preflight,
even if a model is disabled for runtime. Build-time selection is separate.

## BGE-M3: triple-vector embedding

The official `FlagEmbedding.BGEM3FlagModel` supports dense, sparse and
ColBERT projections in one forward pass. By default the API computes only
**dense + sparse**, skipping the ColBERT projection and its large output.
Set `return_colbert: true` in the request to compute and return all three.
These BGE-M3 ColBERT
vectors are 1024D, unrelated to the existing 128D `lightonai/mLateOn`
vectors. Do not store them in the same named Qdrant vector field.

`POST /bge-m3/encode` requires `Authorization: Bearer <API_KEY>`:

```bash
curl -sS https://smartdataalex-jupyter.hf.space/bge-m3/encode \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"input":["What is RAG?", "מהי שליפה סמנטית?"],"input_type":"document","return_colbert":false}'
```

Response contains `model: "BAAI/bge-m3"`, `dense_dimensions: 1024`,
`colbert_dimensions: 1024`, `data[]` entries with `index`, `dense`
(1024 floats), `sparse.indices` (integer tokenizer IDs),
`sparse.values` (learned positive lexical weights) and `tokens_count`.
Only when `return_colbert: true`, each item additionally contains
`colbert`: a variable-length array of 1024D token embeddings.
By default `return_colbert: false`, and the projection is not computed.
The sparse `indices`/`values` structure is directly suitable for Qdrant
sparse vectors. A single request accepts 1–8 texts; inference is serial
within the shared CPU inference gate. Query/document modes require no
manual instruction prefix for BGE-M3.

BGE-M3 loading is controlled by `ENABLED_MODELS` (with legacy
`BGE_M3_ENABLED` fallback when the new variable is absent). Model
checkpoint download is pinned. `BGE_M3_MAX_TOKENS=2048` is the initial **operational** cap on
our shared 16-GiB CPU server; the underlying model supports 8192. To use
larger contexts, increase that variable (maximum 8192) after checking
CPU inference memory and latency. Overlong texts produce HTTP 422, not
silent truncation. Max total tokens per request: 8192. No change to
existing OpenAI-compatible/Voyage/ColBERT response contracts.

BGE-M3 itself does not compress or quantize its ColBERT output. Qdrant
supports storage compression for multivectors through `datatype: turbo4`
(approximately 8x smaller vs float32, requires a compatible Qdrant version).
Set `hnsw_config.m=0` for ColBERT reranking fields. These are Qdrant
collection settings, not parameters for the BGE-M3 inference endpoint;
transfer of raw ColBERT JSON is still large.

## API

Legacy endpoints require `X-API-Key`; Voyage-compatible `/v1/embeddings` uses `Authorization: Bearer <API_KEY>`. Public endpoints: `/health`, `/docs` and `/openapi.json`.

- `POST /embed`: `{"text":"Your text","type":"query"}` defaults to E5. The optional `model` accepts `e5`, `qwen`, or the corresponding full model name. The existing E5 response fields are unchanged: `text`, `type`, `tokens_count`, `embedding_model`, `dimensions`, `embedding`.
- `POST /colbert/encode`: `{"text":"Your text","type":"passage"}` returns `embedding_model`, `tokens_count`, `dimensions` and `token_embeddings` (matrix of individual 128D vectors).
- `GET /models`: readiness, dimensions, limits, normalization method, current/peak RSS and startup RAM readings for all models.
- `GET /model`: original E5-only metadata contract.
- `GET /health`: returns 200 when all configured models have completed startup.

Example (replace URL and key):

```bash
curl -sS -X POST 'http://localhost:7860/embed' \
  -H 'X-API-Key: YOUR_API_KEY' -H 'Content-Type: application/json' \
  -d '{"text":"What does the author mean?","type":"query","model":"qwen"}'

curl -sS -X POST 'http://localhost:7860/colbert/encode' \
  -H 'X-API-Key: YOUR_API_KEY' -H 'Content-Type: application/json' \
  -d '{"text":"שלום עולם","type":"passage"}'
```

E5 keeps `query: ` / `passage: ` prefixes, attention-mask-aware mean pooling, **no extra L2 normalization**. Qwen uses its official query instruction + last-token pooling + L2 normalization; documents have no instruction. PyLate mLateOn handles query and passage separately, retaining token vectors and their model-native normalization. Token limits include model markers and special tokens. The legacy API
rejects overlong inputs (HTTP 422), while E5 Voyage and OpenAI-compatible
endpoints truncate inputs exceeding 512 tokens, logging affected batch indices. A single CPU inference runs at a time; up to 16 additional HTTP requests wait
for a free inference slot (40 seconds by default). A full queue or expired wait
returns HTTP 429 with `Retry-After`. Configure `INFERENCE_MAX_WAITERS` and
`INFERENCE_WAIT_SECONDS` as needed. Dify's OpenAI-compatible embedding provider
uses a 60-second HTTP timeout, so keep the server's wait timeout below that.

## OpenAI-compatible API (separate from Voyage)

**OpenAI SDK-compatible base URL:** `https://YOUR-SPACE.hf.space/openai/v1`.
This separate namespace is necessary because the existing Voyage-compatible
`/v1/embeddings` endpoint is already in production and must retain its default
`input_type` handling. Point OpenAI-compatible custom-provider integrations
to the base URL ending in `/openai/v1`, not to the Voyage `/v1` URL.

- `POST /openai/v1/embeddings`: accepts `model`, `input` (string or string array),
  optional `dimensions`, `encoding_format` (`float` or `base64`) and `user`.
  Also accepts and ignores the non-standard boolean `drop_params` sent by
  some OpenAI-compatible connection checks; other unknown fields remain errors.
  Responds in the OpenAI `data[].embedding` / `usage.prompt_tokens` /
  `usage.total_tokens` format. Only fixed native dimensions are supported.
- `GET /openai/v1/models`: dense model aliases for provider selection.
- Bearer authentication uses the **same** `API_KEY` as Voyage; no new secrets.
- Dense E5, Qwen3 and BGE-M3 (`bge-m3` or `BAAI/bge-m3`). BGE-M3
  returns only its 1024D dense embedding; sparse and ColBERT are not
  computed on this route. Use `/bge-m3/encode` for the hybrid API.
  mLateOn ColBERT remains on `/colbert/encode` and Voyage.
- **Crucial:** this endpoint forwards raw input to the tokenizer **without adding any
  query/document instructions**. Supply prefixes in the client configuration.
  For **E5 only**, inputs exceeding 512 tokens are truncated from the right
  to 512 tokens, including the prefix and special tokens. Short inputs are
  unchanged; truncation is logged with the input index and original token count.
  Qwen and BGE-M3 inputs are not truncated (BGE-M3 uses
  `BGE_M3_MAX_TOKENS=2048` by default). Unsupported dimensions return an error.

For BGE-M3 in Dify: use model `bge-m3`, embedding dimensions `1024`,
and leave query/document prefixes empty. Only dense vectors are returned
to standard OpenAI-compatible clients; the native BGE-M3 endpoint keeps
dense + sparse and optional ColBERT.

For E5 configure in Dify:
- Document prefix: `passage:`
- Query prefix: `query:`
- Model max chunks per batch: start with `16` (Dify's default is `1`,
  which is not this server's limit). Dify inserts a space after the prefix.

The OpenAI endpoint accepts up to 1000 strings per request, subject to
65,536 total **processed** input tokens (after E5 truncation) and internal
padding-aware microbatches. Large
requests may still exceed the client's timeout; prefer batches of 8–16 on CPU.

For Qwen3:
- Document prefix: *empty*
- Query prefix (with a newline between the lines):

  ```text
  Instruct: Given a question, retrieve relevant passages that answer the question
  Query:
  ```

The Qwen query prefix contains an actual newline before `Query:`; ensure the
client inserts a newline character rather than the two characters backslash+n.
Use `Query:` immediately before the question, matching the model's original
instruction template.

Example with the standard Python OpenAI SDK:

```python
from openai import OpenAI

client = OpenAI(
    base_url="https://YOUR-SPACE.hf.space/openai/v1",
    api_key="YOUR_API_KEY",
)
# The calling application has to apply the configured prefix.
embedding = client.embeddings.create(
    model="e5",
    input=["passage: Первое предложение", "passage: Второе предложение"],
)
print(len(embedding.data[0].embedding))  # 384
```

## Official Voyage AI SDK compatibility

The **unmodified Voyage Python client** can talk directly to our service.
The base URL includes `/v1` (not `/v1/embeddings`):

```python
import voyageai

client = voyageai.Client(
    api_key="YOUR_API_KEY",
    base_url="https://YOUR-SPACE.hf.space/v1",
)

documents = client.embed(
    ["Первый чанк", "Второй чанк"],
    model="e5",               # or "qwen"
    input_type="document",
    truncation=False,
)
print(documents.embeddings)   # list of 384D or 1024D vectors
print(documents.total_tokens)

query = client.embed(
    ["Что происходило в книге?"],
    model="qwen",
    input_type="query",
)
```

The official client sends `POST /v1/embeddings` with Bearer authorization,
`input`, `model`, `input_type`, `truncation`, and optional output settings.
For Voyage **E5 only**, chunks longer than 512 tokens (including the injected
`query: ` or `passage: ` prefix and special tokens) are automatically truncated
from the right, identically to the OpenAI E5 endpoint. This fallback applies
even with `truncation=False`; short texts are unchanged. Qwen and ColBERT
still reject overlong inputs.
Our response implements Voyage's standard `object`/`data`/`model`/`usage`
envelope; the SDK's `EmbeddingsObject.embeddings` and `total_tokens` work.

Model names: `e5`, `qwen`, `colbert` (or the corresponding full Hugging Face IDs).
`input_type="query"` selects query instructions, `input_type="document"` selects
document/passage encoding; an omitted type defaults to document.
A single string or a list of 1–1000 strings is accepted.

`model="colbert"` is a **non-standard multi-vector extension**:
the standard Voyage client will expose `embeddings` as a 3-D array
`[document][token][128]`. This is not interchangeable with regular
Voyage dense embeddings or dense-only vector database integrations.

All three models are already loaded in RAM. Batch calls use **actual model
batch inference** where possible: E5 and Qwen run padded tensor batches,
and mLateOn uses PyLate's batch encoding. The server partitions requests
by padded token cost to protect 16-GiB CPU memory. Requests are limited to
65,536 aggregate tokens for dense vectors or 32,768 for ColBERT.
The request-wide token budget is enforced after E5 truncation; over-budget
requests and overlong Qwen/ColBERT inputs still return HTTP 422. Only float
output and the fixed model
dimensions are supported; quantization, changing dimensions, and Voyage's
other API families are not implemented.

Unmodified-SDK integration smoke test:

```bash
pip install voyageai==0.5.0
SPACE_URL=http://localhost:7860 API_TEST_KEY='your-strong-api-key' \
  python scripts/smoke_voyage_sdk.py
```

## Deployment, RAM and rollback

Docker downloads immutable checkpoints at image build, and runtime operates offline. Startup fails if any of the three models cannot be loaded or exceeds the configured resident memory budget. The default `MEMORY_BUDGET_MB=15360` reserves memory within a nominal 16-GiB container. Run exactly **one** Uvicorn worker and one concurrent inference.

```bash
docker build -t vectorization:three-models .
docker run --rm -p 7860:7860 --memory 16g \
  -e API_KEY='your-strong-api-key' vectorization:three-models
```

16 GiB RAM is a target, **not verified sufficient** for 32768-token Qwen inference. Validate longest contexts on the target server, including peak RSS, CPU time and stability. Do not lower the advertised token limits if tests fail.

### Tests

```bash
python -m pip install torch==2.8.0 --index-url https://download.pytorch.org/whl/cpu
python -m pip install -r requirements-dev.txt
pytest -q tests/test_app.py --junitxml=unit-results.xml

SPACE_URL=http://localhost:7860 API_TEST_KEY='your-strong-api-key' \
  python scripts/smoke_live.py
```

Unit tests stub model weights. Only a real Space smoke test demonstrates end-to-end inference. For separate 16-GiB RAM/CPU benchmark runs using the downloaded Docker model weights:

```bash
docker run --rm --memory 16g --entrypoint python -e BENCH_LONG=1 \
  -v "$PWD/reports:/service/reports" \
  -e BENCH_REPORT=/service/reports/bench-result.json \
  vectorization:three-models scripts/bench_local.py
```

The benchmark loads all three models at startup and saves timings plus process RSS after every case.

### Preserve the existing Space

GitHub Actions runs unit tests on PRs and pushes. **Automatic production deployment from `main` is opt-in:** set the GitHub repository variable `HF_AUTO_DEPLOY_ENABLED=true` and secrets `GIT_TOKEN` plus `API_TEST_KEY`. After a successful test run on each new commit to `main`, Actions backs up the current Space, deploys with commit/revision protection, and runs the old API and official Voyage SDK smoke tests. If a post-deployment smoke check fails, the workflow attempts a revision-guarded automatic restore from the backup. Manual deployment is also available from `main` through `workflow_dispatch` (`deploy=true`) with `HF_DEPLOY_ENABLED=true`. Commits to feature branches and PRs never deploy the production Space. Production runs and PR runs use separate concurrency groups, so a PR update cannot interrupt a deployment. The existing workflow first makes a complete backup of `SmartDataAlex/Jupyter`, uploads it as a CI artifact, deploys only if the Space revision matches the backup and runs real authenticated smoke tests. If the backup fails, deployment is blocked.

Manual rollback using a saved backup:

```bash
HF_TOKEN='space-scoped-write-token' python scripts/restore_space.py \
  SmartDataAlex-Jupyter-<revision>.tar.gz \
  --confirm-space SmartDataAlex/Jupyter
```

Store a permanent backup separately; GitHub Actions artifacts expire.