source/docs/model.md

# Nanobe model package

The package provides MLX inference for text, rendered document pages, images, audio and video with sound. It includes the language and multimodal weights, retrieval head, media adapters, document-page merger, audio-window pooler, Perception Encoder audio and text components, tokenizers and Python inference helpers.

## Embeddings

The active `geometry.json` defines a 2,690-coordinate model embedding. Folder indexing extends it to 2,696 coordinates with the package's calibration. The text scale is 0.94 and the raw acoustic query weight is 0.02. The model has a 640-coordinate retrieval head and a 1,024-coordinate native component, with acoustic features and padding completing the output. Head size is not the final embedding dimension.

The model geometry, tensor values, quantization settings, tokenizer settings, prompts, pooling and sampling constants define inference. The model and its helper files must remain together.

## Python inference

Install the matching source release with `uv sync --frozen` and set `NANOBE_MODEL_DIR` as described in the source README. The runnable `examples/search.py` ranks supplied files using either a text query or a reference file. It uses the same file encoder as the folder CLI.

### Text query against a file

Run this from the source directory after creating the README's demo files:

```python
import os
from pathlib import Path

import numpy as np

from nanobe.inference import encode_file
from nanobe.runtime import load, load_lanes, load_query

package = Path(os.environ["NANOBE_MODEL_DIR"])
query = load_query(package).encode_query("Saturn through a telescope")
runtime = load(package)
calibration = load_lanes(package)

kind, segments = encode_file(runtime, Path("demo/observatory.txt"))
vectors = np.stack([calibration.item(segment.vector, kind) for segment in segments])
scores = vectors @ query / (np.linalg.norm(vectors, axis=1) * np.linalg.norm(query))
best = int(np.argmax(scores))
print(segments[best].label, float(scores[best]))
```

`encode_file` returns the file kind and a list of segments. Every segment has an `ordinal`, `label`, raw `vector` and extracted `lexical` text when available. The raw vector has 2,690 coordinates. `calibration.item` transforms it into a 2,696-coordinate vector with the file kind's scale and extra coordinates. `load_query(...).encode_query` already returns a calibrated 2,696-coordinate query. Do not calibrate that query a second time or compare it directly with raw file vectors.

`load_query` loads only the components needed for text queries. `load` loads the complete runtime used by `encode_file`. Reuse these objects when encoding multiple inputs. The full runtime's lower-level `encode_query` method returns 2,690 coordinates; it needs `calibration.query` before comparison with calibrated items.

### Reference file as a query

A reference file can have multiple segments. Its query vectors use `calibration.query`, while the files being searched still use `calibration.item`:

```python
_, reference = encode_file(runtime, Path("reference.txt"))
queries = np.stack([calibration.query(segment.vector) for segment in reference])
scores = queries @ vectors.T
scores /= np.linalg.norm(queries, axis=1)[:, None] * np.linalg.norm(vectors, axis=1)[None, :]
query, item = np.unravel_index(np.argmax(scores), scores.shape)
print(reference[query].label, segments[item].label, float(scores[query, item]))
```

For several files, keep each file's best segment-pair score and sort descending. Exclude the reference file itself when it also appears among the files being searched. `examples/search.py` performs those steps.

### Segments and retrieval behavior

The shared file encoder applies the documented readers, text chunks, PDF page selection, audio/video windows, audio pooling and DOC/PPT conversion. Replacing the example paths with supported images, documents, recordings or videos uses those same rules. A whole file may return multiple vectors; averaging them changes the retrieval method.

These examples perform exact semantic comparisons on full-precision embeddings. The folder CLI stores segment values in float16, uses an int8 approximate index to select candidates, reranks with calibrated stored vectors, and selects the best segment per file. Small numerical differences can follow from storage precision. Text searches also combine semantic and lexical ranks, so their final scores and ordering can differ from pure semantic scoring. Reference-file CLI searches use semantic retrieval only.

## File readers

The Python reference implementation decodes AVIF and HEIC as RGB images and rasterizes SVG on white within a 2048-by-2048 box. HEIC uses the primary image, applies container orientation and reduces HDR to 8 bits. SVG text depends on installed fonts and the renderer's platform fallback.

RTF supplies extracted text. XLS/XLSX supply saved cell values, sheet names and repeated first-row headers without formula recalculation. Workbooks share eight 6,000-character segment slots across the first eight nonempty sheets and report truncation warnings. DOC/PPT require LibreOffice and use converted PDF pages. Nanobe finds LibreOffice on PATH or at its standard macOS application location. Conversion has a 120-second batch timeout.

DOCX/PPTX/EPUB supply extracted text. PDF and converted document inputs use the first 24 pages plus up to eight later pages. Audio and video retain whole-file embeddings and bounded additional segments. Missing decoders or converters, empty inputs and damaged files can fail. These readers prepare model inputs; they do not add a separate embedding model. The source README documents the complete format list and limits.

## Attribution

The base model is `jinaai/jina-embeddings-v5-omni-small-retrieval` at revision `12949877f0092093f366c6450340011320152a05`. Perception Encoder components derive from `facebook/pe-av-small`; the text component records revision `dd050762bb9704ae9cd996ca45532a98f81d817e`. Retained provenance hashes identify upstream inputs and are not runtime configuration.

See the accompanying upstream notices. No cloud inference service or upstream embedding-size claim applies to this Nanobe package.