| --- |
| library_name: transformers |
| pipeline_tag: feature-extraction |
| tags: |
| - metaencoder |
| - muse_glimmer |
| - multimodal-embedding |
| - image-text-retrieval |
| - video-retrieval |
| - document-retrieval |
| --- |
| |
| # MetaEncoder v1 |
|
|
|  |
|
|
| ## Usage |
|
|
| Requires `transformers>=5.15` (the first release with `muse_glimmer`); see `requirements.txt`. One 80 GB GPU holds |
| the model in bf16. |
|
|
| ```python |
| from modeling_metaencoder import MetaEncoder |
| |
| model = MetaEncoder.from_pretrained("facebook/meta-encoder-v1") |
| ``` |
|
|
| `MetaEncoder` has three calls. Callers pass content and task descriptions; the wrapper writes every prompt itself, in |
| the layout the encoder was trained on (see [Prompt formatting](#prompt-formatting)). |
|
|
| | Call | Use it to | Returns | |
| | :--- | :--- | :--- | |
| | `encode(inputs, instruction=None)` | embed inputs to store, cluster or compare | `(n, 6656)` float32 tensor, L2-normalised | |
| | `retrieve(query, corpus, top_k=None, query_instruction=None, corpus_instruction=None)` | rank a corpus against a query | `list[Hit]`, best first | |
| | `decide(state, questions, images=None, video=None)` | answer Jev-style questions about a state | Jev answers with probabilities | |
|
|
| ### Inputs |
|
|
| `encode` and `retrieve` take *items*. An item is a string, a PIL image, or a dict with any of `text`, `image`, |
| `images`, `video` (a list of frames) and `instruction`. The instruction says what the embedding is for, e.g. |
| "Represent the given image." or "Find a document image that matches the given query."; the same input embeds |
| differently under different instructions, and queries and documents usually take different ones. |
|
|
| ### 1. Encode |
|
|
| ```python |
| from PIL import Image |
| |
| docs = model.encode([{"image": Image.open("chart.png")}, {"image": Image.open("slide.png")}], |
| instruction="Understand the content of the provided document image.") |
| MetaEncoder.similarity(docs, docs) # cosine similarity matrix |
| ``` |
|
|
| `inputs` is one item or a list of them; `instruction` applies to the items that carry none of their own. |
|
|
| ### 2. Retrieve |
|
|
| ```python |
| hits = model.retrieve("Which quarter had the highest revenue?", docs, top_k=1, |
| query_instruction="Find a document image that matches the given query.") |
| hits[0].index, hits[0].score, hits[0].item # position in the corpus, cosine similarity, the corpus item |
| ``` |
|
|
| `corpus` is a list of items, or the tensor `encode` returned for them: encode a large corpus once and search the |
| tensor repeatedly. |
|
|
| ### 3. Decide |
|
|
| `decide` takes a Jev decision request, a state and named questions, and answers each question: |
|
|
| ```python |
| result = model.decide( |
| {"query": "used iphone 15"}, |
| { |
| "shopping": {"type": "noul", "instructions": "Is the user shopping for a product?"}, |
| "topic": {"type": "choice", "instructions": "Which topic is the query about?", |
| "criteria": {"electronics": "", "food": "", "travel": "", "other": ""}}, |
| "intent": {"type": "score", "instructions": "How strong is the purchase intent?", |
| "criteria": ["none", "weak", "moderate", "strong"]}, |
| }, |
| ) |
| result["questions"]["shopping"] # {"value": True, "probabilities": {"false": 0.016, "true": 0.984}, |
| # "confidence": 0.881, "scores": {...}} |
| result["questions"]["topic"]["value"], result["questions"]["intent"]["value"] # "electronics", 1.89 |
| ``` |
|
|
| - `state`: what the questions are about. A string, or any JSON value (an object of fields, a list of records). |
| - `images`, `video`: the state's media: a PIL image or a list of them, and a list of video frames. |
| - `questions`: `{name: {"type", "instructions", "criteria"}}`, with `instructions` the question itself: |
|
|
| | `type` | `criteria` | `value` | `probabilities` keys | |
| | :--- | :--- | :--- | :--- | |
| | `noul` | optional `{"true": ..., "false": ...}` descriptions | `True` / `False` | `"false"`, `"true"` | |
| | `choice` | `{option: description}`, 2 to 26 options; descriptions may be empty | the chosen option | the options | |
| | `score` | the ordered list of level labels, e.g. `["none", "weak", "strong"]` | expected level index (float) | `"0"` .. `"n-1"` | |
|
|
| Each answer also has `confidence` (1 minus the normalised entropy of `probabilities`) and `scores` (the raw cosine |
| similarities). Questions are answered independently; for a question that only applies given another's answer, |
| decide the first and ignore the second as needed. |
|
|
| `decide_batch(states, questions, images=None, video=None)` answers the same questions for many states (one |
| `images` / `video` entry per state). |
|
|
| Probabilities are `softmax(score / 0.02)`, the contrastive training temperature. To fit the temperature to your |
| data, decide a labelled set and call `model.calibrate(answers, labels)` with the per-question answers |
| (`result["questions"][name]`) and their correct keys (the option, `True`/`False`, or the level index); later |
| `decide` calls use the fitted temperature. |
|
|
|
|
| ## Prompt formatting |
|
|
| The wrapper formats every input as in training, so callers never write prompts: |
|
|
| - **System turn** (`chat_template.jinja`): the fixed header ending in "Represent the user's input.", then the item's |
| instruction after a blank line when it has one. The date line is fixed, so an input embeds the same on any day. |
| - **User turn**: images and video frames first, then the text. Images keep their native resolution, snapped to a |
| 28-pixel grid, up to 4096 merged tokens. Video is encoded frame by frame; more than 8 frames are subsampled |
| uniformly to 8, as in training. |
| - **Pooling**: the last position, the user turn's `<|eot|>`, with left padding. Inputs are never truncated (that |
| would drop the pooled token); training sequences were at most 16,384 tokens. |
|
|
| `decide` turns each question into a closed-set decision row of training, scored as retrieval over its options: |
|
|
| | Part | Rendered as | |
| | :--- | :--- | |
| | query text | the state, a blank line, `Task: {instructions}`, then the options: `Options: a, b, c`, or one `- option` per line when an option is long or punctuated | |
| | state | a string as is; a JSON value with `json.dumps` | |
| | query instruction | "Read the task definition and select the option it calls for." for a string state; "Answer the question about the scenario by selecting the correct option." for a JSON state | |
| | options | `noul`: `No, Yes`; `choice`: the criteria keys; `score`: the level labels | |
| | candidates | one per option: the option, then its description (if any) on the next line | |
| | candidate instruction | `Option for the task: {instructions}` when every option is a short label (at most 5 words), else none | |
|
|
| Option descriptions are optional. The yes/no, choice and score rows of training used bare option labels; candidate |
| descriptions were trained only as records of the entities that options name. |
|
|
| ### Without the wrapper |
|
|
| `modeling_metaencoder.py` only formats the prompt and pools; the same embedding with public classes directly: |
|
|
| ```python |
| import torch, torch.nn.functional as F |
| from safetensors.torch import load_file |
| from transformers import AutoModel, AutoProcessor |
| |
| model = AutoModel.from_pretrained(repo, dtype=torch.bfloat16, device_map="auto").eval() # MuseGlimmerModel |
| processor = AutoProcessor.from_pretrained(repo) |
| processor.tokenizer.padding_side = "left" |
| head = load_file(f"{repo}/embedding_head.safetensors")["embedding_head.weight"].to(model.device, torch.bfloat16) |
| |
| messages = [{"role": "system", "content": "Represent the given image."}, # optional instruction |
| {"role": "user", "content": [{"type": "image"}, {"type": "text", "text": "a dog"}]}] |
| text = processor.apply_chat_template(messages, tokenize=False) |
| inputs = processor(text=[text], images=[image], add_special_tokens=False, max_image_tokens=4096, |
| return_tensors="pt").to(model.device) |
| with torch.no_grad(): |
| hidden = model(**inputs).last_hidden_state[:, -1] |
| embedding = F.normalize((hidden @ head.T).float(), dim=-1) |
| |
