fusion-embedding-2-tactus

Tactus is the tactile sensor pack for Eximius Labs' fusion-embedding stack. It maps a short window of pressure-array frames (a 32x32 taxel grid, the signal class produced by resistive/FSR gloves, e-skins, and instrumented robot hands) into the Qwen3-VL-Embedding-2B text embedding space, so touch becomes searchable in plain language: recognition is a text query, not a trained classifier head.

Tactus reads low-dimensional pressure arrays, not optical tactile images. Optical sensors (GelSight, DIGIT) already have strong models (TVL, UniTouch, Sparsh); the cheap, widely-shipped resistive arrays did not. To our knowledge Tactus is the first open model to put this sensor class in a text-aligned, cross-modal embedding space.

Tactus is part of the fusion-embedding family alongside Tremor (motion) and Ember (thermal). Its embeddings target the canonical readout of fusion-embedding-2, so tactile windows are directly comparable to that model's text, image, video, and audio in one 2048-d space, and drop into the Engram memory layer (pip install engram-robomem) as a first-class sense.

GitHub | fusion-embedding-2 | Live playground | Family report (arXiv:2607.18666) | Tactus report: arXiv link lands with this week's submission

Model Overview

Tactus architecture: calibrated pressure windows pass through an MAE-pretrained per-frame trunk, a learned frame fusion, and a trained projector into the fusion-embedding shared space, where touch becomes searchable in natural language alongside every other modality

Tactus is a trained CNN trunk plus projector over pressure windows. Each 32x32 frame passes through a ResNet-18-width trunk (3x3 stem, four stages; 32x32 -> 4x4 spatial map); the K frames of a grasp window are fused by a learned 1x1 convolution over their concatenated feature maps, pooled, and projected into the frozen base's 2048-d text space. The trunk is initialized by masked-autoencoder pretraining (mask 0.6, per-patch normalized targets) on 144k unlabeled STAG-family pressure frames, then fine-tuned contrastively against canonical text embeddings of natural grasp phrases.

The design choice that matters is the data path: pressure is normalized with the sensor's own calibration affine (clip((raw - 500) / 150, 0, 1), the STAG reference preprocessing), and pretraining stays same-sensor. In our ablations, correct normalization and same-sensor MAE were worth more than every architecture change combined, while cross-sensor pretraining pooling gave nothing, consistent with published findings (HTT, TacVerse).

Feature	Value
Parameters	~2.06B frozen Qwen base (text side); 13.5M trained (trunk + projector)
Modality	tactile pressure (32x32 taxel array; 548 active sensors in the training glove)
Supported tasks	`zero-shot object recognition from touch`, `text -> touch retrieval`
Input	one grasp window `[F, 32, 32]` (F frames, any F; trained at K=8) or a single frame
Input scale	STAG calibration affine `clip((raw - 500) / 150, 0, 1)`; uint8 0-255 maps accepted
Embedding dimension	2048 (canonical whitened readout; directly comparable across modalities)
Pooling strategy	last-token pooling (text side)
Base model	Qwen/Qwen3-VL-Embedding-2B via fusion-embedding-2-2b-preview (frozen)
Pretraining	same-sensor MAE, 144k frames incl. unlabeled; supervised test frames excluded
Trained components	trunk + conv frame-fusion + projector, 13.5M; shipped as `model.safetensors`
Distribution	~65 MB trained head; the frozen base downloads from its own repository

See it in action

Real held-out grasps, recognized from pressure alone. Each panel is a genuine STAG test frame (the most active frame of that class in the held-out split, by total pressure) with the text query the model matches it against: no camera, no trained classifier head. Across the full test split the model averages 0.77 top-1 and 0.94 top-3 over 27 such queries.

Four real held-out STAG test pressure maps with their text queries: a mug, scissors, a full can, safety glasses, each recognized from the 32x32 pressure pattern alone

Training and Evaluation

Tactus trains in two stages on the STAG datasets (Sundaram et al., Nature 2019): a masked-autoencoder pretrain over every STAG-family pressure frame (classification + blindfolded + weights + handposes, 144k frames including unlabeled ones, supervised test frames excluded), then contrastive fine-tuning of the whole head against canonical text embeddings of grasp phrases, with STAG-style cluster sampling (each training window draws diverse frames from across a recording rather than consecutive near-duplicates).

Evaluation is 27-way object recognition on fully held-out test recordings, scored as cosine ranking against text queries (open-vocabulary; the model never trains a classifier head).

	top-1 (27-way)	top-3	recording-level top-1
This checkpoint	0.817	0.951	0.741
Recipe mean (4 independent runs)	0.771 +/- 0.062	0.935	0.722
Training from scratch (no MAE), mean of 3	0.705	0.905	0.691
STAG 2019 supervised closed-set CNN	0.76	-	-
chance	0.037	0.111	0.037

Interpreting these numbers: the recipe's mean exceeds the original paper's supervised CNN while performing a harder task (open-vocabulary text queries against a frozen language space, versus a 27-way trained classifier), though by less than one standard error; we describe the result as matching to exceeding the original baseline, with best runs at 0.83, rather than claiming a definitive margin. Top-3 accuracy is stable across every run. Our evaluation mirrors STAG's cluster-sampling test protocol but is not their byte-identical harness. Same-sensor MAE pretraining accounts for about +7 points over training from scratch. Full recipe, ablations, and negative results: results.json and the GitHub repository.

Usage

Requirements

torch (CUDA recommended), numpy, safetensors
pip install fusion-embedding[hf] for the text side (the canonical whitened readout Tactus was trained against; embedding text any other way will misrank)
The frozen base downloads from EximiusLabs/fusion-embedding-2-2b-preview.

via inference.py (this repository)

import numpy as np
from inference import TactusEmbedder

ta = TactusEmbedder.from_pretrained("EximiusLabs/fusion-embedding-2-tactus",
                                    revision="v0.1-preview")

# a grasp window: [F, 32, 32] pressure frames (uint8 0-255 or floats in [0, 1]);
# for raw sensor counts pass raw="stag" to apply the calibration affine
window = np.load("grasp.npy")

for text, score in ta.rank(window, ["a mug", "scissors", "a full soda can", "an empty hand"]):
    print(f"{score:+.3f}  {text}")

# or embed both sides into the shared space directly
p = ta.embed_pressure(window)          # 2048-d, L2-normalized
t = ta.embed_text(["holding a mug"])   # canonical text embedding, same space

Pressure embeddings land in the same space as fusion-embedding-2's text, image, video, and audio, and as Tremor's motion, so cross-modal search over a robot session works out of the box through Engram. Match text against pressure through this API rather than embedding text with the raw base model; Tactus was trained against the canonical whitened readout, and other text paths will misrank.

Related models

Tactus joins the fusion-embedding sense packs, all built on fusion-embedding-2:

Model	Sense	Signal
This model	touch	32x32 pressure/taxel arrays
fusion-embedding-2-tremor	motion	3-axis accelerometer windows
fusion-embedding-2-tremor-g1	motion (Unitree G1 head)	robot IMU
fusion-embedding-2-ember	heat	thermal infrared images

All packs embed into one 2048-d space, so a query can match across senses. The Engram memory layer (pip install engram-robomem) wires them into a searchable robot session memory with temporal reasoning.

License

The trained weights in this repository are released under CC-BY-NC-4.0 (non-commercial).

This reflects the training data's lineage: Tactus is trained on the STAG datasets, which are released for non-commercial research use. A commercially-clean retrain (on permissively licensed pressure corpora) is future work; a commercial license may follow.

Limitations

Run-to-run variance. The training recipe's top-1 varies +/-0.06 across seeds (0.70-0.83 over four runs). The released checkpoint is a strong draw, and the mean is reported alongside it. Seed stabilization is active work.
One sensor family. Trained on one glove (STAG's 32x32 grid, 548 taxels). Our cross-sensor experiments show transfer to other taxel geometries needs fine-tuning, not zero-shot use; the input path accepts any [F,32,32] window, and other resolutions must be resampled.
27-object vocabulary at eval. Open-vocabulary means text queries, not tested open-set generalization to arbitrary unseen object categories; treat novel-category recognition as unvalidated.
Research preview. Not a production classifier. The intended use is language-addressable touch inside a multimodal memory, not high-stakes recognition.
English text only, through the canonical readout (fusion-embedding package); do not embed text with the raw base model.

Citation

If you use Tactus, please cite this repository and the dataset it builds on:

@misc{tactus2026,
  title  = {Tactus: a tactile pressure sensor pack for the fusion-embedding space},
  author = {Tonmoy, Abdul Basit},
  year   = {2026},
  note   = {Eximius Labs. Model weights CC-BY-NC-4.0.},
  url    = {https://huggingface.co/EximiusLabs/fusion-embedding-2-tactus}
}

Tactus trains on STAG (Sundaram et al., Learning the signatures of the human grasp using a scalable tactile glove, Nature 2019); please cite that work when using the benchmark numbers. The text space is Qwen3-VL-Embedding-2B.