Jeong-USYD/owac-j5-raw-encoder

OWAC J5 RAW Encoder — EMA step 12,818

Self-supervised RAW visual encoder trained by the OWAC project. This is the original J5 step-12,818 EMA encoder used to initialize the subsequent C1 policies. It is not a C1/D17 policy-finetuned encoder or a D20 scalar-measurement encoder.

Model

ViT-S/16 architecture: width 384, 12 layers, 6 attention heads, 4 register tokens. A learned radiometric response maps three input channels to nine channels (original magnitude, asinh-compressed response, and relative channel response). The implementation uses the Transformers DINOv3 ViT architecture, without loading pretrained DINOv3 weights.

J5 inherits the project's RAW-only J4 EMA encoder, with J3 RAW features used as a frozen spatial-relation reference. Training combines DINO/iBOT-style self-supervision, exposure consistency, spatial relations, feature diversity, and past-frame soft correspondence. No RGB teacher, external pretrained visual weights, action labels, or language labels are used in J5 training. Training was stopped and saved at 12,818 of the planned 30,000 J5 updates.

Input and output

• Input: float tensor [B, 3, H, W] of linear three-channel pseudo-RAW; the standard demonstrated size is 256 × 256, patch size 16.
• Match the original data convention: LIBERO stored values are divided by 255; RoboTwin floating-point RAW values are used directly. This is dataset-specific scaling, not generic sensor calibration. Do not apply RGB ImageNet normalization or clip HDR values by default.
• Output: CLS features [B, 384] and patch features [B, H/16 * W/16, 384] (256 patches at 256 × 256). Four register tokens are excluded from the patch output.
• Deployment uses a single frame; temporal correspondence was a training objective.

Usage

Download this repository and install requirements.txt. From the downloaded directory:

import json
import torch
from safetensors.torch import load_file
from modeling_raw_encoder import RawEncoder

config = json.load(open('config.json'))
encoder = RawEncoder(config).eval()
encoder.load_state_dict(load_file('model.safetensors'), strict=True)
# raw: your correctly scaled [B, 3, 256, 256] floating-point RAW tensor
with torch.inference_mode():
    cls_features, patch_features = encoder(raw)

encoder.pt contains the identical EMA tensors under encoder, step 12818, and config.model for compatibility with OWAC's J5 initialization loader. Only model configuration is included; private runtime paths, tracking configuration, optimizer state and training data are excluded. model.safetensors is the recommended tensor-only format. modeling_raw_encoder.py contains the two inference classes extracted from the original archived J5 training source. export_info.json records strict loading, tensor equality, and a synthetic shape/finite-output smoke check; this check is not a capability evaluation.

Evidence and limitations

Trained on the original LIBERO and RoboTwin subsets of RawVLA-Bench with their original splits. These observations are simulated three-channel pseudo-RAW, not native single-channel monochrome or Bayer measurements. This checkpoint does not implement the later D20 native 10-bit mono/Bayer interface. Do not replicate a mono channel and describe that as validated sensor transfer.

The original final step-12,818 checkpoint did not receive a new visual evaluation. Earlier visual probes are diagnostic and do not establish general object semantics, motion segmentation, or real-camera transfer. This encoder alone does not produce actions; downstream C1/D17 closed-loop success rates cannot be attributed to this standalone checkpoint.

Source and training record: OWAC, J5 experiment.

Public availability does not itself specify a license grant. No additional weight license is asserted in this release.