VideoMAE_Base_K400

VideoMAE Base is a self-supervised video transformer pretrained on Kinetics-400. This EIDORA package converts the pretrained encoder into a 768-dimensional normalized clip embedding.

This ONNX package was produced by EIDORA from the upstream model listed below. EIDORA is not the original model creator.

Best for

• Clip-level video similarity and retrieval.
• Clustering video clips using both spatial and temporal visual information.
• Comparing motion-aware video representations with frame-based image embeddings.
• Exploratory analysis of video collections where text alignment is not required.

Not ideal for

• Text-to-video retrieval because this checkpoint is not language-aligned.
• Audio analysis because the model uses only visual frames.
• Representing an entire long video with a single pass; the ONNX model consumes one 16-frame clip at a time.
• Assuming Kinetics-derived similarity directly represents historical, cultural, or semantic similarity.

Compute tier

medium — Provisional until EIDORA reference CPU, CUDA, and CoreML benchmarks are recorded.

Inputs

• Modality: video
• Tensor layout: NTCHW
• Input tensor: [batch, 16, 3, 224, 224]
• Color space: RGB
• Type: float32
• Value range: [0.0, 1.0]

Output

• Name: embedding
• Shape: [batch, 768]
• Type: float32
• Normalized: true
• Similarity: cosine
• Pooling/adaptation: mean of final-layer VideoMAE spatiotemporal tokens

Usage in EIDORA

Use eidora-model.yaml as the runtime contract.

Provider compatibility is recorded under validation/providers/. A provider is considered validated only when its result reports status: passed for the matching model.onnx SHA256.

Preprocessing summary

• Video decoding: outside ONNX as RGB
• Temporal input: exactly 16 frames; frame selection is performed outside ONNX
• Resize: shorter edge to 224 using bilinear
• Center crop: 224 x 224 outside ONNX
• Rescale: 1/255 outside ONNX
• Mean: [0.485, 0.456, 0.406]
• Std: [0.229, 0.224, 0.225]
• Mean/std normalization inside ONNX: true

Training data and provenance

• Base checkpoint: MCG-NJU/videomae-base
• Checkpoint source: https://huggingface.co/MCG-NJU/videomae-base
• Training data: Kinetics-400, self-supervised pretraining for 1600 epochs
• Upstream license note: CC-BY-NC-4.0

Authorship and citation

• Original model: VideoMAE Base
• Paper: VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training
• Paper URL: https://proceedings.neurips.cc/paper_files/paper/2022/hash/416f9cb3276121c42eebb86352a4354a-Abstract-Conference.html
• Official implementation: https://github.com/MCG-NJU/VideoMAE

@inproceedings{tong2022videomae,
  title={VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training},
  author={Tong, Zhan and Song, Yibing and Wang, Jue and Wang, Limin},
  booktitle={Advances in Neural Information Processing Systems},
  volume={35},
  pages={10078--10093},
  year={2022},
  doi={10.52202/068431-0732}
}

Evaluation

Conversion parity is stored in conversion_report.json. Provider checks are stored in validation/providers/. A release is complete only after CPU, CUDA and CoreML tests pass for the exact model hash.

Limitations and safety

This checkpoint was pretrained on Kinetics-400 and therefore reflects the visual and action distribution of that dataset. It has no audio or language alignment. EIDORA converts the encoder output into one 768-dimensional embedding by mean-pooling the final spatiotemporal tokens and applying L2 normalization. The ONNX model accepts exactly 16 RGB frames at 224 x 224; decoding, temporal frame selection, resizing, cropping, and rescaling are performed outside the graph. The upstream checkpoint is licensed CC-BY-NC-4.0, so its non-commercial restriction must be considered before redistribution or use.

License and attribution

Based on the VideoMAE Base checkpoint released by MCG-NJU. The upstream checkpoint is licensed under CC-BY-NC-4.0.

Versioning

• Package version: 0.1.0
• ONNX opset: 17
• Exporter: eidora-onnx 1.0.0