README.md

metadata

pipeline_tag: feature-extraction
tags:
  - eidora
  - eidora-model-zoo
  - onnx
  - onnxruntime
  - embeddings
  - video
  - modality:video
  - videomae
license: cc-by-nc-4.0
base_model: MCG-NJU/videomae-base

VideoMAE Base — Video Embeddings

VideoMAE Base is a self-supervised video transformer pretrained on Kinetics-400. This EIDORA model provides normalized 768-dimensional clip-level video features.

Training Data and Checkpoint Provenance

• Architecture: VideoMAE Base masked-autoencoder transformer for spatiotemporal video features.
• Original checkpoint: MCG-NJU/videomae-base.
• Pinned revision: dc740ceda42fce44faed2ea03c6d447db72f6af9.
• Training data: Kinetics-400, self-supervised pretraining for 1600 epochs
• Training method: Self-supervised masked video pretraining on Kinetics-400.
• Source: VideoMAE Base (original author attribution retained).

Model Overview

• Model: VideoMAE Base — Video Embeddings
• EIDORA identifier: EIDORA/VideoMAE_Base_K400
• Modality: Video
• Embedding dimensions: 768
• Embedding normalization: L2-normalized
• Similarity measure: Cosine similarity
• Package schema: Version 3
• Model version: 0.1.0

EIDORA Use and Supported Modalities

Supported modality: Video only.

Researchers can use this package to:

• Compare short video clips by their visual representations.
• Explore clustering and retrieval in video collections.
• Study clip-level visual variation alongside frame-based image embeddings.

The VideoMAE checkpoint has no separate EIDORA image/text/multimodal variants in the captured public catalogue.

Input and Preprocessing

EIDORA decodes a video and prepares a sequence of frames before ONNX inference:

1. Select a 2-second excerpt using the beginning mode by default.
2. Sample 16 RGB frames.
3. Resize the shorter edge to 224 pixels and center-crop to 224 × 224 using bilinear interpolation.
4. Rescale by 0.00392156862745098; the configured channel mean is [0.0, 0.0, 0.0] and standard deviation [1.0, 1.0, 1.0].
5. Arrange frames as BTCHW and pass a float32 tensor shaped [batch, 16, 3, 224, 224].

Audio, captions and transcripts are not inputs to this model.

Output Embeddings

The ONNX output is float32 with shape ['batch', 768]. Each vector has 768 dimensions, is L2-normalized, and is intended for cosine similarity comparisons.

EIDORA samples 16 video frames, averages the final-layer spatiotemporal token features and applies L2 normalization.

Runtime and Compatibility

The package uses model.onnx and schema-v3 runtime declarations for video preparation and embedding output.

It declares CPUExecutionProvider. Actual execution availability depends on the installed ONNX Runtime and EIDORA Desktop environment. config.yaml is authoritative for graph inputs, preprocessing, provider and embedding dimensions.

Limitations

• This model uses video frames only; it does not analyze audio, text or narrated speech.
• The source training set is action-oriented and may differ from cultural or historical footage.
• The checkpoint license restricts use to non-commercial contexts.

Results should be evaluated against the intended research data and questions.

Research Context and References

Main paper

VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training

Paper: https://proceedings.neurips.cc/paper_files/paper/2022/hash/416f9cb3276121c42eebb86352a4354a-Abstract-Conference.html

The publication describes the upstream learning method and original model. The embeddings returned by EIDORA additionally depend on the specific preprocessing and output adaptation documented here.

Primary citation (BibTeX)

@inproceedings{tong2022videomae,
  title={VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training},
  author={Tong, Zhan and Song, Yibing and Wang, Jue and Wang, Limin},
  booktitle={Advances in Neural Information Processing Systems},
  volume={35},
  pages={10078--10093},
  year={2022},
  doi={10.52202/068431-0732}
}

License and Attribution

The upstream MCG-NJU VideoMAE checkpoint is licensed CC BY-NC 4.0. It is restricted to non-commercial uses; converting the model to ONNX does not remove those restrictions. See the included LICENSE and upstream checkpoint. EIDORA credits the VideoMAE authors.

See the LICENSE file included in this repository.

Package Information

• Model repository: EIDORA/VideoMAE_Base_K400
• Package version: 0.1.0
• Package schema: 3
• EIDORA configuration: config.yaml
• ONNX graph(s): model.onnx
• Supported modality: Video
• License identifier: cc-by-nc-4.0
• Declared provider: CPUExecutionProvider
• Required supplementary files: LICENSE

The configuration defines the authoritative model-facing runtime contract. Other packaged assets, where present, do not introduce additional inference modalities.