Blame for README.md

---
pipeline_tag: feature-extraction
tags:
- eidora
- eidora-model-zoo
- onnx
- onnxruntime
- embeddings
- video
- modality:video
- videomae
license: cc-by-nc-4.0
base_model: MCG-NJU/videomae-base
---

# VideoMAE Base — Video Embeddings

VideoMAE Base is a self-supervised video transformer pretrained on Kinetics-400. This EIDORA model provides normalized 768-dimensional clip-level video features.

## Training Data and Checkpoint Provenance

- **Architecture:** VideoMAE Base masked-autoencoder transformer for spatiotemporal video features.
- **Original checkpoint:** [MCG-NJU/videomae-base](https://huggingface.co/MCG-NJU/videomae-base).
- **Pinned revision:** `dc740ceda42fce44faed2ea03c6d447db72f6af9`.
- **Training data:** Kinetics-400, self-supervised pretraining for 1600 epochs
- **Training method:** Self-supervised masked video pretraining on Kinetics-400.
- **Source:** VideoMAE Base (original author attribution retained).

## Model Overview

- **Model:** VideoMAE Base — Video Embeddings
- **EIDORA identifier:** `EIDORA/VideoMAE_Base_K400`
- **Modality:** Video
- **Embedding dimensions:** 768
- **Embedding normalization:** L2-normalized
- **Similarity measure:** Cosine similarity
- **Package schema:** Version 3
- **Model version:** `0.1.0`

## EIDORA Use and Supported Modalities

**Supported modality:** Video only.

Researchers can use this package to:

- Compare short video clips by their visual representations.
- Explore clustering and retrieval in video collections.
- Study clip-level visual variation alongside frame-based image embeddings.

The VideoMAE checkpoint has no separate EIDORA image/text/multimodal variants in the captured public catalogue.

## Input and Preprocessing

EIDORA decodes a video and prepares a sequence of frames before ONNX inference:

1. Select a **2-second** excerpt using the **beginning** mode by default.
2. Sample **16** RGB frames.
3. Resize the shorter edge to **224** pixels and center-crop to **224 × 224** using **bilinear** interpolation.
4. Rescale by `0.00392156862745098`; the configured channel mean is `[0.0, 0.0, 0.0]` and standard deviation `[1.0, 1.0, 1.0]`.
5. Arrange frames as `BTCHW` and pass a `float32` tensor shaped `[batch, 16, 3, 224, 224]`.

Audio, captions and transcripts are not inputs to this model.

## Output Embeddings

The ONNX output is `float32` with shape `['batch', 768]`. Each vector has **768 dimensions**, is **L2-normalized**, and is intended for **cosine similarity** comparisons.

EIDORA samples 16 video frames, averages the final-layer spatiotemporal token features and applies L2 normalization.

## Runtime and Compatibility

The package uses `model.onnx` and schema-v3 runtime declarations for video preparation and embedding output.

It declares **`CPUExecutionProvider`**. Actual execution availability depends on the installed ONNX Runtime and EIDORA Desktop environment. `config.yaml` is authoritative for graph inputs, preprocessing, provider and embedding dimensions.

## Limitations

- This model uses video frames only; it does not analyze audio, text or narrated speech.
- The source training set is action-oriented and may differ from cultural or historical footage.
- The checkpoint license restricts use to non-commercial contexts.

Results should be evaluated against the intended research data and questions.

## Research Context and References

### Main paper

**VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training**

Paper: https://proceedings.neurips.cc/paper_files/paper/2022/hash/416f9cb3276121c42eebb86352a4354a-Abstract-Conference.html

The publication describes the upstream learning method and original model. The embeddings returned by EIDORA additionally depend on the specific preprocessing and output adaptation documented here.

### Primary citation (BibTeX)

```bibtex
@inproceedings{tong2022videomae,
  title={VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training},
  author={Tong, Zhan and Song, Yibing and Wang, Jue and Wang, Limin},
  booktitle={Advances in Neural Information Processing Systems},
  volume={35},
  pages={10078--10093},
  year={2022},
  doi={10.52202/068431-0732}
}
```

## License and Attribution

The upstream MCG-NJU VideoMAE checkpoint is licensed **CC BY-NC 4.0**. It is restricted to non-commercial uses; converting the model to ONNX does not remove those restrictions. See the included `LICENSE` and upstream checkpoint. EIDORA credits the VideoMAE authors.

See the `LICENSE` file included in this repository.

## Package Information

- **Model repository:** `EIDORA/VideoMAE_Base_K400`
- **Package version:** `0.1.0`
- **Package schema:** `3`
- **EIDORA configuration:** `config.yaml`
- **ONNX graph(s):** `model.onnx`
- **Supported modality:** Video
- **License identifier:** `cc-by-nc-4.0`
- **Declared provider:** CPUExecutionProvider
- **Required supplementary files:** `LICENSE`

The configuration defines the authoritative model-facing runtime contract. Other packaged assets, where present, do not introduce additional inference modalities.