| --- |
| pipeline_tag: feature-extraction |
| tags: |
| - eidora |
| - eidora-model-zoo |
| - onnx |
| - onnxruntime |
| - embeddings |
| - video |
| - modality:video |
| - videomae |
| license: cc-by-nc-4.0 |
| base_model: MCG-NJU/videomae-base |
| --- |
| |
| # VideoMAE Base — Video Embeddings |
|
|
| VideoMAE Base is a self-supervised video transformer pretrained on Kinetics-400. This EIDORA model provides normalized 768-dimensional clip-level video features. |
|
|
| ## Training Data and Checkpoint Provenance |
|
|
| - **Architecture:** VideoMAE Base masked-autoencoder transformer for spatiotemporal video features. |
| - **Original checkpoint:** [MCG-NJU/videomae-base](https://huggingface.co/MCG-NJU/videomae-base). |
| - **Pinned revision:** `dc740ceda42fce44faed2ea03c6d447db72f6af9`. |
| - **Training data:** Kinetics-400, self-supervised pretraining for 1600 epochs |
| - **Training method:** Self-supervised masked video pretraining on Kinetics-400. |
| - **Source:** VideoMAE Base (original author attribution retained). |
|
|
| ## Model Overview |
|
|
| - **Model:** VideoMAE Base — Video Embeddings |
| - **EIDORA identifier:** `EIDORA/VideoMAE_Base_K400` |
| - **Modality:** Video |
| - **Embedding dimensions:** 768 |
| - **Embedding normalization:** L2-normalized |
| - **Similarity measure:** Cosine similarity |
| - **Package schema:** Version 3 |
| - **Model version:** `0.1.0` |
|
|
| ## EIDORA Use and Supported Modalities |
|
|
| **Supported modality:** Video only. |
|
|
| Researchers can use this package to: |
|
|
| - Compare short video clips by their visual representations. |
| - Explore clustering and retrieval in video collections. |
| - Study clip-level visual variation alongside frame-based image embeddings. |
|
|
| The VideoMAE checkpoint has no separate EIDORA image/text/multimodal variants in the captured public catalogue. |
|
|
| ## Input and Preprocessing |
|
|
| EIDORA decodes a video and prepares a sequence of frames before ONNX inference: |
|
|
| 1. Select a **2-second** excerpt using the **beginning** mode by default. |
| 2. Sample **16** RGB frames. |
| 3. Resize the shorter edge to **224** pixels and center-crop to **224 × 224** using **bilinear** interpolation. |
| 4. Rescale by `0.00392156862745098`; the configured channel mean is `[0.0, 0.0, 0.0]` and standard deviation `[1.0, 1.0, 1.0]`. |
| 5. Arrange frames as `BTCHW` and pass a `float32` tensor shaped `[batch, 16, 3, 224, 224]`. |
|
|
| Audio, captions and transcripts are not inputs to this model. |
|
|
| ## Output Embeddings |
|
|
| The ONNX output is `float32` with shape `['batch', 768]`. Each vector has **768 dimensions**, is **L2-normalized**, and is intended for **cosine similarity** comparisons. |
|
|
| EIDORA samples 16 video frames, averages the final-layer spatiotemporal token features and applies L2 normalization. |
|
|
| ## Runtime and Compatibility |
|
|
| The package uses `model.onnx` and schema-v3 runtime declarations for video preparation and embedding output. |
|
|
| It declares **`CPUExecutionProvider`**. Actual execution availability depends on the installed ONNX Runtime and EIDORA Desktop environment. `config.yaml` is authoritative for graph inputs, preprocessing, provider and embedding dimensions. |
|
|
| ## Limitations |
|
|
| - This model uses video frames only; it does not analyze audio, text or narrated speech. |
| - The source training set is action-oriented and may differ from cultural or historical footage. |
| - The checkpoint license restricts use to non-commercial contexts. |
|
|
| Results should be evaluated against the intended research data and questions. |
|
|
| ## Research Context and References |
|
|
| ### Main paper |
|
|
| **VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training** |
|
|
| Paper: https://proceedings.neurips.cc/paper_files/paper/2022/hash/416f9cb3276121c42eebb86352a4354a-Abstract-Conference.html |
| |
| The publication describes the upstream learning method and original model. The embeddings returned by EIDORA additionally depend on the specific preprocessing and output adaptation documented here. |
| |
| ### Primary citation (BibTeX) |
| |
| ```bibtex |
| @inproceedings{tong2022videomae, |
| title={VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training}, |
| author={Tong, Zhan and Song, Yibing and Wang, Jue and Wang, Limin}, |
| booktitle={Advances in Neural Information Processing Systems}, |
| volume={35}, |
| pages={10078--10093}, |
| year={2022}, |
| doi={10.52202/068431-0732} |
| } |
| ``` |
| |
| ## License and Attribution |
| |
| The upstream MCG-NJU VideoMAE checkpoint is licensed **CC BY-NC 4.0**. It is restricted to non-commercial uses; converting the model to ONNX does not remove those restrictions. See the included `LICENSE` and upstream checkpoint. EIDORA credits the VideoMAE authors. |
| |
| See the `LICENSE` file included in this repository. |
| |
| ## Package Information |
| |
| - **Model repository:** `EIDORA/VideoMAE_Base_K400` |
| - **Package version:** `0.1.0` |
| - **Package schema:** `3` |
| - **EIDORA configuration:** `config.yaml` |
| - **ONNX graph(s):** `model.onnx` |
| - **Supported modality:** Video |
| - **License identifier:** `cc-by-nc-4.0` |
| - **Declared provider:** CPUExecutionProvider |
| - **Required supplementary files:** `LICENSE` |
| |
| The configuration defines the authoritative model-facing runtime contract. Other packaged assets, where present, do not introduce additional inference modalities. |
| |
