Add InternVideo2-CLIP-anomaly-detection v1.0 (TAO fine-tune, research release)

Adds the InternVideo2-CLIP-anomaly-detection v1.0 release files: the TAO `video_clip` fine-tune of OpenGVLab InternVideo2-CLIP-L14 (vision LoRA on the last 12 blocks, frozen MobileCLIP-B-LT text encoder, preservation regularization) trained on the PhysicalAI-Event-Videos train subset, released for non-commercial research under the NVIDIA Internal Scientific Research and Development Model License already set on this repository.

**Contents (19 files, 4.9 GB)**
- `model.safetensors` — fine-tuned weights with the 24 LoRA modules merged (486 tensors, 372,752,643 parameters, FP32, TAO state-dict key names) + `model_config.yaml`.
- `model_epoch_002_step_04011.no_teacher.pth` — TAO checkpoint, LoRA unmerged, teacher/optimizer state removed + `train_experiment.yaml`.
- `onnx_opset23/` — FP32 vision and text towers (opset 23, dynamic batch, `.onnx` + `.onnx.data`), ModelOpt AutoCast FP16 towers for strongly typed TensorRT builds, deploy config, export spec, CLIP BPE tokenizer.
- `README.md` — model card (research release), `MANIFEST.sha256`.

**Verification before upload**
- `model.safetensors` loads `strict=True` into a `peft`-disabled `VideoCLIPPlModel` and reproduces the merged-checkpoint reference embeddings exactly (max abs diff 0).
- FP32 ONNX towers vs the safetensors model on seeded and real-query probes: min cosine 0.99999994, max abs diff 5.7e-7.
- Checkpoint and ONNX SHA-256 values equal the internal v1.0 release manifest.

Accuracy: PhysicalAI-Event-Videos test subset event-only retrieval mAP 63.12% / Hit@1 59.11% / Hit@5 87.04% (base 54.34 / 50.69 / 78.49); Kinetics-400 zero-shot Top-1 69.19% / Top-5 89.64%. Details in the README.

Repository stays private until the release review completes. Contact: Alice Li (alicli@nvidia.com).

.gitattributes

@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+onnx_opset23/iv2clip_v1.0_step04011_opset23_text.onnx.data filter=lfs diff=lfs merge=lfs -text
+onnx_opset23/iv2clip_v1.0_step04011_opset23_vision.onnx.data filter=lfs diff=lfs merge=lfs -text

MANIFEST.sha256

@@ -0,0 +1,18 @@
+981db373f677a511f287c62dc823452267befd07e975c7e01881130e065aa4d9  README.md
+31992985a49ff1840a2c862e9fee029b57eb37b0eefcde1da09128e927e9dab6  model.safetensors
+282f74e63df566998249e4ab81ad34343a0f1a12358fc976e4d44d702d7e9070  model_config.yaml
+a7c2b4148a882497d110dc2f5a3d124efb1210303ed59daec9ccb6eb4d073c63  model_epoch_002_step_04011.no_teacher.pth
+a03f526bcf8372cf7516fe95c1d8c6eb4ec4e00b93fa5dd14dd753b838bc645e  onnx_opset23/export_spec_v1.0_step04011_separate_opset23.yaml
+b98dd5a682c3ef664382eaa6770ef89809c98e46e5e3809c98994d8331150faf  onnx_opset23/iv2clip_v1.0_step04011_opset23_config.yaml
+6e41fa1c7fb94e6b438375048603f818dc6c6b5ee35f3d7259905905a3760f0b  onnx_opset23/iv2clip_v1.0_step04011_opset23_text.autocast_fp16.onnx
+5e55caf759e88dbb85e37a8e124129070bcca2d57d85fd629aef43ce7954a51e  onnx_opset23/iv2clip_v1.0_step04011_opset23_text.onnx
+cacc6493c3a7863c1cd20bfec11600d4f2aeef953849f1cfa9fe87081887787f  onnx_opset23/iv2clip_v1.0_step04011_opset23_text.onnx.data
+9fd691f7c8039210e0fced15865466c65820d09b63988b0174bfe25de299051a  onnx_opset23/iv2clip_v1.0_step04011_opset23_tokenizer/merges.txt
+2cdb3b8331a60c92fc1e55a13e9fd61fd2293c5a51275fdcccd62b780052530e  onnx_opset23/iv2clip_v1.0_step04011_opset23_tokenizer/special_tokens_map.json
+43d4281659ce30b6aca2c7485944b3d73925bb7f87c7234daf3b6b094a6d618c  onnx_opset23/iv2clip_v1.0_step04011_opset23_tokenizer/tokenizer.json
+39bae5313c2da6467866dee758e754d69f0e302a875027b142531cb9c51c7c28  onnx_opset23/iv2clip_v1.0_step04011_opset23_tokenizer/tokenizer_config.json
+5047b556ce86ccaf6aa22b3ffccfc52d391ea4accdab9c2f2407da5b742d4363  onnx_opset23/iv2clip_v1.0_step04011_opset23_tokenizer/vocab.json
+141e726ae5dcf0bf7c3f21054e186ce8021410c403f73574e0d3dbb430be38f5  onnx_opset23/iv2clip_v1.0_step04011_opset23_vision.autocast_fp16.onnx
+63ee64deaef832b9b7ccfd340c809642f7e8285689d1a0f8cd3478527f1fb189  onnx_opset23/iv2clip_v1.0_step04011_opset23_vision.onnx
+27e70bca3f0ca169e1a6ae41fb3e1cafe6371d746f405ac0498cf54d6fc15f67  onnx_opset23/iv2clip_v1.0_step04011_opset23_vision.onnx.data
+7f3a5f6117c7226e4e237f432b68ed893e81093c77251a37b7633c0667681a96  train_experiment.yaml

README.md

@@ -3,4 +3,315 @@ license: other
 license_name: nvidia-internal-scientific-research-and-development-model-license
 license_link: >-
   https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-internal-scientific-research-and-development-model-license/
+tags:
+- nvidia
+- tao
+- video-text-retrieval
+- video-embeddings
+- physical-ai
+- anomaly-detection
+- lora
+- fine-tuned
+- pytorch
+- onnx
+- tensorrt
 ---
+
+# Model Overview
+
+### Description
+
+InternVideo2-CLIP encodes video clips and text descriptions into a shared embedding space for text-to-video retrieval and zero-shot action classification. This model card describes an NVIDIA Train, Adapt, Optimize (TAO) [`video_clip`](https://docs.nvidia.com/tao/tao-toolkit/latest/text/embedding/video_clip/overview.html) fine-tune of the OpenGVLab InternVideo2-CLIP-L14 model with a MobileCLIP-B text encoder. Fine-tuning uses vision Low-Rank Adaptation (LoRA) and preservation regularization on the **train subset of [NVIDIA PhysicalAI-Event-Videos](https://huggingface.co/datasets/nvidia/PhysicalAI-Event-Videos)**. Retrieval evaluation uses the dataset's **test subset**; Kinetics-400 is reported separately as a general-domain benchmark. This model has been post-trained on the InternVideo2 CLIP-L14 model, and use of the InternVideo2 CLIP-L14 model is governed by Apache 2.0.
+
+The model is **released for non-commercial research use only**. The output is an embedding, rather than a generated caption, bounding box, identity, or calibrated event probability. This model is for research and development only.
+
+### License/Terms of Use
+
+GOVERNING TERMS: Use of this model is governed by the [NVIDIA Internal Scientific Research and Development Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-internal-scientific-research-and-development-model-license/). **Additional Information**: [Apache 2.0](https://huggingface.co/datasets/choosealicense/licenses/blob/main/markdown/apache-2.0.md) for the InternVideo2 CLIP-L14 base model, [Apple Machine Learning Research Model License](https://github.com/apple-aiml-research/ml-mobileclip/blob/main/LICENSE_MODELS) for the MobileCLIP-B-LT text encoder, and [MIT](https://huggingface.co/datasets/choosealicense/licenses/blob/main/markdown/mit.md) for the CLIP tokenizer files. <br>
+
+Apple Machine Learning Research Model is licensed under the Apple Machine Learning Research Model License Agreement.
+
+### Deployment Geography
+
+Global
+
+### Use Case
+
+Intended users are developers evaluating video retrieval and event-search systems for physical-world video collections, including warehouse, retail, traffic, and general activity footage. A text query is compared with video embeddings to rank candidate clips. Zero-shot classification compares video embeddings with prompted action-class descriptions. The model is not for use in commercial products or services.
+
+The model is not validated as an autonomous safety decision system. Similarity scores require application-specific evaluation and should not be interpreted as proof that an event occurred.
+
+### Release Date
+
+Hugging Face 11/03/2026 via [https://huggingface.co/nvidia/InternVideo2-CLIP-anomaly-detection](https://huggingface.co/nvidia/InternVideo2-CLIP-anomaly-detection) (private until the release review completes). This model is released on Hugging Face only; it is not published on NGC.
+
+## References(s)
+
+- [NVIDIA PhysicalAI-Event-Videos](https://huggingface.co/datasets/nvidia/PhysicalAI-Event-Videos): public reference for the training and evaluation subsets and the annotation workflow.
+- [OpenGVLab InternVideo](https://github.com/OpenGVLab/InternVideo): upstream model family and implementation.
+- [InternVideo2 distillation assets](https://huggingface.co/OpenGVLab/InternVideo2_distillation_models): upstream vision and alignment weights.
+- [MobileCLIP](https://github.com/apple-aiml-research/ml-mobileclip): source of the MobileCLIP-B-LT text encoder, licensed under the [Apple Machine Learning Research Model License](https://github.com/apple-aiml-research/ml-mobileclip/blob/main/LICENSE_MODELS) (research-only).
+- [NVIDIA TAO InternVideo2-CLIP (`video_clip`) overview](https://docs.nvidia.com/tao/tao-toolkit/latest/text/embedding/video_clip/overview.html): model description and data input format for the TAO pipeline.
+- [NVIDIA TAO InternVideo2-CLIP training, evaluation, inference, and export guide](https://docs.nvidia.com/tao/tao-toolkit/latest/text/embedding/video_clip/training.html): experiment specification, `train`, `evaluate`, `inference`, and `export` subtasks, and ONNX/TensorRT usage notes.
+- [Using InternVideo2-CLIP embeddings](https://docs.nvidia.com/tao/tao-toolkit/latest/text/embedding/video_clip/applications.html): loading the exported ONNX model and using the embeddings for text-to-video retrieval, zero-shot classification, and anomaly detection.
+- [InternVideo2-CLIP with TAO Deploy](https://docs.nvidia.com/tao/tao-toolkit/latest/text/tao_deploy/video_clip.html): TensorRT engine generation and TensorRT-side evaluation and inference.
+- [NVIDIA TAO PyTorch](https://github.com/NVIDIA/tao_pytorch_backend): TAO model implementation; release-specific compatibility remains to be confirmed.
+- [Kinetics dataset](https://github.com/cvdfoundation/kinetics-dataset): action-recognition evaluation data.
+
+## Model Architecture
+
+**Architecture Type:** Transformer.<br>
+**Network Architecture:** InternVideo2-CLIP-L14 vision encoder with a frozen, unmodified Apple MobileCLIP-B-LT text encoder and a shared 512-dimensional embedding space.<br>
+**Base Model:** OpenGVLab InternVideo2-CLIP-L14.<br>
+**Number of model parameters:** Approximately **3.7 × 10^8** (373,342,467, of which 589,824 are LoRA parameters); excludes the preservation teacher and optimizer state.
+
+The vision transformer has 24 blocks, 14-pixel spatial patches, a hidden width of 1,024, and 16 attention heads. A visual projection and alignment layer map video features to the text embedding space. The text tower remains frozen during this fine-tune. LoRA targets the `qkv` and `proj` modules in the last 12 vision blocks, with rank 8, alpha 16, and dropout 0.05.
+
+The training recipe uses video-text contrastive loss, preservation regularization, bfloat16 (BF16) precision, and AdamW (learning rate 1e-3, cosine schedule, 100 warmup steps) with an effective batch of 64. Training started from the base weights, and the released checkpoint was selected on the validation set described below. The fine-tune was run with the TAO `video_clip` `train` subtask; the experiment-specification format and these recipe fields are documented in the [TAO training guide](https://docs.nvidia.com/tao/tao-toolkit/latest/text/embedding/video_clip/training.html#training-the-model). These training details do not specify deployment hardware requirements.
+
+## Input(s)
+
+**Input Type(s):** Video and text; each encoder can be used independently.<br>
+**Input Format(s):** RGB video frames and UTF-8 text strings.<br>
+**Input Parameters:** Video is a temporal sequence of two-dimensional RGB frames; text is a one-dimensional token sequence.
+
+**Other Properties Related to Input:**
+
+- The measured video protocol samples eight frames uniformly across each supplied clip and resizes each frame to 224 × 224 using bicubic interpolation. Published evaluation clips are already trimmed and receive no second temporal crop.
+- RGB values are converted to floating point in [0, 1] and normalized using mean `(0.485, 0.456, 0.406)` and standard deviation `(0.229, 0.224, 0.225)`.
+- The TAO adapter receives video tensors shaped `[batch, 8, 3, 224, 224]`; the backbone internally permutes them to channel-first video layout.
+- The CLIP tokenizer produces 77-token sequences, including special tokens, with padding/truncation to the context length. Text tensors have shape `[batch, 77]`.
+- Evaluation uses English descriptions. Multilingual quality and alternative frame counts or resolutions have not been established for this checkpoint.
+- Audio is not an input to the evaluated model.
+
+## Output(s)
+
+**Output Type(s):** Video embedding and text embedding.<br>
+**Output Format(s):** Floating-point vectors.<br>
+**Output Parameters:** One 512-dimensional vector per video clip or text input; batched shape `[batch, 512]`.<br>
+**Other Properties Related to Output:** The TAO adapter supports L2-normalized outputs. Cosine similarity ranks video-text pairs; it is not a calibrated probability. The retrieval workflow additionally produces rankings and similarity scores.
+
+Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA's hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions. CPU throughput and cross-hardware equivalence were not measured.
+
+## Software Integration
+
+**Runtime Engine(s):** PyTorch through NVIDIA TAO [`video_clip`](https://docs.nvidia.com/tao/tao-toolkit/latest/text/embedding/video_clip/training.html) (accuracy evaluation); TensorRT 10.16 with a strongly typed FP16 ONNX export (performance measurement), deployable through [TAO Deploy](https://docs.nvidia.com/tao/tao-toolkit/latest/text/tao_deploy/video_clip.html). The inspected TAO source includes inference, evaluation, and ONNX export paths. The minimum public TAO release supporting this exact checkpoint is **pending confirmation**.
+
+**Supported Hardware Microarchitecture Compatibility:** NVIDIA Ampere (A100) is verified for the reported accuracy evaluation. NVIDIA Hopper (H100), Ada Lovelace (L40S), and Blackwell (RTX PRO 6000 Server Edition, GB300) were measured for TensorRT performance.
+
+**Supported Operating System(s):** Linux in the recorded evaluation environment.
+
+Integration requires a checkpoint-compatible model configuration, the base assets, tokenizer, and the matching LoRA configuration. The training checkpoint also stores preservation-teacher state; loading or packaging must handle that state consistently. System-level testing with application-specific data is required before deployment.
+
+The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.
+
+## Model Version(s)
+
+| Version | Configuration |
+| --- | --- |
+| **InternVideo2-CLIP-L14 TAO fine-tune v1.0 (research release, non-commercial)** | Vision LoRA on the last 12 blocks, frozen text encoder, preservation regularization |
+
+The release is published as the Hugging Face repository `nvidia/InternVideo2-CLIP-anomaly-detection`. The upstream base model below is an evaluation reference, not an additional fine-tuned release variant.
+
+The model integrates into an AI system as an embedding encoder: the video tower embeds clips once into a searchable index, the text tower embeds user queries at search time, and the application ranks clips by cosine similarity. Both towers run through TAO `video_clip` inference or as exported ONNX models inside a retrieval or event-search service.
+
+## Training, Testing, and Evaluation Datasets
+
+### Dataset Overview
+
+All training data comes from the **train subset of [NVIDIA PhysicalAI-Event-Videos](https://huggingface.co/datasets/nvidia/PhysicalAI-Event-Videos)**. All retrieval testing and evaluation data comes from its **test subset**. Kinetics-400 is a separate general-domain action-classification benchmark.
+
+The train subset used for this fine-tune contains **2,743 source videos, 6,059 chunks, and 85,621 caption entries** (counts from the training manifest; caption entries are training pairs, not necessarily unique text strings). The test subset has **393 clips from 166 source videos and 3,040 event queries**; its 3,000 person-attribute-search (PAS) queries are excluded from all domain results reported here. General-domain evaluation uses 19,877 successfully decoded Kinetics-400 validation clips.
+
+Checkpoint selection used an earlier relevance revision of the same test gallery; the updated relevance annotations do not constitute a new independent held-out test.
+
+### Training Dataset
+
+**Dataset:** [NVIDIA PhysicalAI-Event-Videos](https://huggingface.co/datasets/nvidia/PhysicalAI-Event-Videos), train subset.<br>
+**Data Modality:** Video and English text.<br>
+**Video Training Data Size:** Less than 10,000 hours of fine-tuning video.<br>
+**Text Training Data Size:** Less than a Billion Tokens (85,621 caption entries, each at most 77 tokens).<br>
+**Data Collection Method:** Hybrid: Synthetic, Manually-Collected <br>
+**Labeling Method:** Hybrid: Automated, Manually-Labeled (automated pseudo-labeling produces captions and queries, with human review and correction for a subset of annotations).<br>
+
+Training metadata expands selected caption, anomaly, and action query fields into video-text pairs. Legacy dense-caption and scene-caption entries are excluded by this run's selection recipe. The data includes descriptions and footage of human activities, anomalous events, and physical-world scenes, including synthetic and real people.
+
+The training manifest has zero media overlap with the test gallery used for checkpoint selection; up to 825 training query strings also appear among the evaluation queries. This is not a comprehensive perceptual-duplicate audit.
+
+### Testing Dataset
+
+All retrieval testing data comes from the **test subset of [NVIDIA PhysicalAI-Event-Videos](https://huggingface.co/datasets/nvidia/PhysicalAI-Event-Videos)**. An earlier relevance revision of this gallery was also used for validation and checkpoint selection, so the reported result is not an independent held-out generalization estimate.
+
+**Data Modality:** Video and English text.<br>
+**Data Collection Method:** Hybrid: Synthetic, Manually-Collected <br>
+**Labeling Method:** Hybrid: Automated, Manually-Labeled (automated query and relevance annotation with human review and correction) <br>
+**Properties:** 393 trimmed video clips from 166 source videos and 3,040 English natural-language event queries (264 anomaly, 1,369 activity, 64 location, 1,343 precise-moment), each with one or more labeled relevant clips. Content is surveillance-style physical-world footage of everyday human activity and anomalous events, including fights, falls, and theft, in warehouse, retail, street, and indoor scenes. Queries are short English phrases or sentences; no other languages are included.
+
+### Evaluation Dataset
+
+All numbers below were measured on the released checkpoint.
+
+**Data Collection Method by dataset:**
+- PhysicalAI-Event-Videos test subset: Hybrid: Synthetic, Manually-Collected <br>
+- Kinetics-400: Hybrid: Automated, Manually-Collected <br>
+
+**Labeling Method by dataset:**
+- PhysicalAI-Event-Videos test subset: Hybrid: Automated, Manually-Labeled <br>
+- Kinetics-400: Manually-Labeled <br>
+
+#### PhysicalAI-Event-Videos test subset — event-only retrieval
+
+All retrieval evaluation data comes from the **test subset of [NVIDIA PhysicalAI-Event-Videos](https://huggingface.co/datasets/nvidia/PhysicalAI-Event-Videos)**: **393 clips** and **3,040 English event queries**, comprising 264 anomaly, 1,369 activity, 64 location, and 1,343 precise-moment queries. All rows use the September 16, 2026 relevance revision, released trimmed clips, and eight uniformly sampled frames. IV2CLIP uses 224-pixel inputs, BF16, and batch size four on an A100. The zero-shot Qwen3-VL-Embedding baselines use their model-specific preprocessing and resolution, BF16, and video batch size one. Their text and video instructions are respectively `Represent the user's input for video anomaly retrieval.` and `Represent this video clip for anomaly retrieval.`
+
+For each query, all 393 clips are ranked by cosine similarity. **mAP** averages per-query average precision across all labeled relevant clips. **Hit@K** is the fraction of queries with at least one relevant clip in the first K results; it is not multi-relevance Recall@K. All query records, including repeated texts, are scored. PAS results are not reported.
+
+| Model | mAP | Hit@1 | Hit@5 |
+| --- | ---: | ---: | ---: |
+| Upstream IV2CLIP-L14 base | 54.34% | 50.69% | 78.49% |
+| **This model (TAO fine-tune)** | **63.12%** | **59.11%** | **87.04%** |
+| Qwen3-VL-Embedding-2B (zero-shot) | 59.67% | 57.40% | 84.67% |
+| Qwen3-VL-Embedding-8B (zero-shot) | 60.58% | 58.03% | 85.26% |
+
+| Query category | Queries | mAP | Hit@1 | Hit@5 |
+| --- | ---: | ---: | ---: | ---: |
+| Anomaly | 264 | 67.77% | 75.76% | 95.08% |
+| Activity | 1,369 | 69.32% | 73.70% | 91.23% |
+| Location | 64 | 99.29% | 100.00% | 100.00% |
+| Precise-moment search | 1,343 | 54.16% | 39.02% | 80.57% |
+
+Native retrieval metrics were independently recomputed from the saved embeddings and agreed to within 1e-15. AUC is omitted because the pinned native evaluator had a score/label alignment defect.
+
+**Protocol limitation:** The result is not an independent held-out generalization estimate, because checkpoint selection used an earlier relevance revision of the same gallery.
+
+#### Kinetics-400 — zero-shot action classification
+
+Kinetics-400 contains real-world human-action video with human action-class labels. All rows use eight sampled frames, the same 400 class names, the prompt `A video of a person {label}`, and cosine ranking. IV2CLIP uses 224 × 224 inputs; Qwen uses model-specific preprocessing and resolution, with text and video instructions `Represent this action class for matching to a video.` and `Represent this video for action classification.` Of 19,881 available validation clips, 19,877 decoded successfully; four were excluded. Top-K accuracy measures whether the ground-truth action class is among the K highest-scoring prompted classes. These are archived evaluations under the shared protocol, not a single identical model-input pipeline; pretraining overlap with Kinetics-400 cannot be ruled out for the zero-shot models.
+
+| Model | Top-1 accuracy | Top-5 accuracy |
+| --- | ---: | ---: |
+| Upstream IV2CLIP-L14 base, recorded same-protocol reference | 70.11% | 89.60% |
+| **This model (TAO fine-tune)** | **69.19%** | **89.64%** |
+| Qwen3-VL-Embedding-2B (zero-shot) | 67.62% | 88.70% |
+| Qwen3-VL-Embedding-8B (zero-shot) | 69.57% | 89.94% |
+
+The fine-tune improves the measured domain retrieval score (+8.78 percentage points mAP over the base) while reducing Kinetics-400 Top-1 accuracy by 0.92 percentage points versus the recorded base reference. The Kinetics evaluation used merged LoRA weights (24 modules) on the 19,877 decodable validation clips. It did not retrain the model.
+
+## Inference
+
+**Acceleration Engine:** PyTorch (accuracy results); TensorRT 10.16 (performance results) <br>
+**Test Hardware:** <br>
+- NVIDIA Ampere A100
+- NVIDIA Blackwell RTX PRO 6000 Server Edition
+- NVIDIA Hopper H100 80GB HBM3
+- NVIDIA Ada Lovelace L40S
+- NVIDIA Blackwell GB300 (DGX Station)
+
+Accuracy results above are from PyTorch; domain retrieval ran on one NVIDIA Ampere A100 GPU, and the distributed Kinetics-400 evaluation ran on multiple NVIDIA Ampere A100 80-GB GPUs, which describes the evaluation setup, not a minimum inference requirement. TensorRT accuracy has not been benchmarked on the evaluation sets.
+
+### NVIDIA Performance Lab (PerLab) TensorRT Performance
+
+NVIDIA PerLab measured TensorRT performance (PBR 309421) with `trtexec`, TensorRT 10.16.0.72, CUDA 13.2.51, cuDNN 9.20, driver 595.58.03, and the `nvcr.io/nvidia/tao/tao-toolkit:7.0.1-pyt` container. The engine was built with `--stronglyTyped` from a ModelOpt AutoCast FP16 ONNX export (opset 23, RMSNorm layers kept in FP32) with an identical architecture. Each inference takes an `8 × 3 × 224 × 224` video clip and a 77-token text input on one GPU. The engine is about 0.76 GB.
+
+| GPU | Throughput, batch 1 (clips/s) | GPU compute, batch 1 (ms) | Throughput, batch 8 (clips/s) | Throughput, batch 32 (clips/s) | GPU compute, batch 32 (ms) |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| NVIDIA Blackwell RTX PRO 6000 Server Edition | 101.1 | 9.89 | 162.4 | 157.9 | 202.7 |
+| NVIDIA Hopper H100 80GB HBM3 | 152.9 | 6.54 | 249.2 | 257.9 | 124.1 |
+| NVIDIA Ada Lovelace L40S | 79.2 | 12.63 | 85.9 | 83.5 | 383.1 |
+| NVIDIA Blackwell GB300 (DGX Station) | 287.2 | 3.48 | 511.0 | 526.8 | 60.7 |
+
+A clip is one video-text pair. GPU compute is the mean `trtexec` GPU compute time per inference call.
+
+**Note:** The measured engine combines the vision and text towers in one graph, so every call runs both towers. Deployment can run faster with the towers separated: video clips are embedded once at indexing time with a vision-tower engine, and only the lightweight text tower runs for each query. Separate vision-tower and text-tower ONNX files are provided for this purpose. Separate-tower performance was not measured in PBR 309421.
+
+### Files in this repository
+
+Every file is covered by `MANIFEST.sha256` (`sha256sum -c MANIFEST.sha256`). The ONNX files and the two checkpoints encode the same weights: the PyTorch towers rebuilt from `model.safetensors` reproduce the FP32 ONNX embeddings to within 6 × 10⁻⁷ (minimum cosine 0.99999994) on seeded and real-query probes, and `model.safetensors` reproduces the embeddings of `model_epoch_002_step_04011.no_teacher.pth` after LoRA merge exactly.
+
+| File | Bytes | SHA-256 (prefix) | Purpose |
+| --- | ---: | --- | --- |
+| `model.safetensors` | 1,491,073,916 | `31992985a49ff184…` | Fine-tuned weights with the 24 vision LoRA modules merged into the base linear layers (486 tensors, 372,752,643 parameters, FP32; TAO `VideoCLIPPlModel` state-dict key names, `peft` disabled). Loads with `model_config.yaml`. |
+| `model_config.yaml` | 1,346 | `282f74e63df56699…` | TAO `video_clip` model block (`peft.enabled: false`) for loading `model.safetensors`. |
+| `model_epoch_002_step_04011.no_teacher.pth` | 1,493,567,543 | `a7c2b4148a882497…` | TAO training checkpoint (epoch 2, step 4,011) with the LoRA adapters unmerged: 534 tensors, preservation-teacher and optimizer state removed. Loads with `train_experiment.yaml` and `regularization.enabled: false`. |
+| `train_experiment.yaml` | 6,153 | `7f3a5f6117c7226e…` | Complete TAO `video_clip` training specification of the fine-tune (paths replaced by placeholders). |
+| `onnx_opset23/iv2clip_v1.0_step04011_opset23_vision.onnx` | 3,914,064 | `63ee64deaef832b9…` | FP32 vision tower, ONNX opset 23, dynamic batch, LoRA merged; 96 fused `RMSNormalization` ops. Keep with its `.onnx.data` file. |
+| `onnx_opset23/iv2clip_v1.0_step04011_opset23_vision.onnx.data` | 1,237,385,216 | `27e70bca3f0ca169…` | External weights of the FP32 vision tower. |
+| `onnx_opset23/iv2clip_v1.0_step04011_opset23_text.onnx` | 1,230,030 | `5e55caf759e88dbb…` | FP32 text tower, ONNX opset 23, dynamic batch. Keep with its `.onnx.data` file. |
+| `onnx_opset23/iv2clip_v1.0_step04011_opset23_text.onnx.data` | 253,755,392 | `cacc6493c3a7863c…` | External weights of the FP32 text tower. |
+| `onnx_opset23/iv2clip_v1.0_step04011_opset23_vision.autocast_fp16.onnx` | 619,141,817 | `141e726ae5dcf0bf…` | Vision tower for FP16 TensorRT deployment (ModelOpt AutoCast: FP16 bulk, `RMSNormalization` kept FP32, FP32 I/O, single file). Build strongly typed. |
+| `onnx_opset23/iv2clip_v1.0_step04011_opset23_text.autocast_fp16.onnx` | 126,946,278 | `6e41fa1c7fb94e6b…` | Text tower for FP16 TensorRT deployment (AutoCast, FP32 I/O, single file). Build strongly typed. |
+| `onnx_opset23/iv2clip_v1.0_step04011_opset23_config.yaml` | 6,588 | `b98dd5a682c3ef66…` | Deploy configuration written by the TAO exporter (`model.init_logit_scale` = 4.544891, i.e. exp = 94.1502; `init_logit_bias` = 0). |
+| `onnx_opset23/export_spec_v1.0_step04011_separate_opset23.yaml` | 6,003 | `a03f526bcf8372cf…` | TAO `video_clip export` specification that produced the ONNX files (separate towers, opset 23, dynamic batch). |
+| `onnx_opset23/iv2clip_v1.0_step04011_opset23_tokenizer/` | 5,030,306 (5 files) | see `MANIFEST.sha256` | Hugging Face tokenizer equivalent to the model's CLIP BPE tokenizer (`openai/clip-vit-large-patch14`, MIT): `tokenizer.json`, `vocab.json`, `merges.txt`, `tokenizer_config.json`, `special_tokens_map.json`. |
+
+### Loading the weights
+
+The TAO `video_clip` model constructor needs the upstream base assets referenced in the specification files: the InternVideo2 vision and alignment weights (`model.internvideo2clip_hf_id: OpenGVLab/InternVideo2_distillation_models`, resolved through the Hugging Face cache) and the Apple MobileCLIP-B-LT text-encoder file (`model.text_encoder`, `mobileclip_blt.pt`). Both are subject to their own licenses listed above.
+
+```python
+from omegaconf import OmegaConf
+from safetensors.torch import load_file
+from nvidia_tao_pytorch.config.video_clip.default_config import VideoCLIPExperimentConfig
+from nvidia_tao_pytorch.multimodal.video_clip.model.pl_video_clip_model import VideoCLIPPlModel
+
+# merged weights (no LoRA modules)
+cfg = OmegaConf.merge(OmegaConf.structured(VideoCLIPExperimentConfig), OmegaConf.load("model_config.yaml"))
+model = VideoCLIPPlModel(experiment_spec=cfg).eval()
+model.load_state_dict(load_file("model.safetensors"), strict=True)
+
+# or the TAO checkpoint with the LoRA adapters still separate
+cfg = OmegaConf.merge(OmegaConf.structured(VideoCLIPExperimentConfig), OmegaConf.load("train_experiment.yaml"))
+cfg.regularization.enabled = False
+model = VideoCLIPPlModel.load_from_checkpoint("model_epoch_002_step_04011.no_teacher.pth", map_location="cpu", experiment_spec=cfg).eval()
+```
+
+### Using the ONNX towers
+
+- Vision tower: input `image` float32 `[B, 8, 3, 224, 224]`; output `image_embedding` float32 `[B, 512]`, L2-normalized. Sample 8 frames uniformly over the clip (`linspace(0, n-1, 8)`), RGB, bicubic resize to 224 × 224, scale to [0, 1], normalize with mean `(0.485, 0.456, 0.406)` and standard deviation `(0.229, 0.224, 0.225)`.
+- Text tower: inputs `input_ids` and `attention_mask`, both int64 `[B, 77]` (the mask is accepted for compatibility and ignored); output `text_embedding` float32 `[B, 512]`, L2-normalized. Tokenize with the bundled tokenizer using `padding="max_length", truncation=True, max_length=77`.
+- The FP32 files also output the constants `logit_scale` (94.1502, already exponentiated) and `logit_bias` (0). Similarity is `logit_scale * (image_embedding · text_embedding) + logit_bias`; ranking only needs the dot product.
+- The opset-23 graphs use the fused `RMSNormalization` operator: onnxruntime 1.23 or newer is required. The AutoCast FP16 files must be built **strongly typed** in TensorRT; a weakly typed `--fp16` build re-lowers the vision RMSNorm to half precision and collapses the vision embeddings.
+
+```bash
+trtexec --onnx=onnx_opset23/iv2clip_v1.0_step04011_opset23_vision.autocast_fp16.onnx --stronglyTyped \
+  --minShapes=image:1x8x3x224x224 --optShapes=image:8x8x3x224x224 --maxShapes=image:16x8x3x224x224 \
+  --saveEngine=iv2clip_vision_fp16.plan
+trtexec --onnx=onnx_opset23/iv2clip_v1.0_step04011_opset23_text.autocast_fp16.onnx --stronglyTyped \
+  --minShapes=input_ids:1x77,attention_mask:1x77 --optShapes=input_ids:8x77,attention_mask:8x77 \
+  --maxShapes=input_ids:16x77,attention_mask:16x77 --saveEngine=iv2clip_text_fp16.plan
+```
+
+### How to use this model
+
+Use a TAO installation containing the `video_clip` InternVideo2-CLIP-L14 implementation, the matching base assets and tokenizer, and a complete experiment specification compatible with this checkpoint. The [TAO InternVideo2-CLIP overview](https://docs.nvidia.com/tao/tao-toolkit/latest/text/embedding/video_clip/overview.html) describes the model and data input format; the [training, evaluation, inference, and export guide](https://docs.nvidia.com/tao/tao-toolkit/latest/text/embedding/video_clip/training.html) documents the experiment specification and every subtask below. Set `model.type: internvideo2-clip-l14`, `model.num_frames: 8`, and `model.image_size: 224`, and restore the vision-LoRA configuration described above. The files in this repository are listed under **Files in this repository** above.
+
+**Train** ([guide](https://docs.nvidia.com/tao/tao-toolkit/latest/text/embedding/video_clip/training.html#training-the-model)): fine-tune from the base weights with the recipe described under Model Architecture.
+
+```bash
+video_clip train -e /path/to/train.yaml
+```
+
+**Evaluate** ([guide](https://docs.nvidia.com/tao/tao-toolkit/latest/text/embedding/video_clip/training.html#evaluating-the-model)): compute retrieval metrics against a query/gallery set with this checkpoint selected in the evaluation section.
+
+```bash
+video_clip evaluate -e /path/to/evaluate.yaml
+```
+
+**Inference** ([guide](https://docs.nvidia.com/tao/tao-toolkit/latest/text/embedding/video_clip/training.html#running-inference)): use `inference.mode: embeddings` for embedding extraction or `inference.mode: retrieval` for ranked retrieval with the configured query and gallery inputs.
+
+```bash
+video_clip inference -e /path/to/inference.yaml
+```
+
+**Export and deploy** ([export guide](https://docs.nvidia.com/tao/tao-toolkit/latest/text/embedding/video_clip/training.html#exporting-the-model), [TAO Deploy](https://docs.nvidia.com/tao/tao-toolkit/latest/text/tao_deploy/video_clip.html), [using the embeddings](https://docs.nvidia.com/tao/tao-toolkit/latest/text/embedding/video_clip/applications.html)): export the towers to ONNX, build a TensorRT engine and run TensorRT-side evaluation and inference with TAO Deploy, and consume the ONNX embeddings in a retrieval or event-search application. Export and deployment validation must use the same tokenizer, normalization, LoRA weights, and output dimensions as the PyTorch path.
+
+```bash
+video_clip export -e /path/to/export.yaml
+```
+
+## Ethical Considerations
+
+NVIDIA believes Trustworthy AI is a shared responsibility. Developers should evaluate this model against their intended domain and address potential misuse before deployment.
+
+Users must have the necessary rights and permissions for input images and video, including content depicting people or containing intellectual property. Embedding extraction does not anonymize source content. Retrieval errors can produce false alarms or missed events; inspect source clips and validate downstream decisions in context. Commercial deployment is not permitted under the Apple MobileCLIP license; it would require a separate Apple commercial license or a replacement text encoder.
+
+Please report model quality, risk, security vulnerabilities, or NVIDIA AI concerns through [NVIDIA's reporting channel](https://www.nvidia.com/en-us/support/submit-security-vulnerability/).

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:31992985a49ff1840a2c862e9fee029b57eb37b0eefcde1da09128e927e9dab6
+size 1491073916

model_config.yaml

@@ -0,0 +1,45 @@
+# Model block for loading model.safetensors (LoRA already merged) into a TAO video_clip VideoCLIPPlModel with peft disabled:
+#   cfg = OmegaConf.merge(OmegaConf.structured(VideoCLIPExperimentConfig), OmegaConf.load('model_config.yaml'))
+#   pl = VideoCLIPPlModel(experiment_spec=cfg); pl.load_state_dict(safetensors.torch.load_file('model.safetensors'), strict=True)
+# To load model_epoch_002_step_04011.no_teacher.pth instead, use train_experiment.yaml (peft enabled) with regularization.enabled: false.
+model:
+  type: internvideo2-clip-l14
+  adaptor_name: null
+  freeze_vision_encoder: true
+  freeze_text_encoder: true
+  image_size: 224
+  init_logit_scale: 4.605170185988092
+  init_logit_bias: 0.0
+  canonicalize_text: false
+  pretrained_ckpt: null
+  internvideo2clip_hf_id: OpenGVLab/InternVideo2_distillation_models
+  vision_encoder: null
+  text_encoder: /models/mobileclip_blt.pt
+  clip_head: null
+  num_frames: 8
+  use_flash_attn: false
+  use_fused_rmsnorm: false
+  use_fused_mlp: false
+peft:
+  enabled: false
+  method: lora
+  vision:
+    enabled: false
+    target_modules:
+    - qkv
+    - proj
+    num_last_blocks: 12
+    rank: 8
+    alpha: 16
+    dropout: 0.05
+  text:
+    enabled: false
+    target_modules:
+    - qkv_proj
+    - out_proj
+    num_last_blocks: 3
+    rank: 8
+    alpha: 16
+    dropout: 0.05
+regularization:
+  enabled: false

model_epoch_002_step_04011.no_teacher.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a7c2b4148a882497d110dc2f5a3d124efb1210303ed59daec9ccb6eb4d073c63
+size 1493567543

onnx_opset23/export_spec_v1.0_step04011_separate_opset23.yaml

@@ -0,0 +1,299 @@
+# TAO video_clip export spec that produced onnx_opset23/*: separate towers, opset 23, dynamic batch, LoRA merged by the exporter.
+model_name: null
+encryption_key: nvidia_tao
+results_dir: /results/export
+wandb:
+  enable: false
+  project: TAO Toolkit
+  entity: ''
+  group: ''
+  tags:
+  - training
+  - base-init
+  - v5-validation
+  - no-rwf
+  - 1n8g
+  reinit: false
+  sync_tensorboard: false
+  save_code: false
+  name: ''
+  run_id: ''
+model:
+  type: internvideo2-clip-l14
+  adaptor_name: null
+  freeze_vision_encoder: true
+  freeze_text_encoder: true
+  image_size: 224
+  init_logit_scale: 4.605170185988092
+  init_logit_bias: 0.0
+  canonicalize_text: false
+  pretrained_ckpt: null
+  internvideo2clip_hf_id: OpenGVLab/InternVideo2_distillation_models
+  vision_encoder: null
+  text_encoder: /models/mobileclip_blt.pt
+  clip_head: null
+  num_frames: 8
+  use_flash_attn: false
+  use_fused_rmsnorm: false
+  use_fused_mlp: false
+dataset:
+  train:
+    batch_size: 8
+    num_workers: 12
+    type: video_text
+    video_text:
+      metadata: /data/PhysicalAI-Event-Videos/train/labels.json
+      format: vadr1_chunks
+      data_root: null
+      split: train
+      path_prefix_mapping: {}
+      caption_fields:
+      - train_caption
+      caption_mode: all
+      idx_mode: sample_id
+      idx_field: null
+      task_type: retrieval
+      anomaly_only: false
+      num_frames: ${model.num_frames}
+      relevance_file: null
+  val:
+    batch_size: 4
+    num_workers: 12
+    type: video_text
+    video_text:
+      metadata: /data/PhysicalAI-Event-Videos/test/labels.json
+      format: vadr1_chunks
+      data_root: null
+      split: test
+      path_prefix_mapping: {}
+      caption_fields:
+      - queries
+      - anomaly_queries
+      - action_queries
+      - dense_caption
+      - scene_caption
+      caption_mode: first
+      idx_mode: sample_id
+      idx_field: null
+      task_type: retrieval
+      anomaly_only: false
+      num_frames: ${model.num_frames}
+      relevance_file: /data/PhysicalAI-Event-Videos/test/relevance.json
+  metrics:
+    mode: retrieval
+    exclude_categories:
+    - Normal
+    - Abnormal
+  inference:
+    batch_size: 16
+    num_workers: 8
+    type: video_text
+    video_text:
+      metadata: null
+      format: auto
+      data_root: null
+      split: null
+      path_prefix_mapping: {}
+      caption_fields:
+      - caption
+      caption_mode: first
+      idx_mode: sample_id
+      idx_field: null
+      task_type: retrieval
+      anomaly_only: false
+      num_frames: 8
+      relevance_file: null
+  augmentation:
+    scale:
+    - 1.0
+    - 1.0
+    color_jitter: []
+    grayscale: 0.0
+  pin_memory: true
+  seed: 42
+train:
+  num_gpus: 8
+  gpu_ids:
+  - 0
+  - 1
+  - 2
+  - 3
+  - 4
+  - 5
+  - 6
+  - 7
+  num_nodes: 1
+  seed: 42
+  cudnn:
+    benchmark: false
+    deterministic: true
+  num_epochs: 5
+  checkpoint_interval: 1
+  checkpoint_interval_unit: epoch
+  validation_interval: 1
+  resume_training_checkpoint_path: null
+  results_dir: /results/train
+  checkpointer:
+    enable_topk: false
+    monitor: null
+    mode: null
+    save_top_k: 1
+    filename: model_best_{epoch:03d}
+    dirpath: null
+    auto_insert_metric_name: false
+  optim:
+    optimizer_type: adamw
+    vision_lr: 0.001
+    text_lr: 0.001
+    weight_decay: 0.2
+    betas:
+    - 0.9
+    - 0.98
+    eps: 1.0e-06
+    warmup_steps: 100
+    scheduler: cosine
+  loss_type: internvideo2_vtc
+  precision: bf16
+  grad_clip_norm: null
+  grad_checkpointing: false
+  distributed_strategy: ddp
+  pretrained_model_path: null
+  val_check_interval: null
+evaluate:
+  checkpoint: null
+  results_dir: null
+  num_gpus: 8
+  gpu_ids:
+  - 0
+  - 1
+  - 2
+  - 3
+  - 4
+  - 5
+  - 6
+  - 7
+  batch_size: 4
+  num_workers: 12
+  video_text:
+    metadata: null
+    format: auto
+    data_root: null
+    split: null
+    path_prefix_mapping: {}
+    caption_fields:
+    - caption
+    caption_mode: first
+    idx_mode: sample_id
+    idx_field: null
+    task_type: retrieval
+    anomaly_only: false
+    num_frames: 8
+    relevance_file: null
+  video_embeddings_file: null
+  text_embeddings_file: null
+  overwrite_embeddings: false
+  search:
+    enabled: false
+    search_metric: cosine
+    normalize: true
+    top_k: 10
+  trt_engine: null
+  text_file: null
+inference:
+  checkpoint: null
+  results_dir: null
+  num_gpus: 8
+  gpu_ids:
+  - 0
+  - 1
+  - 2
+  - 3
+  - 4
+  - 5
+  - 6
+  - 7
+  batch_size: 4
+  num_workers: 12
+  video_text:
+    metadata: null
+    format: auto
+    data_root: null
+    split: null
+    path_prefix_mapping: {}
+    caption_fields:
+    - caption
+    caption_mode: first
+    idx_mode: sample_id
+    idx_field: null
+    task_type: retrieval
+    anomaly_only: false
+    num_frames: 8
+    relevance_file: null
+  video_embeddings_file: null
+  text_embeddings_file: null
+  overwrite_embeddings: false
+  search:
+    enabled: false
+    search_metric: cosine
+    normalize: true
+    top_k: 5
+  mode: embeddings
+  query:
+    input_texts: []
+    input_videos: []
+    text_file: null
+    num_frames: ${model.num_frames}
+peft:
+  enabled: true
+  method: lora
+  vision:
+    enabled: true
+    target_modules:
+    - qkv
+    - proj
+    num_last_blocks: 12
+    rank: 8
+    alpha: 16
+    dropout: 0.05
+  text:
+    enabled: false
+    target_modules:
+    - qkv_proj
+    - out_proj
+    num_last_blocks: 3
+    rank: 8
+    alpha: 16
+    dropout: 0.05
+regularization:
+  enabled: true
+  embedding_mse_weight: 0.05
+  cosine_weight: 0.05
+  similarity_weight: 0.1
+export:
+  checkpoint: /models/model_epoch_002_step_04011.no_teacher.pth
+  onnx_file: /results/export/iv2clip_v1.0_step04011_opset23.onnx
+  encoder_type: separate
+  opset_version: 23
+  batch_size: -1
+  input_height: 224
+  input_width: 224
+  gpu_id: 0
+  on_cpu: false
+  input_channel: 3
+  verbose: false
+  results_dir: /results/export
+gen_trt_engine:
+  results_dir: null
+  gpu_id: 0
+  onnx_file: ???
+  trt_engine: ???
+  timing_cache: null
+  batch_size: -1
+  verbose: false
+  tensorrt:
+    workspace_size: 1024
+    min_batch_size: 1
+    opt_batch_size: 1
+    max_batch_size: 16
+    layers_precision: []
+    data_type: fp32

onnx_opset23/iv2clip_v1.0_step04011_opset23_config.yaml

@@ -0,0 +1,299 @@
+model_name: null
+encryption_key: nvidia_tao
+results_dir: /workspace/alicli/experiments/internvideo2clip/physicalai_no_rwf_base_5ep_v5_1n8g_093026/export_step04011_separate_opset23/export
+wandb:
+  enable: false
+  project: TAO Toolkit
+  entity: ''
+  group: ''
+  tags:
+  - training
+  - base-init
+  - v5-validation
+  - no-rwf
+  - 1n8g
+  reinit: false
+  sync_tensorboard: false
+  save_code: false
+  name: iv2clip_no_rwf_093026_base_5ep_v5_1n8g
+  run_id: ''
+model:
+  type: internvideo2-clip-l14
+  adaptor_name: null
+  freeze_vision_encoder: true
+  freeze_text_encoder: true
+  image_size: 224
+  init_logit_scale: 4.544891357421875
+  init_logit_bias: 0.0
+  canonicalize_text: false
+  pretrained_ckpt: null
+  internvideo2clip_hf_id: OpenGVLab/InternVideo2_distillation_models
+  vision_encoder: null
+  text_encoder: /workspace/alicli/models/internvideo2_clip_l14/mobileclip_blt.pt
+  clip_head: null
+  num_frames: 8
+  use_flash_attn: false
+  use_fused_rmsnorm: false
+  use_fused_mlp: false
+dataset:
+  train:
+    batch_size: 8
+    num_workers: 12
+    type: video_text
+    video_text:
+      metadata: /workspace/alicli/local_artifacts/093026_iv2clip-no-rwf-retrain/train_no_rwf.json
+      format: vadr1_chunks
+      data_root: null
+      split: train
+      path_prefix_mapping: {}
+      caption_fields:
+      - train_caption
+      caption_mode: all
+      idx_mode: sample_id
+      idx_field: null
+      task_type: retrieval
+      anomaly_only: false
+      num_frames: ${model.num_frames}
+      relevance_file: null
+  val:
+    batch_size: 4
+    num_workers: 12
+    type: video_text
+    video_text:
+      metadata: /workspace/data/AI_City_Challenge_2026/Annotated_batches/kpi_annotated_chunks_5sec_ps.actionq.split.json
+      format: vadr1_chunks
+      data_root: null
+      split: test
+      path_prefix_mapping: {}
+      caption_fields:
+      - queries
+      - anomaly_queries
+      - action_queries
+      - dense_caption
+      - scene_caption
+      caption_mode: first
+      idx_mode: sample_id
+      idx_field: null
+      task_type: retrieval
+      anomaly_only: false
+      num_frames: ${model.num_frames}
+      relevance_file: /workspace/data/AI_City_Challenge_2026/iv2clip_test/domain_test_v5_082626.json
+  metrics:
+    mode: retrieval
+    exclude_categories:
+    - Normal
+    - Abnormal
+  inference:
+    batch_size: 16
+    num_workers: 8
+    type: video_text
+    video_text:
+      metadata: null
+      format: auto
+      data_root: null
+      split: null
+      path_prefix_mapping: {}
+      caption_fields:
+      - caption
+      caption_mode: first
+      idx_mode: sample_id
+      idx_field: null
+      task_type: retrieval
+      anomaly_only: false
+      num_frames: 8
+      relevance_file: null
+  augmentation:
+    scale:
+    - 1.0
+    - 1.0
+    color_jitter: []
+    grayscale: 0.0
+  pin_memory: true
+  seed: 42
+train:
+  num_gpus: 8
+  gpu_ids:
+  - 0
+  - 1
+  - 2
+  - 3
+  - 4
+  - 5
+  - 6
+  - 7
+  num_nodes: 1
+  seed: 42
+  cudnn:
+    benchmark: false
+    deterministic: true
+  num_epochs: 5
+  checkpoint_interval: 1
+  checkpoint_interval_unit: epoch
+  validation_interval: 1
+  resume_training_checkpoint_path: null
+  results_dir: /workspace/alicli/experiments/internvideo2clip/physicalai_no_rwf_base_5ep_v5_1n8g_093026/train
+  checkpointer:
+    enable_topk: false
+    replace_periodic: false
+    monitor: null
+    mode: null
+    save_top_k: 1
+    filename: model_best_{epoch:03d}
+    dirpath: null
+    auto_insert_metric_name: false
+  optim:
+    optimizer_type: adamw
+    vision_lr: 0.001
+    text_lr: 0.001
+    weight_decay: 0.2
+    betas:
+    - 0.9
+    - 0.98
+    eps: 1.0e-06
+    warmup_steps: 100
+    scheduler: cosine
+  loss_type: internvideo2_vtc
+  precision: bf16
+  grad_clip_norm: null
+  grad_checkpointing: false
+  distributed_strategy: ddp
+  pretrained_model_path: null
+  val_check_interval: null
+evaluate:
+  checkpoint: null
+  results_dir: null
+  num_gpus: 8
+  gpu_ids:
+  - 0
+  - 1
+  - 2
+  - 3
+  - 4
+  - 5
+  - 6
+  - 7
+  batch_size: 4
+  num_workers: 12
+  video_text:
+    metadata: null
+    format: auto
+    data_root: null
+    split: null
+    path_prefix_mapping: {}
+    caption_fields:
+    - caption
+    caption_mode: first
+    idx_mode: sample_id
+    idx_field: null
+    task_type: retrieval
+    anomaly_only: false
+    num_frames: 8
+    relevance_file: null
+  video_embeddings_file: null
+  text_embeddings_file: null
+  overwrite_embeddings: false
+  search:
+    enabled: false
+    search_metric: cosine
+    normalize: true
+    top_k: 10
+  trt_engine: null
+  text_file: null
+inference:
+  checkpoint: null
+  results_dir: null
+  num_gpus: 8
+  gpu_ids:
+  - 0
+  - 1
+  - 2
+  - 3
+  - 4
+  - 5
+  - 6
+  - 7
+  batch_size: 4
+  num_workers: 12
+  video_text:
+    metadata: null
+    format: auto
+    data_root: null
+    split: null
+    path_prefix_mapping: {}
+    caption_fields:
+    - caption
+    caption_mode: first
+    idx_mode: sample_id
+    idx_field: null
+    task_type: retrieval
+    anomaly_only: false
+    num_frames: 8
+    relevance_file: null
+  video_embeddings_file: null
+  text_embeddings_file: null
+  overwrite_embeddings: false
+  search:
+    enabled: false
+    search_metric: cosine
+    normalize: true
+    top_k: 5
+  mode: embeddings
+  query:
+    input_texts: []
+    input_videos: []
+    text_file: null
+    num_frames: ${model.num_frames}
+peft:
+  enabled: true
+  method: lora
+  vision:
+    enabled: true
+    target_modules:
+    - qkv
+    - proj
+    num_last_blocks: 12
+    rank: 8
+    alpha: 16
+    dropout: 0.05
+  text:
+    enabled: false
+    target_modules:
+    - qkv_proj
+    - out_proj
+    num_last_blocks: 3
+    rank: 8
+    alpha: 16
+    dropout: 0.05
+regularization:
+  enabled: true
+  embedding_mse_weight: 0.05
+  cosine_weight: 0.05
+  similarity_weight: 0.1
+export:
+  checkpoint: /workspace/alicli/experiments/internvideo2clip/physicalai_no_rwf_base_5ep_v5_1n8g_093026/train/model_epoch_002_step_04011.pth
+  onnx_file: /workspace/alicli/experiments/internvideo2clip/physicalai_no_rwf_base_5ep_v5_1n8g_093026/export_step04011_separate_opset23/iv2clip_v1.0_step04011_opset23.onnx
+  encoder_type: separate
+  opset_version: 23
+  batch_size: -1
+  input_height: 224
+  input_width: 224
+  gpu_id: 0
+  on_cpu: false
+  input_channel: 3
+  verbose: false
+  results_dir: /workspace/alicli/experiments/internvideo2clip/physicalai_no_rwf_base_5ep_v5_1n8g_093026/export_step04011_separate_opset23/export
+gen_trt_engine:
+  results_dir: null
+  gpu_id: 0
+  onnx_file: ???
+  trt_engine: ???
+  timing_cache: null
+  batch_size: -1
+  verbose: false
+  tensorrt:
+    workspace_size: 1024
+    min_batch_size: 1
+    opt_batch_size: 1
+    max_batch_size: 16
+    layers_precision: []
+    data_type: fp32

onnx_opset23/iv2clip_v1.0_step04011_opset23_text.autocast_fp16.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6e41fa1c7fb94e6b438375048603f818dc6c6b5ee35f3d7259905905a3760f0b
+size 126946278

onnx_opset23/iv2clip_v1.0_step04011_opset23_text.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5e55caf759e88dbb85e37a8e124129070bcca2d57d85fd629aef43ce7954a51e
+size 1230030

onnx_opset23/iv2clip_v1.0_step04011_opset23_text.onnx.data

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cacc6493c3a7863c1cd20bfec11600d4f2aeef953849f1cfa9fe87081887787f
+size 253755392

onnx_opset23/iv2clip_v1.0_step04011_opset23_tokenizer/special_tokens_map.json

@@ -0,0 +1,30 @@
+{
+  "bos_token": {
+    "content": "<|startoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

onnx_opset23/iv2clip_v1.0_step04011_opset23_tokenizer/tokenizer_config.json

@@ -0,0 +1,31 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "49406": {
+      "content": "<|startoftext|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49407": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<|startoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "do_lower_case": true,
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "extra_special_tokens": {},
+  "model_max_length": 77,
+  "pad_token": "<|endoftext|>",
+  "tokenizer_class": "CLIPTokenizer",
+  "unk_token": "<|endoftext|>"
+}

onnx_opset23/iv2clip_v1.0_step04011_opset23_vision.autocast_fp16.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:141e726ae5dcf0bf7c3f21054e186ce8021410c403f73574e0d3dbb430be38f5
+size 619141817

onnx_opset23/iv2clip_v1.0_step04011_opset23_vision.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:63ee64deaef832b9b7ccfd340c809642f7e8285689d1a0f8cd3478527f1fb189
+size 3914064

onnx_opset23/iv2clip_v1.0_step04011_opset23_vision.onnx.data

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:27e70bca3f0ca169e1a6ae41fb3e1cafe6371d746f405ac0498cf54d6fc15f67
+size 1237385216

train_experiment.yaml

@@ -0,0 +1,297 @@
+# TAO video_clip training specification used for nvidia/InternVideo2-CLIP-anomaly-detection v1.0
+# (run physicalai_no_rwf_base_5ep_v5_1n8g_093026, released checkpoint = epoch 2 / step 4011).
+# Paths are placeholders: model.text_encoder = Apple MobileCLIP-B-LT weights (mobileclip_blt.pt), dataset.* = PhysicalAI-Event-Videos
+# train/test subsets. Base vision/alignment weights resolve from model.internvideo2clip_hf_id via the Hugging Face cache.
+model_name: null
+encryption_key: nvidia_tao
+results_dir: /results/train
+wandb:
+  enable: false
+  project: TAO Toolkit
+  entity: ''
+  group: ''
+  tags: []
+  reinit: false
+  sync_tensorboard: false
+  save_code: false
+  name: ''
+  run_id: ''
+model:
+  type: internvideo2-clip-l14
+  adaptor_name: null
+  freeze_vision_encoder: true
+  freeze_text_encoder: true
+  image_size: 224
+  init_logit_scale: 4.605170185988092
+  init_logit_bias: 0.0
+  canonicalize_text: false
+  pretrained_ckpt: null
+  internvideo2clip_hf_id: OpenGVLab/InternVideo2_distillation_models
+  vision_encoder: null
+  text_encoder: /models/mobileclip_blt.pt
+  clip_head: null
+  num_frames: 8
+  use_flash_attn: false
+  use_fused_rmsnorm: false
+  use_fused_mlp: false
+dataset:
+  train:
+    batch_size: 8
+    num_workers: 12
+    type: video_text
+    video_text:
+      metadata: /data/PhysicalAI-Event-Videos/train/labels.json
+      format: vadr1_chunks
+      data_root: null
+      split: train
+      path_prefix_mapping: {}
+      caption_fields:
+      - train_caption
+      caption_mode: all
+      idx_mode: sample_id
+      idx_field: null
+      task_type: retrieval
+      anomaly_only: false
+      num_frames: ${model.num_frames}
+      relevance_file: null
+  val:
+    batch_size: 4
+    num_workers: 12
+    type: video_text
+    video_text:
+      metadata: /data/PhysicalAI-Event-Videos/test/labels.json
+      format: vadr1_chunks
+      data_root: null
+      split: test
+      path_prefix_mapping: {}
+      caption_fields:
+      - queries
+      - anomaly_queries
+      - action_queries
+      - dense_caption
+      - scene_caption
+      caption_mode: first
+      idx_mode: sample_id
+      idx_field: null
+      task_type: retrieval
+      anomaly_only: false
+      num_frames: ${model.num_frames}
+      relevance_file: /data/PhysicalAI-Event-Videos/test/relevance.json
+  metrics:
+    mode: retrieval
+    exclude_categories:
+    - Normal
+    - Abnormal
+  inference:
+    batch_size: 16
+    num_workers: 8
+    type: video_text
+    video_text:
+      metadata: null
+      format: auto
+      data_root: null
+      split: null
+      path_prefix_mapping: {}
+      caption_fields:
+      - caption
+      caption_mode: first
+      idx_mode: sample_id
+      idx_field: null
+      task_type: retrieval
+      anomaly_only: false
+      num_frames: 8
+      relevance_file: null
+  augmentation:
+    scale:
+    - 1.0
+    - 1.0
+    color_jitter: []
+    grayscale: 0.0
+  pin_memory: true
+  seed: 42
+train:
+  num_gpus: 8
+  gpu_ids:
+  - 0
+  - 1
+  - 2
+  - 3
+  - 4
+  - 5
+  - 6
+  - 7
+  num_nodes: 1
+  seed: 42
+  cudnn:
+    benchmark: false
+    deterministic: true
+  num_epochs: 5
+  checkpoint_interval: 1
+  checkpoint_interval_unit: epoch
+  validation_interval: 1
+  resume_training_checkpoint_path: null
+  results_dir: /results/train
+  checkpointer:
+    enable_topk: false
+    monitor: null
+    mode: null
+    save_top_k: 1
+    filename: model_best_{epoch:03d}
+    dirpath: null
+    auto_insert_metric_name: false
+  optim:
+    optimizer_type: adamw
+    vision_lr: 0.001
+    text_lr: 0.001
+    weight_decay: 0.2
+    betas:
+    - 0.9
+    - 0.98
+    eps: 1.0e-06
+    warmup_steps: 100
+    scheduler: cosine
+  loss_type: internvideo2_vtc
+  precision: bf16
+  grad_clip_norm: null
+  grad_checkpointing: false
+  distributed_strategy: ddp
+  pretrained_model_path: null
+  val_check_interval: null
+evaluate:
+  checkpoint: null
+  results_dir: null
+  num_gpus: 8
+  gpu_ids:
+  - 0
+  - 1
+  - 2
+  - 3
+  - 4
+  - 5
+  - 6
+  - 7
+  batch_size: 4
+  num_workers: 12
+  video_text:
+    metadata: null
+    format: auto
+    data_root: null
+    split: null
+    path_prefix_mapping: {}
+    caption_fields:
+    - caption
+    caption_mode: first
+    idx_mode: sample_id
+    idx_field: null
+    task_type: retrieval
+    anomaly_only: false
+    num_frames: 8
+    relevance_file: null
+  video_embeddings_file: null
+  text_embeddings_file: null
+  overwrite_embeddings: false
+  search:
+    enabled: false
+    search_metric: cosine
+    normalize: true
+    top_k: 10
+  trt_engine: null
+  text_file: null
+inference:
+  checkpoint: null
+  results_dir: null
+  num_gpus: 8
+  gpu_ids:
+  - 0
+  - 1
+  - 2
+  - 3
+  - 4
+  - 5
+  - 6
+  - 7
+  batch_size: 4
+  num_workers: 12
+  video_text:
+    metadata: null
+    format: auto
+    data_root: null
+    split: null
+    path_prefix_mapping: {}
+    caption_fields:
+    - caption
+    caption_mode: first
+    idx_mode: sample_id
+    idx_field: null
+    task_type: retrieval
+    anomaly_only: false
+    num_frames: 8
+    relevance_file: null
+  video_embeddings_file: null
+  text_embeddings_file: null
+  overwrite_embeddings: false
+  search:
+    enabled: false
+    search_metric: cosine
+    normalize: true
+    top_k: 5
+  mode: embeddings
+  query:
+    input_texts: []
+    input_videos: []
+    text_file: null
+    num_frames: ${model.num_frames}
+peft:
+  enabled: true
+  method: lora
+  vision:
+    enabled: true
+    target_modules:
+    - qkv
+    - proj
+    num_last_blocks: 12
+    rank: 8
+    alpha: 16
+    dropout: 0.05
+  text:
+    enabled: false
+    target_modules:
+    - qkv_proj
+    - out_proj
+    num_last_blocks: 3
+    rank: 8
+    alpha: 16
+    dropout: 0.05
+regularization:
+  enabled: true
+  embedding_mse_weight: 0.05
+  cosine_weight: 0.05
+  similarity_weight: 0.1
+export:
+  checkpoint: null
+  onnx_file: null
+  encoder_type: separate
+  opset_version: 23
+  batch_size: 1
+  input_height: 224
+  input_width: 224
+  gpu_id: 0
+  on_cpu: false
+  input_channel: 3
+  verbose: false
+  results_dir: null
+gen_trt_engine:
+  results_dir: null
+  gpu_id: 0
+  onnx_file: ???
+  trt_engine: ???
+  timing_cache: null
+  batch_size: -1
+  verbose: false
+  tensorrt:
+    workspace_size: 1024
+    min_batch_size: 1
+    opt_batch_size: 1
+    max_batch_size: 16
+    layers_precision: []
+    data_type: fp32