| --- |
| license: mit |
| tags: |
| - pytorch |
| - video |
| - self-supervised-learning |
| - feature-extraction |
| --- |
| |
| # P-JEPA |
|
|
| Self-supervised temporal encoders for **precomputed video features**. These |
| checkpoints learn temporal representations over feature sequences; they do not |
| extract features from raw video. All encoder weights are released under MIT. |
|
|
| | Checkpoint | Training features | Output width | Attention | |
| |---|---|---|---| |
| | `lemon_plstitch_block32.pt` | LEMON, PL-Stitch, 768-D, 1 fps | 768 | 32-token blocks | |
| | `assembly101_tsm_block64.pt` | Assembly101, TSM, 2048-D, 3.75 fps | 1408 | 64-token blocks | |
| | `assembly101_tsm_clip.pt` | Assembly101, TSM, 2048-D | 1408 | Clip-causal (oracle) | |
| | `egoprocel_fact_features_clip.pt` | EgoProceL, FACT I3D features, 2048-D, 4 fps | 1408 | Clip-causal (oracle) | |
| | `egoprocel_pooled_clip.pt` | EgoProceL, pooled V-JEPA features, 1408-D, 4 fps | 1408 | Clip-causal (oracle) | |
|
|
| **Use block-causal models going forward.** Clip-causal models require true |
| segment boundaries, which are generally unavailable in real-world inference. |
| They are historical oracle comparisons. Block-causal attention permits access |
| to the whole current block; it is not zero-latency frame-by-frame streaming. |
|
|
|  |
|
|
| **Frozen transfer / zero-shot encoder application to Cholec80.** |
| The LEMON-trained P-JEPA encoder is applied without fine-tuning on Cholec80. |
| The t-SNE panels compare PL-Stitch features (left) with frozen P-JEPA |
| representations (right): 100 frames sampled uniformly from each of the 40 |
| held-out videos, colored by relative time from start (purple) to end (yellow). |
| Cholec80 development labels were used for checkpoint selection; supervised |
| linear probes are separate from this representation visualization. |
|
|
| ## Use |
|
|
| Install the [P-JEPA code](https://github.com/FelTris/pjepa) (Python 3.11+, PyTorch 2.6+; currently private pending release) and |
| `huggingface_hub`, then: |
|
|
| ```python |
| import torch |
| from huggingface_hub import hf_hub_download |
| from pjepa.checkpoints import load_model |
| from pjepa.inference.encoder import encode |
| |
| path = hf_hub_download("FelTris/pjepa", "lemon_plstitch_block32.pt") |
| model, metadata = load_model(path, device="cpu") |
| features = torch.randn(1, 128, 768) # replace with PL-Stitch features |
| representations = encode(model, features) |
| ``` |
|
|
| Oracle encoders additionally require `segment_lengths`. Use the matching feature |
| backbone and timing for each checkpoint. Downstream classification/segmentation |
| requires a separately fitted head; small linear heads accompany the code release. |
|
|
| Precomputed inputs and matching annotations are available at |
| [FelTris/pjepa_features](https://huggingface.co/datasets/FelTris/pjepa_features). |
| The dataset card documents sampling rates, feature extractors, and download instructions. |
|
|
| ## Protocol and validation |
|
|
| The FACT-input P-JEPA encoder was trained on the original mixed-view |
| EgoProceL split (731 train / 183 test). The public feature archive now matches |
| the pooled archive's 337 train / 86 test video IDs, excluding three CMU static |
| camera views. Its saved encoder weights are unchanged; the current archive does |
| not reproduce its original training-data coverage. |
|
|
| LEMON encoder selection uses Cholec80 development videos; the test videos are |
| held out. Historical EgoProceL runs use a split named `test` for selection, so |
| it must not be described as untouched test evaluation. These are research |
| representations, not validated clinical decision systems. |
|
|
| Strict loading and numerical parity were checked against the source models, |
| including complete-video GPU inference. No new benchmark scores are claimed |
| here. See [the manifest](manifest.json) for architectures and artifact hashes; |
| [SHA256SUMS](SHA256SUMS) verifies the downloads. Dataset and input-feature terms |
| remain separate from the encoder weight license. |
|
|
