Super-squash branch 'main' using huggingface_hub

.gitattributes

@@ -0,0 +1,37 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+assets/GenerationUCO3D_Basketball.gif filter=lfs diff=lfs merge=lfs -text
+assets/GenerationK600_Yoga.gif filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,132 @@
+---
+license: other
+pipeline_tag: text-to-video
+tags:
+- video-tokenizer
+- text-to-video
+- class-to-video
+- autoregressive
+- checkpoint
+---
+
+# SemanTok
+
+Predictable Semantic Tokens for Efficient Autoregressive Video Generation
+
+<p align="center">
+  <img src="./assets/GenerationUCO3D_Basketball.gif" width="49%" alt="Text-to-video on uCO3D at token budgets k=4 to 256">
+  <img src="./assets/GenerationK600_Yoga.gif" width="49%" alt="Class-to-video on Kinetics-600 at token budgets k=4 to 256">
+</p>
+
+SemanTok is a flexible (coarse-to-fine) video tokenizer whose first tokens carry the clip's semantics.
+Every nested token prefix is trained to reconstruct DINOv2 features, so an autoregressive (AR) model
+can stop after any number of tokens *k* per frame and still produce a semantically faithful video. A
+201M SemanTok AR model matches or beats a VideoFlexTok AR model 3.4× its size.
+
+This repository holds the paper's tokenizers and AR models for both SemanTok and the VideoFlexTok
+baseline, on Kinetics-600 (class-to-video) and uCO3D (text-to-video).
+
+Please note: For individuals or organizations generating annual revenue of US $1,000,000 (or local currency equivalent) or more, regardless of the source of that revenue, you must obtain an enterprise commercial license directly from Stability AI before commercially using SemanTok, derivative works of SemanTok, or outputs from SemanTok. See https://stability.ai/license and https://stability.ai/enterprise.
+
+## Model Description
+
+- Developed by: [Stability AI](https://stability.ai/)
+- Authors: Mikhail Dereviannykh, Vikram Voleti, Simon Donné, Mallikarjun Byrasandra Ramalinga Reddy, Shimon Vainer, Mark Boss
+- Model type: Flexible-length video tokenizer and class- / text-conditioned autoregressive video generation models
+- Model details: the tokenizer encodes each of 5 latent frames (17 frames, 128×128) into 256 ordered FSQ tokens (64k codes); any prefix of *k* tokens per frame decodes to a video through a rectified-flow decoder. SemanTok adds DINOv2 features to the encoder input, the DINO class token to the first register token, and dense and class DINO heads on every kept prefix during training (the heads are not shipped). The AR models are LLaMA-style transformers that predict tokens coarse to fine up to the chosen budget *k*.
+- Base stack: [VideoFlexTok](https://github.com/apple/ml-videoflextok) (tokenizer architecture), [VidTok](https://github.com/microsoft/VidTok) VAE, [DINOv2](https://github.com/facebookresearch/dinov2)-L, umT5 text encoder (via Wan 2.1) for uCO3D captions
+
+## Model Sources
+
+- Repository: https://github.com/Stability-AI/SemanTok
+- Project page: https://semantoken.github.io/
+- Paper: coming soon
+
+For installation, evaluation protocols, and data preparation, please use the [GitHub repository](https://github.com/Stability-AI/SemanTok).
+
+## Usage
+
+```bash
+git clone https://github.com/Stability-AI/SemanTok && cd SemanTok
+pip install -e .          # Python >= 3.10, CUDA GPU
+huggingface-cli download StabilityLabs/SemanTok --local-dir ckpts
+CKPT=ckpts
+
+# Reconstruct a clip from its first k tokens per frame
+python scripts/reconstruct.py --ckpt-root $CKPT --tokenizer k600-semantok --video clip.mp4 --ks 1 4 16 64 256
+
+# Class-to-video (Kinetics-600)
+python scripts/generate.py --ckpt-root $CKPT --ar k600-semantok-d16 --class-name "yoga" --ks 4 16 64 256
+
+# Text-to-video (uCO3D)
+python scripts/generate.py --ckpt-root $CKPT --ar uco3d-semantok-d16 \
+    --prompt "A small orange basketball on a plaid tablecloth" --ks 4 16 64
+```
+
+From Python:
+
+```python
+from semantok import Tokenizer
+from semantok.data.video import load_kinetics_clip
+
+tok = Tokenizer(f"{CKPT}/tokenizers/k600-semantok")
+clip = load_kinetics_clip("clip.mp4")[None]          # [1, 3, 17, 128, 128] in [-1, 1]
+tokens = tok.encode(clip)                            # [1, 5, 256] FSQ ids, coarse to fine
+video = tok.decode(tokens, k=16)                     # decode from the first 16 tokens per frame
+```
+
+## Files
+
+Every model is a directory with `config.json` and `model.safetensors`:
+
+```
+tokenizers/{k600,uco3d}-{videoflextok,semantok}/
+ar/{k600,uco3d}-{videoflextok,semantok}-d{10,12,16,20,24,30,36}/
+```
+
+| tokenizer | data | training |
+|---|---|---|
+| `k600-videoflextok`, `k600-semantok` | Kinetics-600 | 200k steps (131B tokens) |
+| `uco3d-videoflextok`, `uco3d-semantok` | uCO3D | 100k steps (66B tokens) |
+
+| AR depth | d10 | d12 | d16 | d20 | d24 | d30 | d36 |
+|---|---|---|---|---|---|---|---|
+| parameters | 49M | 85M | 201M | 393M | 679M | 1.33B | 2.29B |
+
+- Kinetics-600 AR models are class-conditioned (597 classes); uCO3D AR models are conditioned on umT5 caption embeddings.
+- Kinetics-600 d10–d24 AR models were trained for 20k steps and d30/d36 for 40k; all uCO3D AR models for 20k.
+- AR weights are the raw (non-EMA) weights the paper evaluated.
+- The tokenizers do not include the VidTok VAE when it is identical to the public one; the loader fetches it from [`EPFL-VILAB/videoflextok_d18_d18_k600`](https://huggingface.co/EPFL-VILAB/videoflextok_d18_d18_k600).
+- `SHA256SUMS` lists the checksum of every `model.safetensors`.
+
+**Training differences between the arms.** The VideoFlexTok tokenizers aggregated DINO features over a latent frame with its first frame and SemanTok with the frame average; this setting also sets the decoder-REPA target. The Kinetics-600 SemanTok tokenizer trained its first 15k steps without the per-frame class-token fix.
+
+## License
+
+- Community License: Free for research, non-commercial, and commercial use by organizations and individuals generating annual revenue of US $1,000,000 (or local currency equivalent) or less, regardless of the source of that revenue.
+- If your annual revenue exceeds US $1M, any commercial use of this model or derivative works requires an Enterprise License directly from Stability AI.
+
+Please refer to Stability AI's Community License at https://stability.ai/license for more information.
+
+## Intended Uses
+
+Intended uses include research on video tokenization and autoregressive video generation, in particular on generation at a variable token budget, and reproduction of the paper's evaluations.
+
+## Out-of-Scope Uses
+
+SemanTok is not intended to generate factual representations of people, events, products, or places. Use must comply with Stability AI's Acceptable Use Policy and license terms.
+
+## Citation
+
+```bibtex
+@misc{dereviannykh2026semantok,
+  title  = {SemanTok: Predictable Semantic Tokens for Efficient Autoregressive Video Generation},
+  author = {Dereviannykh, Mikhail and Voleti, Vikram and Donn{\'e}, Simon and
+            Reddy, Mallikarjun Byrasandra Ramalinga and Vainer, Shimon and Boss, Mark},
+  year   = {2026}
+}
+```
+
+## Safety
+
+Users should evaluate generated outputs for their own application constraints and apply additional mitigations where needed. Report safety issues to safety@stability.ai and security issues to security@stability.ai.

SHA256SUMS

@@ -0,0 +1,32 @@
+ff5f847ea76c8bde202cdd0fa6bf09635664b4af6b803491d47d257ad718e231  ./ar/k600-semantok-d10/model.safetensors
+fd97f97e04517282eb550a97e7f7b00316eb3687824ccf6e44358db66250e9a2  ./ar/k600-semantok-d12/model.safetensors
+2ee5781ab7285eeaaa73a6f9fab83506003b2adf070eadb04a5c4db1137fe5ec  ./ar/k600-semantok-d16/model.safetensors
+ff8d51326bf6c8ac7944fd587bf0b7a52b948da6f936a613af92f8d462b0183f  ./ar/k600-semantok-d20/model.safetensors
+7752200ccae0af59acf57a3097bfe9cb706da7bee39387c490ac793c07070463  ./ar/k600-semantok-d24/model.safetensors
+36e592b55e65574d2e40110cee6a6db9b9d1504aabfcfa980e4e49c69e9e5dd7  ./ar/k600-semantok-d30/model.safetensors
+532d806584e94037d62c3193ca005287f9660205161de76373a49b33fe705d5d  ./ar/k600-semantok-d36/model.safetensors
+65466b1044a9f267ebb7ad3270e27ce92e83c52776d0f80fc451aa4dac9d8f8e  ./ar/k600-videoflextok-d10/model.safetensors
+eb53df5f402bf1d53bc4332c3b1fbaf315aba6f61049e976a5e4d889eceb2eb4  ./ar/k600-videoflextok-d12/model.safetensors
+4569292ff82da7717e35b0eef67fe411510c82526e10837cc40469c866e1f91c  ./ar/k600-videoflextok-d16/model.safetensors
+7bfc3562c393e8391a9c775284c9335116c535fb097693043f924787221fa86a  ./ar/k600-videoflextok-d20/model.safetensors
+9ef5661437626f1b31c1052262c2aa4bd107391b40838630f3c31d05d3f768ff  ./ar/k600-videoflextok-d24/model.safetensors
+ef5a5b89b754e241d6aad9596936b75ca2ca257b5fe93c95a69ec2d6bcf48f8f  ./ar/k600-videoflextok-d30/model.safetensors
+c8aa14f38e38df6a7fcac9aea023acbd6e83bfa1ede9b030f5c8d26ab744a1e9  ./ar/k600-videoflextok-d36/model.safetensors
+a62978bebed7f2503b55e9f0367fe16ecd48361a0eac7ef14e54465a46073ed6  ./ar/uco3d-semantok-d10/model.safetensors
+c9ae6ac4631c8ce6c1bb28aab18fd9ab890a4ab4ef97e08a80a7d62ba12ee3f5  ./ar/uco3d-semantok-d12/model.safetensors
+4c2b94c7be392140e3a24683d1028bb72d76835d855e7ed683427239ce9f91b8  ./ar/uco3d-semantok-d16/model.safetensors
+01b9ab8c3533d2b096b4feac38d1d9c6645d79a9c318728902f4c6f0ef7b15b2  ./ar/uco3d-semantok-d20/model.safetensors
+68ef1349cd67bc52d8a489a7a21f800361f1f04b7c1a8e59984e924ee31e8f79  ./ar/uco3d-semantok-d24/model.safetensors
+9fa91df1447ce0abd57488dbe28cdf36c43813146d7a7e4470429a3a3683bba4  ./ar/uco3d-semantok-d30/model.safetensors
+eb9884114816bfc715173d8324fe05f979f1909cdfe56bfc95481b12313e9cc8  ./ar/uco3d-semantok-d36/model.safetensors
+a92d5b2f5bd2d96a687975c5ad147ec60383179f18e080af3f9aeeabc866b6c9  ./ar/uco3d-videoflextok-d10/model.safetensors
+01645f4a0fde5b69ce390eaf6856f4e07dc2a7256dae91936bd027dcb1b6a5bd  ./ar/uco3d-videoflextok-d12/model.safetensors
+fc2f9480537deb186a30505aee1002392b45661f4231e44db64bdbb68b36ef78  ./ar/uco3d-videoflextok-d16/model.safetensors
+fd02d410692ea07d12bdfa01a50c2a6f9a371ffb04b4687b13ed72a70520baad  ./ar/uco3d-videoflextok-d20/model.safetensors
+433cc4c5322be234c294ca85c854da8358b8d89d275abe83007842c423ff9069  ./ar/uco3d-videoflextok-d24/model.safetensors
+8ecfc3b8813466b9eb349c92c937d201040415255aa1efbd5437a97cd69cbaac  ./ar/uco3d-videoflextok-d30/model.safetensors
+c73e06891e25d405e3edc39a350c60372279355ce24f0a690f5d689abc837d6f  ./ar/uco3d-videoflextok-d36/model.safetensors
+f7ea784c98bade273203978cf61f49e0a6848121b1586c3296075b6a770482bc  ./tokenizers/k600-semantok/model.safetensors
+af997a2a45b5da8bf1f6f4792a88271fe6a545928dd98e3be9cc4eb0526b1dd3  ./tokenizers/k600-videoflextok/model.safetensors
+e4620539b16e4f7831fd104a5aac9a127111477b393153bb68ddb838307a38e8  ./tokenizers/uco3d-semantok/model.safetensors
+92a7e721c052ce1dd7f62295578684589f892ed5d38201af0174f6e25872e292  ./tokenizers/uco3d-videoflextok/model.safetensors