Add files using upload-large-folder tool

README.md

@@ -26,6 +26,8 @@ can stop after any number of tokens *k* per frame and still produce a semantical
 This repository holds the paper's tokenizers and AR models for both SemanTok and the VideoFlexTok
 baseline, on Kinetics-600 (class-to-video) and uCO3D (text-to-video).
 
+![SemanTok overview: the tokenizer (left) and AR generation (right)](./assets/overview.png)
+
 Please note: For individuals or organizations generating annual revenue of US $1,000,000 (or local currency equivalent) or more, regardless of the source of that revenue, you must obtain an enterprise commercial license directly from Stability AI before commercially using SemanTok, derivative works of SemanTok, or outputs from SemanTok. See https://stability.ai/license and https://stability.ai/enterprise.
 
 ## Model Description
@@ -77,11 +79,11 @@ video = tok.decode(tokens, k=16)                     # decode from the first 16
 
 ## Files
 
-Every model is a directory with `config.json` and `model.safetensors`:
+Every model is a directory with `config.json` and `model.safetensors` (bf16):
 
 ```
 tokenizers/{k600,uco3d}-{videoflextok,semantok}/
-ar/{k600,uco3d}-{videoflextok,semantok}-d{10,12,16,20,24,30,36}/
+ar/{k600,uco3d}-{videoflextok,semantok}-d{10,12,16,20,24}/
 ```
 
 | tokenizer | data | training |
@@ -89,18 +91,16 @@ ar/{k600,uco3d}-{videoflextok,semantok}-d{10,12,16,20,24,30,36}/
 | `k600-videoflextok`, `k600-semantok` | Kinetics-600 | 200k steps (131B tokens) |
 | `uco3d-videoflextok`, `uco3d-semantok` | uCO3D | 100k steps (66B tokens) |
 
-| AR depth | d10 | d12 | d16 | d20 | d24 | d30 | d36 |
-|---|---|---|---|---|---|---|---|
-| parameters | 49M | 85M | 201M | 393M | 679M | 1.33B | 2.29B |
+| AR depth | d10 | d12 | d16 | d20 | d24 |
+|---|---|---|---|---|---|
+| parameters | 49M | 85M | 201M | 393M | 679M |
 
 - Kinetics-600 AR models are class-conditioned (597 classes); uCO3D AR models are conditioned on umT5 caption embeddings.
-- Kinetics-600 d10–d24 AR models were trained for 20k steps and d30/d36 for 40k; all uCO3D AR models for 20k.
-- AR weights are the raw (non-EMA) weights the paper evaluated.
+- All AR models here were trained for 20k steps. The paper's larger d30 (1.33B) and d36 (2.29B) models are not included.
+- Weights are stored in bf16; the paper evaluated the same weights in fp32 under bf16 autocast.
 - The tokenizers do not include the VidTok VAE when it is identical to the public one; the loader fetches it from [`EPFL-VILAB/videoflextok_d18_d18_k600`](https://huggingface.co/EPFL-VILAB/videoflextok_d18_d18_k600).
 - `SHA256SUMS` lists the checksum of every `model.safetensors`.
 
-**Training differences between the arms.** The VideoFlexTok tokenizers aggregated DINO features over a latent frame with its first frame and SemanTok with the frame average; this setting also sets the decoder-REPA target. The Kinetics-600 SemanTok tokenizer trained its first 15k steps without the per-frame class-token fix.
-
 ## License
 
 - Community License: Free for research, non-commercial, and commercial use by organizations and individuals generating annual revenue of US $1,000,000 (or local currency equivalent) or less, regardless of the source of that revenue.

SHA256SUMS

@@ -1,32 +1,24 @@
-ff5f847ea76c8bde202cdd0fa6bf09635664b4af6b803491d47d257ad718e231  ./ar/k600-semantok-d10/model.safetensors
-fd97f97e04517282eb550a97e7f7b00316eb3687824ccf6e44358db66250e9a2  ./ar/k600-semantok-d12/model.safetensors
-2ee5781ab7285eeaaa73a6f9fab83506003b2adf070eadb04a5c4db1137fe5ec  ./ar/k600-semantok-d16/model.safetensors
-ff8d51326bf6c8ac7944fd587bf0b7a52b948da6f936a613af92f8d462b0183f  ./ar/k600-semantok-d20/model.safetensors
-7752200ccae0af59acf57a3097bfe9cb706da7bee39387c490ac793c07070463  ./ar/k600-semantok-d24/model.safetensors
-36e592b55e65574d2e40110cee6a6db9b9d1504aabfcfa980e4e49c69e9e5dd7  ./ar/k600-semantok-d30/model.safetensors
-532d806584e94037d62c3193ca005287f9660205161de76373a49b33fe705d5d  ./ar/k600-semantok-d36/model.safetensors
-65466b1044a9f267ebb7ad3270e27ce92e83c52776d0f80fc451aa4dac9d8f8e  ./ar/k600-videoflextok-d10/model.safetensors
-eb53df5f402bf1d53bc4332c3b1fbaf315aba6f61049e976a5e4d889eceb2eb4  ./ar/k600-videoflextok-d12/model.safetensors
-4569292ff82da7717e35b0eef67fe411510c82526e10837cc40469c866e1f91c  ./ar/k600-videoflextok-d16/model.safetensors
-7bfc3562c393e8391a9c775284c9335116c535fb097693043f924787221fa86a  ./ar/k600-videoflextok-d20/model.safetensors
-9ef5661437626f1b31c1052262c2aa4bd107391b40838630f3c31d05d3f768ff  ./ar/k600-videoflextok-d24/model.safetensors
-ef5a5b89b754e241d6aad9596936b75ca2ca257b5fe93c95a69ec2d6bcf48f8f  ./ar/k600-videoflextok-d30/model.safetensors
-c8aa14f38e38df6a7fcac9aea023acbd6e83bfa1ede9b030f5c8d26ab744a1e9  ./ar/k600-videoflextok-d36/model.safetensors
-a62978bebed7f2503b55e9f0367fe16ecd48361a0eac7ef14e54465a46073ed6  ./ar/uco3d-semantok-d10/model.safetensors
-c9ae6ac4631c8ce6c1bb28aab18fd9ab890a4ab4ef97e08a80a7d62ba12ee3f5  ./ar/uco3d-semantok-d12/model.safetensors
-4c2b94c7be392140e3a24683d1028bb72d76835d855e7ed683427239ce9f91b8  ./ar/uco3d-semantok-d16/model.safetensors
-01b9ab8c3533d2b096b4feac38d1d9c6645d79a9c318728902f4c6f0ef7b15b2  ./ar/uco3d-semantok-d20/model.safetensors
-68ef1349cd67bc52d8a489a7a21f800361f1f04b7c1a8e59984e924ee31e8f79  ./ar/uco3d-semantok-d24/model.safetensors
-9fa91df1447ce0abd57488dbe28cdf36c43813146d7a7e4470429a3a3683bba4  ./ar/uco3d-semantok-d30/model.safetensors
-eb9884114816bfc715173d8324fe05f979f1909cdfe56bfc95481b12313e9cc8  ./ar/uco3d-semantok-d36/model.safetensors
-a92d5b2f5bd2d96a687975c5ad147ec60383179f18e080af3f9aeeabc866b6c9  ./ar/uco3d-videoflextok-d10/model.safetensors
-01645f4a0fde5b69ce390eaf6856f4e07dc2a7256dae91936bd027dcb1b6a5bd  ./ar/uco3d-videoflextok-d12/model.safetensors
-fc2f9480537deb186a30505aee1002392b45661f4231e44db64bdbb68b36ef78  ./ar/uco3d-videoflextok-d16/model.safetensors
-fd02d410692ea07d12bdfa01a50c2a6f9a371ffb04b4687b13ed72a70520baad  ./ar/uco3d-videoflextok-d20/model.safetensors
-433cc4c5322be234c294ca85c854da8358b8d89d275abe83007842c423ff9069  ./ar/uco3d-videoflextok-d24/model.safetensors
-8ecfc3b8813466b9eb349c92c937d201040415255aa1efbd5437a97cd69cbaac  ./ar/uco3d-videoflextok-d30/model.safetensors
-c73e06891e25d405e3edc39a350c60372279355ce24f0a690f5d689abc837d6f  ./ar/uco3d-videoflextok-d36/model.safetensors
-f7ea784c98bade273203978cf61f49e0a6848121b1586c3296075b6a770482bc  ./tokenizers/k600-semantok/model.safetensors
-af997a2a45b5da8bf1f6f4792a88271fe6a545928dd98e3be9cc4eb0526b1dd3  ./tokenizers/k600-videoflextok/model.safetensors
-e4620539b16e4f7831fd104a5aac9a127111477b393153bb68ddb838307a38e8  ./tokenizers/uco3d-semantok/model.safetensors
-92a7e721c052ce1dd7f62295578684589f892ed5d38201af0174f6e25872e292  ./tokenizers/uco3d-videoflextok/model.safetensors
+788e9b21f9edfb7a38d898b0320e34ea0316145d40bda3fb0012483128e3db71  ar/k600-semantok-d10/model.safetensors
+7ea94704e0781c12f361372e3154259817c19e59e34cde0fe650448d4b79d5ad  ar/k600-semantok-d12/model.safetensors
+b749f8a53cf0705902187e721c3c0f5e77299bd35f481709cbcfe04289d1d04f  ar/k600-semantok-d16/model.safetensors
+cfe3e91973646a94d705371caa97321b9763932a9820ecb6f1a3fcdadc4ae6dc  ar/k600-semantok-d20/model.safetensors
+0bc97e5668aafc564e4711985d09d9c5d32fa7f72e3461ed81c0c31e9265c869  ar/k600-semantok-d24/model.safetensors
+e04c88292f27ce6a8fe7514acd2a0dca781f4e48a0aeb6334beee0226f8ceedd  ar/k600-videoflextok-d10/model.safetensors
+3577be44533cb48b118cd955a22878614f49df540ffab294879ea58804ec5ffa  ar/k600-videoflextok-d12/model.safetensors
+8c502553e206c4299d2cf79eb312a2ea9629915a78800d73c93984d3e0bb5333  ar/k600-videoflextok-d16/model.safetensors
+725cbbdb3e34edb559fc7c4365ffbc09fcb20762f7727d2d0e1931044cd5a771  ar/k600-videoflextok-d20/model.safetensors
+9639dcad36d369b3272ef0e0ec599a2dd69e59ae282a755b26f814b35cd87633  ar/k600-videoflextok-d24/model.safetensors
+312a4d2e7f2b2885a6383c85d4e4978399f0caeeaef2112cac718e4d68872342  ar/uco3d-semantok-d10/model.safetensors
+2ff38d216482af812042664944bbca9444cbd258b95364042b4ce017c62cd9cf  ar/uco3d-semantok-d12/model.safetensors
+5c300d6963b7be7507273e72459d3501203d6d0edea20d9c4ac4371aad45ffb5  ar/uco3d-semantok-d16/model.safetensors
+9f74a5eca722e9b707146159b7f61dcaa54b71d9dc23f4e05b1951160ce08ad4  ar/uco3d-semantok-d20/model.safetensors
+774803d4fdf7c920b9ab7669281070432bc953897a78f0dd0007d758bc90061d  ar/uco3d-semantok-d24/model.safetensors
+5af563fb32b3d2710f08f327d637755bb8a30b65a9e3b3340881e74e61316685  ar/uco3d-videoflextok-d10/model.safetensors
+4bd033332df7e4f96b34c482ecf645a8ee91170dd9739ceeb7cd7ba88aa803e5  ar/uco3d-videoflextok-d12/model.safetensors
+4f1aef2ec9a68a0cac490893eaa8e407519d8c00101c2c650e9f0cf192e14f15  ar/uco3d-videoflextok-d16/model.safetensors
+64ef5331ead35bbf8b5792802c71861eacad7bc4527daabf6a016bc407a632ec  ar/uco3d-videoflextok-d20/model.safetensors
+40d50768abb4f4cb24ad34bdad22d6be2193b310acb4e04119819fe392dcd964  ar/uco3d-videoflextok-d24/model.safetensors
+c58bcee2dee218418355dad8102380a9370b3af77a0cc6212d3f0a72231523c8  tokenizers/k600-semantok/model.safetensors
+fc39ca1f3a4289c7b19e57ade7665b705a918651f673d5c7df2b87ab6600ca96  tokenizers/k600-videoflextok/model.safetensors
+d26ee492afe0ea4b1be767f23cc059817c2f340e6878a7dc6bbf32221e870e14  tokenizers/uco3d-semantok/model.safetensors
+50a3d430d5c8c80c60775e8bf71e742bf8f3078fc140075b2f7d6b946a7f04af  tokenizers/uco3d-videoflextok/model.safetensors