Instructions to use iapple1/CoCode-World-Model with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Inference
- Notebooks
- Google Colab
- Kaggle
Refresh the official English model card
Browse files- README.md +40 -24
- SHA256SUMS.txt +1 -1
README.md
CHANGED
|
@@ -2,9 +2,10 @@
|
|
| 2 |
license: apache-2.0
|
| 3 |
language:
|
| 4 |
- en
|
| 5 |
-
- zh
|
| 6 |
base_model: MiniMaxAI/MiniMax-H3
|
| 7 |
base_model_relation: adapter
|
|
|
|
|
|
|
| 8 |
pipeline_tag: image-to-video
|
| 9 |
tags:
|
| 10 |
- lora
|
|
@@ -12,46 +13,61 @@ tags:
|
|
| 12 |
- video-generation
|
| 13 |
- multi-view
|
| 14 |
- world-model
|
| 15 |
-
- decfg
|
| 16 |
---
|
| 17 |
|
| 18 |
-
# CoCode-World
|
| 19 |
|
| 20 |
-
|
| 21 |
|
| 22 |
-
|
| 23 |
-
|
| 24 |
-
|
| 25 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 26 |
|
| 27 |
## Download
|
| 28 |
|
| 29 |
```bash
|
|
|
|
| 30 |
hf download iapple1/CoCode-World-Model --local-dir checkpoints/step-004000
|
| 31 |
```
|
| 32 |
|
| 33 |
-
|
| 34 |
-
|
| 35 |
-
|
| 36 |
-
|
| 37 |
-
|
| 38 |
-
|
| 39 |
-
|
|
|
|
|
|
|
|
|
|
| 40 |
|
| 41 |
-
|
| 42 |
|
| 43 |
-
## Training
|
| 44 |
|
| 45 |
-
|
| 46 |
|
| 47 |
-
|
| 48 |
|
| 49 |
-
|
| 50 |
|
| 51 |
-
|
| 52 |
|
| 53 |
-
|
| 54 |
|
| 55 |
-
|
| 56 |
|
| 57 |
-
|
|
|
|
| 2 |
license: apache-2.0
|
| 3 |
language:
|
| 4 |
- en
|
|
|
|
| 5 |
base_model: MiniMaxAI/MiniMax-H3
|
| 6 |
base_model_relation: adapter
|
| 7 |
+
datasets:
|
| 8 |
+
- iapple1/CoCode-World-Data
|
| 9 |
pipeline_tag: image-to-video
|
| 10 |
tags:
|
| 11 |
- lora
|
|
|
|
| 13 |
- video-generation
|
| 14 |
- multi-view
|
| 15 |
- world-model
|
|
|
|
| 16 |
---
|
| 17 |
|
| 18 |
+
# CoCode-World
|
| 19 |
|
| 20 |
+
**Shared 3D Proxy Representation for Multi-Player World Models**
|
| 21 |
|
| 22 |
+
[Project Page](https://iapple233.github.io/CCW/) · [Code](https://github.com/IApple233/CoCode-World) · [Dataset](https://huggingface.co/datasets/iapple1/CoCode-World-Data) · [Examples](https://github.com/IApple233/CoCode-World/tree/main/examples/evaluation)
|
| 23 |
+
|
| 24 |
+
## Model overview
|
| 25 |
+
|
| 26 |
+
CoCode-World generates synchronized video from two player perspectives using a shared 3D proxy representation. A reference image defines the visual appearance, a proxy video provides scene structure and motion, and a text prompt describes the scene.
|
| 27 |
+
|
| 28 |
+
This repository provides the official CoCode-World LoRA checkpoint for **MiniMax-H3 Ref2VA**. The recommended EMA weights are used in the released evaluation examples.
|
| 29 |
+
|
| 30 |
+
| Property | Description |
|
| 31 |
+
|---|---|
|
| 32 |
+
| Base model | MiniMax-H3 Ref2VA |
|
| 33 |
+
| Adapter | Rank-64 LoRA |
|
| 34 |
+
| Checkpoint | Step 4000 |
|
| 35 |
+
| Inputs | Reference image, synchronized proxy video, and text prompt |
|
| 36 |
+
| Output | Two player views arranged side by side |
|
| 37 |
+
| Resolution | 2560 × 720 at 24 FPS |
|
| 38 |
|
| 39 |
## Download
|
| 40 |
|
| 41 |
```bash
|
| 42 |
+
pip install -U huggingface_hub
|
| 43 |
hf download iapple1/CoCode-World-Model --local-dir checkpoints/step-004000
|
| 44 |
```
|
| 45 |
|
| 46 |
+
The repository includes `ema.safetensors` for inference, `lora.safetensors` for the task adapter, and training state and configuration files. Use the checkpoint loader in the official code repository. Download the MiniMax-H3 base model separately.
|
| 47 |
+
|
| 48 |
+
## Usage
|
| 49 |
+
|
| 50 |
+
Follow the [installation guide](https://github.com/IApple233/CoCode-World/blob/main/docs/SETUP.md), then download the required models from the code repository:
|
| 51 |
+
|
| 52 |
+
```bash
|
| 53 |
+
python scripts/download_models.py base
|
| 54 |
+
python scripts/download_models.py cocode
|
| 55 |
+
```
|
| 56 |
|
| 57 |
+
The [evaluation guide](https://github.com/IApple233/CoCode-World/blob/main/docs/EVALUATION.md) explains input preparation and generation. Eight ready-to-use examples include reference images, prompts, proxy videos, and generated results.
|
| 58 |
|
| 59 |
+
## Training
|
| 60 |
|
| 61 |
+
Training starts from the official [MiniMax-H3 weights](https://huggingface.co/MiniMaxAI/MiniMax-H3) and [DiffSynth-Studio's Ref2VA DeCFG LoRA](https://www.modelscope.cn/models/DiffSynth-Studio/MiniMax-H3-TrainingAdapter). The DeCFG adapter is used as a frozen training preset, followed by training a separate CoCode-World task LoRA. Inference uses the original MiniMax-H3 base and the CoCode-World EMA LoRA.
|
| 62 |
|
| 63 |
+
Training code and instructions are available in the [training guide](https://github.com/IApple233/CoCode-World/blob/main/docs/TRAINING.md). The public dataset is a selected subset of the original training collection.
|
| 64 |
|
| 65 |
+
## Intended use and limitations
|
| 66 |
|
| 67 |
+
The model supports research on multi-player world models, synchronized multi-view generation, and proxy-conditioned video synthesis. The first-frame reference guides appearance. Generated videos may contain visual artifacts or temporal inconsistencies, and the selected examples do not constitute a quantitative benchmark.
|
| 68 |
|
| 69 |
+
## License and acknowledgments
|
| 70 |
|
| 71 |
+
The task adapter is released under Apache 2.0. The MiniMax-H3 base model is governed by the [MiniMax-H3 Community License Agreement](https://huggingface.co/MiniMaxAI/MiniMax-H3/blob/main/LICENSE). The DeCFG training adapter retains its upstream terms.
|
| 72 |
|
| 73 |
+
CoCode-World builds on [MiniMax-H3](https://huggingface.co/MiniMaxAI/MiniMax-H3), [DiffSynth-Studio](https://github.com/modelscope/DiffSynth-Studio), [Diffusers](https://github.com/huggingface/diffusers), and [Code World Model](https://github.com/buaacyw/code-world-model).
|
SHA256SUMS.txt
CHANGED
|
@@ -1,5 +1,5 @@
|
|
| 1 |
f9e2070c247517b1ddf65f7b11b393484a18da91a958fd18a97bd0f241c3125c LICENSE
|
| 2 |
-
|
| 3 |
9d4e0524ee2b9203c2711f2c0b2a133e17d9ea6cb1280aa9974150f0ba7a038c ema.safetensors
|
| 4 |
e0d4a3cb1c8cb8cb8ef36cefeb45ecc164556404f4003ec39b1af8f435ca602c lora.safetensors
|
| 5 |
99c54ce47fe0aa7fa170ddd530b8de7abed0f02dcc970750c63c545021269f33 model_manifest.json
|
|
|
|
| 1 |
f9e2070c247517b1ddf65f7b11b393484a18da91a958fd18a97bd0f241c3125c LICENSE
|
| 2 |
+
0cf9f82ef2c0845eb5e6f06d45b508016396970928e841b04ae09a8fcad17dd4 README.md
|
| 3 |
9d4e0524ee2b9203c2711f2c0b2a133e17d9ea6cb1280aa9974150f0ba7a038c ema.safetensors
|
| 4 |
e0d4a3cb1c8cb8cb8ef36cefeb45ecc164556404f4003ec39b1af8f435ca602c lora.safetensors
|
| 5 |
99c54ce47fe0aa7fa170ddd530b8de7abed0f02dcc970750c63c545021269f33 model_manifest.json
|