Add comprehensive model card for RealGeneral

README.md

@@ -0,0 +1,82 @@
+---
+pipeline_tag: text-to-video
+library_name: diffusers
+license: apache-2.0
+tags:
+  - video-generation
+  - image-generation
+---
+
+# RealGeneral: Unifying Visual Generation via Temporal In-Context Learning with Video Models
+
+This repository contains the **RealGeneral** model, a novel framework that unifies diverse visual generation tasks by leveraging video models and temporal in-context learning. This work is presented in the paper: [RealGeneral: Unifying Visual Generation via Temporal In-Context Learning with Video Models](https://huggingface.co/papers/2503.10406).
+
+RealGeneral reformulates image generation as a conditional frame prediction task, analogous to in-context learning in Large Language Models (LLMs). It explores video models as a foundation for unified image generation, introducing:
+1.  A **Unified Conditional Embedding** module for multi-modal alignment.
+2.  A **Unified Stream DiT Block** with decoupled adaptive LayerNorm and attention mask to mitigate cross-modal interference.
+
+This approach demonstrates effectiveness in multiple important visual generation tasks, such as customized generation and canny-to-image translation, showcasing significant improvements in subject similarity and image quality.
+
+*   **Project Page:** [https://lyne1.github.io/realgeneral_web/](https://lyne1.github.io/realgeneral_web/)
+*   **GitHub Repository:** [https://github.com/Lyne1/Realgeneral](https://github.com/Lyne1/Realgeneral)
+
+<div align="center">
+    <a href='https://lyne1.github.io/realgeneral_web/'><img src="https://github.com/user-attachments/assets/0c4448a4-93f3-4a63-acc7-488657439e37" alt="RealGeneral Teaser" style="width: 100%; max-width: 650px;"></a>
+</div>
+
+## Usage
+
+You can use the RealGeneral model with the Hugging Face `diffusers` library. This model is based on the `CogVideoXPipeline`, designed for text-to-video generation, and can be adapted for various conditional image generation tasks.
+
+First, make sure you have `diffusers` and other necessary dependencies installed. You might need to install `transformers` and `accelerate` as well:
+
+```bash
+pip install diffusers transformers accelerate torch
+```
+
+Here's a basic example for text-to-video generation using the `CogVideoXPipeline`:
+
+```python
+import torch
+from diffusers import CogVideoXPipeline
+
+# Load the pipeline
+# The 'trust_remote_code=True' is necessary as CogVideoXPipeline is a custom pipeline.
+pipeline = CogVideoXPipeline.from_pretrained(
+    "lyneeeeeeeee/RealGeneral",
+    torch_dtype=torch.float16,
+    trust_remote_code=True
+)
+pipeline.to("cuda")
+
+# Define your prompt
+prompt = "A robot walking in a futuristic city"
+
+# Generate video frames
+# Adjust num_frames, height, and width as needed and based on model capabilities
+generated_video = pipeline(prompt=prompt, num_frames=16, height=512, width=512).videos
+
+# The 'generated_video' will be a list of frames. You can then save it as a GIF or MP4.
+# Example: To save the first generated video (if multiple are generated)
+# from diffusers.utils import export_to_gif
+# export_to_gif(generated_video[0], "robot_video.gif")
+
+print(f"Video generation complete for prompt: '{prompt}'.")
+print("Refer to the project's GitHub repository for detailed instructions on advanced usage, including specific conditional image generation tasks (like Canny-to-Image) and training.")
+```
+
+## Citation
+
+If you find this work useful in your research, please consider citing our paper:
+
+```bibtex
+@misc{lin2025realgeneralunifyingvisualgeneration,
+      title={RealGeneral: Unifying Visual Generation via Temporal In-Context Learning with Video Models},
+      author={Yijing Lin and Mengqi Huang and Shuhan Zhuang and Zhendong Mao},
+      year={2025},
+      eprint={2503.10406},
+      archivePrefix={arXiv},
+      primaryClass={cs.CV},
+      url={https://arxiv.org/abs/2503.10406},
+}
+```