# Ruyi-Models English | [简体中文](./README_CN.md) Welcome to Ruyi-Models! Ruyi is an image-to-video model capable of generating cinematic-quality videos at a **resolution of 768**, with a frame rate of **24 frames per second**, totaling **5 seconds and 120 frames**. It supports **lens control** and **motion amplitude control**. Using a **RTX 3090 or RTX 4090**, you can generate 512 resolution, 120 frames (or 768 resolution, ~72 frames) videos **without any loss of quality**. ## News - Jan 14, 2025: [TeaCache](https://github.com/ali-vilab/TeaCache) and [Enhance-A-Video](https://github.com/NUS-HPC-AI-Lab/Enhance-A-Video) have been added as nodes for ComfyUI, offering faster generation speeds and improved video quality. Simply link these nodes before the sampler node to use them. - Jan 6, 2025: FP8 added for ComfyUI node. GPU memory decreases depending on the modes: bf16 default > fp8 lite > fp8 strong > fp8 extreme. - Dec 24, 2024: The diffusion model is updated to fix the black lines when creating 3:4 or 4:5 videos. ## Table of Contents - [Installation Instructions](#installation-instructions) - [Download Model (Optional)](#download-model-optional) - [How to Use](#how-to-use) - [Showcase](#showcase) - [GPU Memory Optimization](#gpu-memory-optimization) - [License](#license) ## Installation Instructions The installation instructions are simple. Just clone the repo and install the requirements. ```shell git clone https://github.com/IamCreateAI/Ruyi-Models cd Ruyi-Models pip install -r requirements.txt ``` ### For ComfyUI Users #### Method (1): Installation via ComfyUI Manager Download and install [ComfyUI-Manager](https://github.com/ltdrdata/ComfyUI-Manager). ```shell cd ComfyUI/custom_nodes/ git clone https://github.com/ltdrdata/ComfyUI-Manager.git # install requirements pip install -r ComfyUI-Manager/requirements.txt ``` Next, start ComfyUI and open the Manager. Select Custom Nodes Manager, then search for "Ruyi". You should see ComfyUI-Ruyi as shown in the screenshot below. Click "Install" to proceed.

Finally, search for "ComfyUI-VideoHelperSuite" and install it as well. #### Method (2): Manual Installation Download and save this repository to the path *ComfyUI/custom_nodes/Ruyi-Models*. ```shell # download the repo cd ComfyUI/custom_nodes/ git clone https://github.com/IamCreateAI/Ruyi-Models.git # install requirements pip install -r Ruyi-Models/requirements.txt ``` Install the dependency [ComfyUI-VideoHelperSuite](https://github.com/Kosinkadink/ComfyUI-VideoHelperSuite) to display video output (skip this step if already installed). ```shell # download ComfyUI-VideoHelperSuite cd ComfyUI/custom_nodes/ git clone https://github.com/Kosinkadink/ComfyUI-VideoHelperSuite.git # install requirements pip install -r ComfyUI-VideoHelperSuite/requirements.txt ``` ##### For Windows Users When using the Windows operating system, a common distribution is [ComfyUI_windows_portable_nvidia](https://github.com/comfyanonymous/ComfyUI/releases). When launched with `run_nvidia_gpu.bat`, it utilizes the embedded Python interpreter included with the package. Therefore, the environment needs to be set up within this built-in Python. For example, if the extracted directory of the distribution is ComfyUI_windows_portable, you can typically use the following command to download the repository and install the runtime environment: ```shell # download the repo cd ComfyUI_windows_portable\ComfyUI\custom_nodes git clone https://github.com/IamCreateAI/Ruyi-Models.git # install requirements using embedded Python interpreter ..\..\python_embeded\python.exe -m pip install -r Ruyi-Models\requirements.txt ``` ## Download Model (Optional) Download the model and save it to certain path. To directly run our model, it is recommand to save the models into _Ruyi-Models/models_ folder. For ComfyUI users, the path should be _ComfyUI/models/Ruyi_. | Model Name | Type | Resolution | Max Frames | Frames per Second | Storage Space | Download | | :---: | :---: | :---: | :---: | :---: | :---: | :---: | | Ruyi-Mini-7B | Image to Video | 512 & 768 | 120 | 24 | 17 GB | [🤗](https://huggingface.co/IamCreateAI/Ruyi-Mini-7B) | For example, after downloading Ruyi-Mini-7B, the file path structure should be: ``` 📦 Ruyi-Models/models/ or ComfyUI/models/Ruyi/ ├── 📂 Ruyi-Mini-7B/ │ ├── 📂 transformers/ │ ├── 📂 vae/ │ └── 📂 ... ``` > This repository **supports automatic model downloading**, but manual downloading provides more control. For instance, you can download the model to another location and then link it to the *ComfyUI/models/Ruyi* path using symbolic links or similar methods. ## How to Use We provide two ways to run our model. The first is directly using python code. ``` python3 predict_i2v.py ``` Specifically, the script downloads the model to the _Ruyi-Models/models_ folder and uses images from the [_assets_](./assets/) folder as the start and end frames for video inference. You can modify the variables in the script to replace the input images and set parameters such as video length and resolution. For users with more than 24GB of GPU memory, you can use predict_i2v_80g.py to enhance generation speed. For those with less GPU memory, we offer parameters to optimize memory usage, enabling the generation of higher resolution and longer videos by extending the inference time. The effects of these parameters can be found in the [GPU memory optimization section](#gpu-memory-optimization) section below. Or use ComfyUI wrapper in our github repo, the detail of ComfyUI nodes is described in [_comfyui/README.md_](./comfyui/README.md). ## Showcase ### Image to Video Effects

### Camera Control

input	left	right
static	up	down

### Motion Amplitude Control

motion 1

motion 2

motion 3

motion 4

## GPU Memory Optimization We provide the options **`GPU_memory_mode` and `GPU_offload_steps` to reduce GPU memory usage**, catering to different user needs. Generally speaking, using **less GPU memory requires more RAM and results in longer generation times**. Below is a reference table of expected GPU memory usage and generation times. Note that, the GPU memory reported below is the `max_memory_allocated()` value. The values read from nvidia-smi may be higher than the reported values because CUDA occupies some GPU memory (usually between 500 - 800 MiB), and PyTorch's caching mechanism also requests additional GPU memory. Additionally, the community and we have created a detailed table featuring various resolutions and option combinations, which can be found in the [gpu_memory_appendix.md](assets/gpu_memory_appendix.md). We encourage community members to help us complete the table. ### A100 Results - Resolution of 512 | Num frames | normal_mode + 0 steps | normal_mode + 10 steps | normal_mode + 7 steps | normal_mode + 5 steps | normal_mode + 1 steps | low_gpu_mode + 0 steps | | :---: | :---: | :---: | :---: | :---: | :---: | :---: | | 24 frames | 16119MiB
_01:01s_ | 15535MiB
_01:07s_ | 15340MiB
_01:13s_ | 15210MiB
_01:20s_ | 14950MiB
_01:32s_ | 4216MiB
_05:14s_ | | 48 frames | 18398MiB
_01:53s_ | 17230MiB
_02:15s_ | 16840MiB
_02:29s_ | 16580MiB
_02:32s_ | 16060MiB
_02:54s_ | 4590MiB
_09:59s_ | | 72 frames | 20678MiB
_03:00s_ | 18925MiB
_03:31s_ | 18340MiB
_03:53s_ | 17951MiB
_03:57s_ | 17171MiB
_04:25s_ | 6870MiB
_14:42s_ | | 96 frames | 22958MiB
_04:11s_ | 20620MiB
_04:54s_ | 19841MiB
_05:10s_ | 19321MiB
_05:14s_ | 18281MiB
_05:47s_ | 9150MiB
_19:17s_ | | 120 frames | 25238MiB
_05:42s_ | 22315MiB
_06:34s_ | 21341MiB
_06:59s_ | 20691MiB
_07:07s_ | 19392MiB
_07:41s_ | 11430MiB
_24:08s_ | - Resolution of 768 | Num frames | normal_mode + 0 steps | normal_mode + 10 steps | normal_mode + 7 steps | normal_mode + 5 steps | normal_mode + 1 steps | low_gpu_mode + 0 steps | | :---: | :---: | :---: | :---: | :---: | :---: | :---: | | 24 frames | 18971MiB
_02:06s_ | 17655MiB
_02:40s_ | 17217MiB
_02:39s_ | 16925MiB
_02:41s_ | 16339MiB
_03:13s_ | 5162MiB
_13:42s_ | | 48 frames | 24101MiB
_04:52s_ | 21469MiB
_05:44s_ | 20592MiB
_05:51s_ | 20008MiB
_06:00s_ | 18837MiB
_06:49s_ | 10292MiB
_20:58s_ | | 72 frames | 29230MiB
_08:24s_ | 25283MiB
_09:45s_ | 25283MiB
_09:45s_ | 23091MiB
_10:10s_ | 21335MiB
_11:10s_ | 15421MiB
_39:12s_ | | 96 frames | 34360MiB
_12:49s_ | 29097MiB
_14:41s_ | 27343MiB
_15:33s_ | 26174MiB
_15:44s_ | 23834MiB
_16:33s_ | 20550MiB
_43:47s_ | | 120 frames | 39489MiB
_18:21s_ | 32911MiB
_20:39s_ | 30719MiB
_21:34s_ | 29257MiB
_21:48s_ | 26332MiB
_23:02s_ | 25679MiB
_63:01s_ | ### RTX 4090 Results The values marked with `---` in the table indicate that an out-of-memory (OOM) error occurred, preventing generation. - Resolution of 512 | Num frames | normal_mode + 0 steps | normal_mode + 10 steps | normal_mode + 7 steps | normal_mode + 5 steps | normal_mode + 1 steps | low_gpu_mode + 0 steps | | :---: | :---: | :---: | :---: | :---: | :---: | :---: | | 24 frames | 16366MiB
_01:18s_ | 15805MiB
_01:26s_ | 15607MiB
_01:37s_ | 15475MiB
_01:36s_ | 15211MiB
_01:39s_ | 4211MiB
_03:57s_ | | 48 frames | 18720MiB
_02:21s_ | 17532MiB
_02:49s_ | 17136MiB
_02:55s_ | 16872MiB
_02:58s_ | 16344MiB
_03:01s_ | 4666MiB
_05:01s_ | | 72 frames | 21036MiB
_03:41s_ | 19254MiB
_04:25s_ | 18660MiB
_04:34s_ | 18264MiB
_04:36s_ | 17472MiB
_04:51s_ | 6981MiB
_06:36s_ | | 96 frames | -----MiB
_--:--s_ | 20972MiB
_06:18s_ | 20180MiB
_06:24s_ | 19652MiB
_06:36s_ | 18596MiB
_06:56s_ | 9298MiB
_10:03s_ | | 120 frames | -----MiB
_--:--s_ | -----MiB
_--:--s_ | 21704MiB
_08:50s_ | 21044MiB
_08:53s_ | 19724MiB
_09:08s_ | 11613MiB
_13:57s_ | - Resolution of 768 | Num frames | normal_mode + 0 steps | normal_mode + 10 steps | normal_mode + 7 steps | normal_mode + 5 steps | normal_mode + 1 steps | low_gpu_mode + 0 steps | | :---: | :---: | :---: | :---: | :---: | :---: | :---: | | 24 frames | 19223MiB
_02:38s_ | 17900MiB
_03:06s_ | 17448MiB
_03:18s_ | 17153MiB
_03:23s_ | 16624MiB
_03:34s_ | 5251MiB
_05:54s_ | | 48 frames | -----MiB
_--:--s_ | -----MiB
_--:--s_ | 20946MiB
_07:28s_ | 20352MiB
_07:35s_ | 19164MiB
_08:04s_ | 10457MiB
_10:55s_ | | 72 frames | -----MiB
_--:--s_ | -----MiB
_--:--s_ | -----MiB
_--:--s_ | -----MiB
_--:--s_ | -----MiB
_--:--s_ | 15671MiB
_18:52s_ | ## License We’re releasing the model under a permissive **Apache 2.0 license**. ## BibTeX ``` @misc{createai2024ruyi, title={Ruyi-Mini-7B}, author={CreateAI Team}, year={2024}, publisher = {GitHub}, journal = {GitHub repository}, howpublished={\url{https://github.com/IamCreateAI/Ruyi-Models}} } ``` ## Welcome Feedback and Collaborative Optimization We sincerely welcome everyone to actively provide valuable feedback and suggestions, and we hope to work together to optimize our services and products. Your words will help us better understand user needs, allowing us to continuously enhance the user experience. Thank you for your support and attention to our work! You are welcomed to join our [Discord](https://discord.com/invite/nueQFQwwGw) or Wechat Group (Scan QR code to add Ruyi Assistant and join the official group) for further discussion!

Name	Type	Default	Description
ruyi_model	RUYI_MODEL		—
lora_name	COMBO		0 options:
strength_model	FLOAT	1.00-100–100	—

Load LoRA

Inputs (3)

Outputs (1)