DreamX Creator T8
Native ComfyUI V3 nodes for DreamX-Creator joint audio-video generation and causal 2x refinement.
DreamX Team
</div> <div align="center">
:art: ComfyUI Native Nodes by T8star
This repository packages the released DreamX-Creator generator and causal 2x
refiner as native ComfyUI V3 nodes. It provides frontend-importable workflows,
native MODEL, CLIP, VAE, LATENT, AUDIO, GUIDER, and VIDEO types,
ComfyUI model offloading, checkpoint verification, and a guarded Windows SDPA
fallback for 24 GB GPUs.
Install DreamX Creator T8 from ComfyUI Manager, or install it manually:
cd ComfyUI/custom_nodes
git clone https://github.com/T8mars/Comfyui-DreamX-Creator-T8.git
cd Comfyui-DreamX-Creator-T8
python -m pip install -r requirements.txt
Download the model weights separately, restart ComfyUI, then drag one of these frontend workflows onto the canvas:
examples/dreamx_creator_ui.json— first-frame to synchronized video and audio.examples/dreamx_refiner_ui.json— separate 2x refinement pass that preserves audio and FPS.
See COMFYUI.md for the model layout, node graph, VRAM notes,
and verification commands. Model weights are not bundled with the node package.
:link: T8star Links
| Resource | Link | | --- | --- | | Bilibili | T8star on Bilibili | | YouTube | @T8star-Aix | | Seedance API | API signup | | Online AI Apps | RunningHub profile | | ComfyUI Package | Quark download | | Hugging Face | huggingface.co/t8star |
DreamX-Creator 1.0 is a research framework for native joint audio-video generation. Given a first frame and a text prompt, its implemented base generator jointly models modality-specialized video and audio streams, using Gated Cross-Modal Attention and Progressive Joint Training to enable bidirectional audio-video interaction.
The broader system combines Audio-Video Reinforcement Learning with Modality-Aware Multimodal Feedback to improve visual and audio quality, semantic consistency, and fine-grained audio-video synchronization. Autoregressive 1-Step 2K Refinement then upgrades the generated video to high-quality 2K output while preserving content, motion, and audio-aligned timing.
:clapper: Demo
<div align="center"> <video src="https://github.com/user-attachments/assets/e49fc64d-5b31-4c16-be1d-737dc3aef04b" controls></video> </div>:fire: News
- Sep 3, 2026: Open-sourced the model weights and released the inference code for the 7B joint audio-video generator and the Autoregressive 1-Step 2K Refiner.
- Sep 1, 2026: Initialized the DreamX-Creator project repository with its overview and release roadmap.
:calendar: Plan
- :heavy_check_mark: Initialize the DreamX-Creator project repository.
- :heavy_check_mark: Release the DreamX-Creator 1.0 technical report.
- :heavy_check_mark: Release validated model weights, inference code, and configurations.
- [ ] Release distilled, faster models with fewer sampling steps for reduced latency.
:open_file_folder: Repository Structure
COMFYUI.md— native ComfyUI V3 nodes, installation, workflows, and tests.audio_video_generation/— 7B native joint audio-video generator (single GPU). See its README for usage, input overrides, and memory options.video_refiner/— Autoregressive 1-step 2K refiner (SR-DiT 5B). See its README for usage and the full list of inference knobs.checkpoints/— All model weights (not in the git repo). See its README for the expected layout and download instructions.
:package: Model Weights
Model weights are distributed on HuggingFace
and ModelScope and should
be placed under checkpoints/ (details in checkpoints/README.md):
checkpoints/
├── creator/ # DreamX-Creator 1.0 joint generator (7B, LoRA merged)
│ ├── video_model/ # video DiT shards + config
│ ├── audio_model/ # audio DiT + config
│ └── cross_attn_weights.safetensors # gated A2V/V2A cross-modal attention
├── audio_vae/ # CreatorDACVAE audio VAE
├── refiner/ # 2K refiner
│ ├── sr_dit_5b.pt # SR-DiT 5B refiner
│ ├── latent_upsampler_flash.pt # FlashLatentUpsampler (default)
│ ├── latent_upsampler_2d_causal.pt # causal 2D latent upsampler (optional)
│ └── lightvae_nu_scheme3.pt # distilled fast decoder (optional, off by default)
└── wan2.2_ti2v_5b/ # shared Wan2.2-TI2V-5B dependencies
├── Wan2.2_VAE.pth # video VAE
├── models_t5_umt5-xxl-enc-bf16.pth # UMT5-xxl text encoder
└── google/umt5-xxl/ # tokenizer
The wan2.2_ti2v_5b/ directory can also be downloaded directly from
Wan-AI/Wan2.2-TI2V-5B; only the
three entries above are needed.
:rocket: Quickstart
Each subdirectory is self-contained with its own requirements.txt and README.
1. Joint audio-video generation (first frame + prompt to synchronized video with audio):
cd audio_video_generation
pip install -r requirements.txt
./inference.sh # runs the default Verse-Bench case (case1)
See audio_video_generation/README.md for input overrides, CPU-offload options, output details, and multi-GPU sequence-parallel inference.
2. 2K refinement (super-resolve a generated or external video, audio unchanged):
cd ../video_refiner
pip install -r requirements.txt
INPUT=/path/to/video.mp4 bash run_inference.sh
See video_refiner/README.md for the full list of knobs (KV cache, window attention, speed/quality trade-offs).
:books: Citation
If you find DreamX-Creator useful in your research, please consider citing our technical report:
@misc{zhu2026dreamxcreatordemocratizingnativeaudiovideo,
title={DreamX-Creator: Democratizing Native Audio-Video Generation at 2K Resolution},
author={Jiashu Zhu and Yanhao Zheng and Ruitian Tian and Rujing Dang and Shen Zhang and Bingze Song and Jiachen Lei and Ruimin Lin and Jiahong Wu and Xiangxiang Chu},
year={2026},
eprint={2608.31106},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2608.31106},
}
:scroll: License
This project is licensed under the Apache License 2.0. See LICENSE for details.
:sparkles: Acknowledgement
We would like to thank the Wan Team, the OpenMOSS Team, and the VideoX-Fun Team for their outstanding open-source work on Wan, MOVA, and VideoX-Fun, respectively.