FitDiT Load Model

Every FitDiT workflow starts at FitDiT Load Model, and it's the node that makes or breaks the install. Not because it's hard to use - it's four dropdowns - but because it's the first thing that runs, and it will happily tell you, in detail, everything you downloaded wrong. Think of it as the boot sequence for the whole pack.

What it actually loads

The entire FitDiT stack in one shot: two SD3-based diffusion transformers (one encodes the garment, one does the actual try-on), a pose guider, two CLIP vision encoders (ViT-L/14 and the big ViT-bigG/14), plus the DWPose and human-parsing ONNX preprocessors that the mask node uses later. All of that gets packed into a single FITDIT_MODEL output that both FitDiT Generate Mask and FitDiT Virtual Try-on consume. You need exactly one of these in your graph - feed its output to everything.

The inputs

• device - cuda (default), cuda:0 / cuda:1 if you're splitting across GPUs, cpu if you enjoy waiting.
• dtype - bf16 (default) or fp16. Per the upstream authors, bf16 is the "fastest inference and most memory" setting; fp16 trades a little speed for a little less VRAM.
• offload - model CPU offload: moderate speed, moderate VRAM.
• aggressive_offload - sequential CPU offload: slowest, least VRAM.

Try them in that order if you're tight on memory: plain offload first, aggressive only if it still OOMs. People do run this pack on 8GB cards with the right combination.

Model loading gotcha

The node uses local_files_only=True, so it will never download anything for you. It reads from ComfyUI/models/FitDiT_models, and if a subdirectory is missing it raises an error that literally prints the directory tree it expects. That error message is your checklist, not a dead end.

Install and models - the heavy part

cd ComfyUI/custom_nodes
git clone https://github.com/ihmily/ComfyUI-FitDiTx
pip install -r requirements.txt

Then the weights, and this is where the disk space goes:

cd ComfyUI/models
hf download BoyuanJiang/FitDiT --local-dir FitDiT_models

cd ComfyUI/models/clip
hf download openai/clip-vit-large-patch14 --local-dir clip-vit-large-patch14
hf download laion/CLIP-ViT-bigG-14-laion2B-39B-b160k --local-dir CLIP-ViT-bigG-14-laion2B-39B-b160k

Three things worth knowing before you run that:

• The FitDiT weights are gated on Hugging Face. Request access to BoyuanJiang/FitDiT first, or hf download fails with a 401. The two CLIP encoders are public.
• Plan for double-digit gigabytes. Two SD3-scale transformers plus CLIP-bigG plus the ONNX preprocessors add up fast.
• The node looks for the CLIP encoders as image_encoder / image_encoder_bigG inside the model dir first, then falls back to the models/clip folders - so the README's layout works as written.

Dependencies

requirements.txt pulls in torch/torchvision, diffusers, transformers, onnxruntime, opencv-python, scikit-image, accelerate and friends - nothing is pinned. The upstream FitDiT was tested against diffusers 0.31.0 and transformers 4.39.3. If you hit an import error inside the pack's src/ after installing, that's the usual culprit: align your diffusers near that era, or upgrade everything and see if the pack caught up.

One legal aside

FitDiT is CC BY-NC-SA 4.0 - non-commercial only. Fine for personal use and evaluation; if you're building a product on it, you're expected to go through Tencent Cloud's commercial path. Worth knowing before you invest a weekend.