FitDiT Model Loader

Every ComfyUI pack has one node that's actually just plumbing, and in ComfyUI-FitDit that's the FitDiT Model Loader. It doesn't try anything on and it doesn't make a picture. It loads the entire FitDiT try-on stack into memory and hands the result to the other two nodes as a single FITDIT_MODEL blob. Boring, yes - but it's also the only node where a wrong setting costs you twenty minutes of loading instead of ten seconds of rerun, so it's worth understanding before you touch it.

What "loading the model" actually means

FitDiT is a diffusion-transformer try-on model, not a single checkpoint you can stuff into one file. The loader assembles four moving parts from the directory you point it at:

• transformer_garm and transformer_vton - two SD3-style transformer variants, one that understands the garment and one that does the actual try-on, loaded from subfolders of your model directory.
• pose_guider - the small adapter that keeps your model's pose locked, loaded from pose_guider/diffusion_pytorch_model.bin.
• Two CLIP vision encoders - openai/clip-vit-large-patch14 and laion/CLIP-ViT-bigG-14-laion2B-39B-b160k - which download from Hugging Face on first run, not from your model folder. The README warns you to be patient; together those are several gigabytes you only fetch once.

The loader also builds the DWPose and human-parsing preprocessors that the Mask Generator node needs, so one loader call warms up the whole pack.

The inputs

Only two, and you'll realistically touch one:

• model_dir - the path to the folder where you downloaded the FitDiT weights (the README's example is models/FitDiT). This must point at the directory containing transformer_garm, transformer_vton and pose_guider; get it wrong and the loader errors immediately.
• use_fp16 - on by default, and it loads weights in float16. Flip it off and it uses bfloat16 instead. fp16 is the sensible default; bfloat16 is the fallback on cards where fp16 misbehaves. This isn't a quality knob, it's a precision-mode switch.

The output

One FITDIT_MODEL connection that you split into both the FitDiT Mask Generator and the FitDiT Try-On node. Both consume the same blob; you don't load the model twice.

One genuinely useful behavior hides in the source: the loader caches by directory. Call it again with the same model_dir and it returns instantly without reloading. Change the path and it tears everything down and rebuilds - a long wait and a VRAM spike. Pick a directory and stick with it for the session.

Install, the real way

The weights are gated. Head to the FitDiT Hugging Face repo, request access, and wait for approval before you do anything else. Then download the model folder somewhere ComfyUI can read it, clone the pack:

cd ComfyUI/custom_nodes
git clone https://github.com/lgldlk/ComfyUI-FitDit

Restart ComfyUI. ComfyUI Manager works too - it's listed under "FitDiT". The one install gotcha is that the repo ships no requirements.txt; it relies on ComfyUI's bundled torch, diffusers and transformers, plus opencv and onnxruntime that may not be there. If the loader (or the Mask node) fails at import, that's the first thing to check.

Two gotchas worth naming

The FitDiT weights are CC BY-NC-SA 4.0 - non-commercial. Fine for tinkering and research, a legal landmine if you planned to run a paid try-on service on top of it. And don't be surprised the first load takes minutes and a lot of VRAM; that's the price of a purpose-built try-on DiT rather than a generic checkpoint.