Instance Diffusion Tracking Prompt

InstanceDiffusionTrackingPrompt ("Instance Diffusion Tracking Prompt") is the node that makes the pack worth installing. Everything else in ComfyUI-InstanceDiffusion is support staff; this is the engine. It's the thing that lets you point at two people in a video and say "that one, red hair, woman, walking left" and "that one, blue hair, man, walking right" - as separate instructions, not one global prompt that smears their attributes together.

That smearing is the whole problem this solves. A single prompt describing multiple subjects reliably bleeds hair color, clothing, and pose between them - it's the same pain behind regional prompting, and regional prompting's answer (masked conditioning per area) doesn't work when the subjects are moving frame to frame. InstanceDiffusion's answer: give each instance its own prompt and its own tracked position, and let the model fuse them at the right locations.

How it works

The Berkeley research (arXiv 2402.03290) adds three things to SD1.5: UniFusion (the positionnet), a multi-instance fuser sampler, and ScaleU. This node is where the first two meet your prompt. You write per-instance prompt lines in a specific syntax:

"0.person": "a woman with red hair",
"1.person": "a man with blue hair"

The number is the tracker ID, the word is the class, the quoted text is that instance's prompt. The node parses those, CLIP-encodes each one, and matches them to positions from the TRACKING input - which is a per-class, per-tracker map of where each instance is right now. Then it does something sneaky: it wraps the whole thing in an InstanceConditioning that masquerades as a Gligen object, because that's the path ComfyUI's sampler already knows how to set up. During sampling, the fusers (one gated self-attention block per UNet layer, loaded from the FUSERS input) inject each instance's embedding at its tracked location, and the positionnet converts locations into the attention maps. Use -1 as the tracker ID and a prompt applies to every tracked instance of that class - handy for "every person gets a hat".

Inputs that matter

• positive / negative (CONDITIONING) - your ordinary CLIPTextEncode outputs, carrying the global scene prompt.
• clip (CLIP) - from your checkpoint loader; used to encode the per-instance lines.
• tracking (TRACKING) - where the instance positions come from. This is the input everyone gets stuck on, and it does not come from this pack.
• positionnet / fusers - from DownloadInstanceDiffusionModels.
• positive_text / negative_text (STRING, multiline) - the per-instance prompt lines above. Same "id.class": "prompt" format in both.

Outputs are positive and negative CONDITIONING, wired into the KSampler that also receives the ScaleU-patched model from ApplyScaleUModelNode.

Where TRACKING comes from

The single most-asked question about this pack, and the answer is: a tracker node, not this pack. For video, KJNodes' YOLOTrackerNode tracks objects across frames with a YOLO model and emits exactly this TRACKING type - it's in the pack's own example workflows. logtd also ships ComfyUI-TrackingNodes for video tracking. The README lists KJNodes (for bounding boxes), TrackingNodes, AnimateDiff-Evolved, and Video Helper Suite as the companions you'll want.

Installing and gotchas

Manager → search "InstanceDiffusion", or git clone https://github.com/logtd/ComfyUI-InstanceDiffusion into custom_nodes/, restart. Only dependency is huggingface_hub, already in ComfyUI. Then grab the three model files via DownloadInstanceDiffusionModels.

Watch out for these:

• SD1.5 only. All the trained components are baked into SD1.5. No SDXL, no Flux. If that's a dealbreaker, this is a 2024 tool that's aging; if you want tracked multi-subject animation, it still has no real successor.
• Scribbles, points, segments, and masks are unsupported. The README is explicit: the underlying research supports them, but there are no ComfyUI nodes to feed them in yet.
• It's imperfect by design. Instances can still bleed or wander, and tracking quality sets the ceiling - if YOLO loses a subject, so does your prompt. Crank fusers_scale on the loader when subjects drift, and remember each per-instance line sits inside SD1.5's 77-token CLIP limit.