Video to Layout BBOX Node

VACE's "layout" control is the sneaky-good one. While everyone wrestles with pose maps and depth, layout lets you steer a Wan video by drawing a box: track it around the subject and the output follows that region, reliably, frame after frame. This node - "Video to Layout BBOX" - is the ComfyUI-native way to produce that control signal. You hand it a clip (as a batch of frames) plus one initial bounding box, and it tracks that box through every frame with SAM2, then renders the layout visualization VACE actually eats: white frames with colored outlines.

Fair warning up front: this is a one-person wrapper around Alibaba's official VACE-Annotators, and the page's zero impressions reflect that. It's real, it works, but you'll babysit model paths before it runs. Not a drag-and-drop.

How it works

The pipeline is short and easy to follow in the source. Your frame batch gets written to a temp folder as JPEGs (that's what the DEBUG: temp jpeg folder = ... lines in your console are about - harmless). Then the LayoutTrackAnnotator takes over with your bbox in bboxtrack mode: SAM2 seeds on the box in frame one and tracks it across the whole clip, producing a per-frame object mask.

For each mask it does two things: optionally augments it, then draws it. A blank white canvas is created per frame and the mask's contour is stroked in a color from the pack's ram_tag_color_list (or plain black if no label matches). Stack the frames back up and you've got the layout control video.

The inputs that matter

• images (IMAGE, required) - your video as a frame batch. The output of a Load Video / VHS node is exactly this.
• bboxes (BBOX, optional) - the initial box in the first frame. Optional in the schema, essential in practice: no box, nothing for SAM2 to seed on. Wire it from any node that emits ComfyUI's native BBOX type, like a detector from the Impact Pack.
• mask_aug (enum, default bbox) - how each tracked mask becomes a drawing. bbox gives the tight rectangle (hence the node's name), original the raw mask outline, hull the convex hull, and the _expand variants inflate them by mask_aug_ratio first.
• mask_aug_ratio (float 0–1, default 0.1) - the expansion amount, 0.1 being 10%.
• mode - one choice, bboxtrack. The source comments show masktrack/label/caption were planned; they never got exposed.

Installation

cd ComfyUI/custom_nodes
git clone https://github.com/SamTyurenkov/comfyui-vace-preprocessors
pip install -r requirements.txt   # easydict, groundingdino-py, sam2, decord

Or search "comfyui_vace_preprocessors" in ComfyUI Manager. Either way, restart ComfyUI after. The Python deps aren't heavy, but note this pack registers only two nodes - this one and its sibling CombineLayoutTracksNode.

Models - the actual setup cost

Here's the gotcha that bites everyone. bboxtrack tracking only ever touches SAM2's weights, yet the pack builds the underlying annotator in "all" mode, which eagerly loads all three models at init: U2-Net salient, GroundingDINO, and SAM2. The README's model list is not optional decoration - you need all of it, into ComfyUI/models/:

• salient/u2net.pt
• sam2/ - the configs plus sam2.1_hiera_large.pt
• grounding-dino/ - tokenizer, config, and weights, renaming GroundingDINO_SwinT_OGC.py to GroundingDINO_SwinT_OGC.cfg.py

All three live in the ali-vilab/VACE-Annotators repo. That's a few GB and a couple of folder-name mistakes waiting to happen.

Troubleshooting

• Missing-model tracebacks in annotators/ - check the exact folder names in ComfyUI/models/, and that .cfg.py rename. A missing or mistyped filename is the #1 failure.
• CUDA only. The annotator is hardcoded to cuda:{RANK}; there's no CPU path, so a CPU-only or headless box errors at first run.
• VRAM. Three models resident means a few GB overhead on top of your Wan workflow. Not brutal, but real.
• It's I/O-bound. Frames round-trip through temp as lossy JPEG, so long clips spend time on disk, and the re-encode is technically lossy (visually irrelevant for box outlines).

The output is an IMAGE with the same frame count as your input - plug it straight into the layout control input of a VACE workflow and you're drawing with boxes.