ToonCrafterWithSketch

Same trick as its sibling, one extra superpower. ToonCrafterWithSketch runs the same two-keyframe cartoon interpolation as ToonCrafterNode - start frame, end frame, sixteen generated frames in between - but with a ControlNet bolted on. You also hand it a sequence of sketch or storyboard frames, and the in-betweens get pulled toward your roughs instead of whatever the diffusion prior alone decides.

That's the half of the original ToonCrafter pitch most people never touched. The launch demos wowed everyone with straight interpolation; the "reference-based sketch" mode was the quieter second act. This node is that second act inside ComfyUI.

Why it exists: the base node generates good motion between two keyframes, but you don't control composition. If you're storyboarding a shot - or you've got a layout the in-betweens must respect - a rough sketch per frame is a cheap way to pin the composition down. It's the ControlNet idea from the KB's concepts doc applied to the temporal axis: the sketch is the hint, and control_scale is how hard it pulls.

How it works

On top of the ToonCrafter UNet, the node loads a separate ControlNet from checkpoints/sketch_encoder.ckpt. The config (cldm_v21.yaml) expects 1-channel hint input - which is another way of saying it wants grayscale line art, not full-color renders. Your frame_guides IMAGE batch gets concatenated along the temporal dimension and passed in as the control condition, scaled by control_scale. Clean and classic.

Inputs that matter

Everything the base node takes is here - image, image2, ckpt_name, vram_opt_strategy, prompt, seed, eta, cfg_scale, steps, frame_count, fps - plus two:

• frame_guides (IMAGE) - a batch of sketch frames. One per output frame is the natural shape, since they're concatenated into the temporal sequence. Grayscale/line-art hints work best with the 1-channel encoder.
• control_scale (default 0.6, range 0–1) - how hard the sketches pull. 0 ignores them; 1 is maximum. When in doubt, start at the default and nudge.

Note the schema lists frame_guides as required, so the graph wants a wire on it. If you don't have sketches and don't want to build them, use the base ToonCrafterNode instead - this node hard-requires the sketch encoder weights and the guide input.

Output is the same as the base node: one IMAGE batch of frames, ready for VHS VideoCombine to become an mp4.

Setup

Shared with the whole pack: ComfyUI Manager (search "ComfyUI-ToonCrafter"), or clone and install:

cd ComfyUI/custom_nodes
git clone https://github.com/AIGODLIKE/ComfyUI-ToonCrafter
cd ComfyUI-ToonCrafter
python_embeded\python.exe -m pip install -r requirements.txt

Then two model files. The main ToonCrafter weights go in ToonCrafter/checkpoints/tooncrafter_512_interp_v1/ (the fp16 tooncrafter_512_interp-fp16.safetensors from Kijai's repo is the sensible pick), and sketch_encoder.ckpt needs to sit in ComfyUI-ToonCrafter/ToonCrafter/checkpoints/. One quirk that trips people up: the node's checkpoint dropdown deliberately excludes sketch_encoder.ckpt from the list - it's auto-loaded by this node, so don't hunt for it in the dropdown.

Troubleshooting

The pack-level gotchas all apply here too. The xformers/torch trap is the big one - Kijai's advice is pip install xformers --no-deps so pip doesn't silently rebuild torch without GPU support. VRAM is the other: ToonCrafter's dual-reference encoder is heavy, so keep vram_opt_strategy=low on smaller cards and cut steps for first passes. And if the sketches are making the output weird or broken, that's the ControlNet oversteering - drop control_scale toward 0.3 rather than abandoning the guides entirely.

Honest verdict: it has zero search impressions, and there's a reason - the base node does the headline trick, and feeding a frame_guides batch is fiddlier than most people want for a tool they'll run once. But if you're doing actual storyboard-to-animation cleanup, the sketch-steered version is where ToonCrafter turns from "impressive demo" into "something you'd put in a production pipeline."