ComfyUI-PromptMaker-PNGTuber

ComfyUI custom nodes for turning a character video into a video-mouth PNGTuber asset bundle.

The package name keeps the PromptMaker integration target, but the pipeline is not PromptMaker-only. It writes a general pngtuber.videoMouthBundle.v1 manifest, a frame-by-frame mouth track, flat compatibility sprites, and an angle-aware mouth sprite atlas.

The node is local-only. It does not call OpenRouter or any remote image API.

The quality path can also start from a single character image. It uses the installed local Qwen/eigen image-edit nodes to generate clean mouth shapes from the full source image, optionally renders a THA3 idle loop, and writes a PromptMaker-compatible PNGTuber bundle with QA preview contacts.

Quick Start

1. Install the node into ComfyUI/custom_nodes.
2. Install requirements.txt in the same Python environment used by ComfyUI.
3. Restart ComfyUI.
4. Load examples/workflows/pngtuber_video_upload_bundle.json.
5. Press upload video on the single node, choose a video, and queue the workflow.

The default workflow writes a complete bundle under ComfyUI output: loop_mouthless_h264.mp4, mouth_track.json, mouth_sprite_atlas.json, bundle_manifest.json, flat mouth sprites, and angle-aware mouth sprites.

Nodes

• Class type: PNGTuberVideoMouthBuilder
• Class type: PNGTuberVideoUploadToMouthBundle
• Compatibility alias: PromptMakerPNGTuberVideoMouth
• Class type: PNGTuberGeneratedMouthSpriteApplier
• Class type: PNGTuberQualityAssetBuilder
• Class type: PNGTuberVideoEyeBuilder
• Class type: PNGTuberGeneratedEyeSpriteApplier
• Class type: PNGTuberImageEyeBuilder
• Category: PNGTuber/Video Mouth
• Category: PNGTuber/Quality Builder
• Category: PNGTuber/Video Eyes
• Category: PNGTuber/Image Eyes

PNGTuberVideoUploadToMouthBundle is the preferred one-node workflow. It shows a ComfyUI input-folder video selector plus an upload video button and can also accept drag-and-drop video files. Run it by selecting/uploading one video and queueing the workflow.

PNGTuberVideoMouthBuilder is the advanced compatibility node. It exposes the same extraction pipeline but keeps the lower-level video_path input and tuning controls for scripts or older workflows.

PNGTuberGeneratedMouthSpriteApplier takes locally generated Qwen Image Edit outputs and replaces or fills closed/open/half/e/u sprites inside an existing bundle. Use it when the builder marks a closed-mouth or weak-articulation video with articulation.requiresModelGeneration: true.

PNGTuberQualityAssetBuilder is the high-level production node. It accepts an image or a video path:

• Image input: local Qwen/eigen edits the full source image five times (closed/half/open/e/u), erases the source mouth, optionally renders a THA3 mouthless idle loop, and writes runtime mouth sprites plus _qa_overlay_contact.png.
• Video input: Qwen/eigen creates identity-stable mouth sprites from the first frame while the video supplies the per-frame mouth track.

The node refuses obviously invalid mouth localization and always writes a QA contact sheet that overlays the generated sprites back onto the source face.

Outputs

For each input video, the node writes:

• loop_mouthless_h264.mp4
• mouth_track.json
• mouth_sprite_atlas.json
• bundle_manifest.json
• summary.json
• mouth/closed.png
• mouth/half.png
• mouth/open.png
• mouth/e.png
• mouth/u.png
• mouth/angles/angle_m45/{closed,half,open,e,u}.png
• mouth/angles/angle_m30/{closed,half,open,e,u}.png
• mouth/angles/angle_m15/{closed,half,open,e,u}.png
• mouth/angles/angle_p00/{closed,half,open,e,u}.png
• mouth/angles/angle_p15/{closed,half,open,e,u}.png
• mouth/angles/angle_p30/{closed,half,open,e,u}.png
• mouth/angles/angle_p45/{closed,half,open,e,u}.png

Schemas

• bundle_manifest.json: pngtuber.videoMouthBundle.v1
• mouth_track.json: pngtuber.mouthTrack.v1
• mouth_sprite_atlas.json: pngtuber.mouthSpriteAtlas.v1

mouth_track.json includes:

• frames[].quad: four-point mouth placement quad in source video coordinates
• frames[].mouthOpen: normalized mouth openness estimate
• frames[].mouthAngleDegrees: estimated mouth angle
• frames[].spriteSet: nearest atlas set such as angle_m15 or angle_p00
• frames[].qualityScore: mouth signal quality used for sprite candidate selection
• frames[].occlusionScore: high values indicate likely occlusion or weak mouth signal

If the source video does not contain enough mouth-open variation, the bundle is marked with articulation.requiresModelGeneration: true and the node writes:

• mouth_generation_inputs/reference_face_mouth_crop.png
• mouth_generation_inputs/mouth_edit_mask.png
• mouth_generation_inputs/mouth_generation_plan.json

That plan is for a local image-edit model pass, not a remote API call. It is designed for installed ComfyUI Qwen Image Edit assets such as qwen_image_edit_2511_bf16_각도.safetensors plus the local Qwen text encoder and VAE. In this case, extracted mouth sprites should be treated as provisional until PNGTuberGeneratedMouthSpriteApplier replaces them with model-generated sprites.

The applier accepts optional closed_image, open_image, half_image, e_image, and u_image ComfyUI IMAGE inputs. It extracts the mouth component from those edited face crops, writes transparent mouth sprites, rotates them across the atlas angle bins, and updates bundle_manifest.json, mouth_sprite_atlas.json, and summary.json with modelGeneration metadata.

The flat five mouth sprites keep compatibility with players that only support a single mouth set. The angle atlas is for smoother runtime animation when a player supports mouth_sprite_atlas.json and frames[].spriteSet.

Install

Clone into ComfyUI's custom_nodes directory:

cd /path/to/ComfyUI/custom_nodes
git clone https://github.com/hobi2k/ComfyUI-PromptMaker-PNGTuber.git

Install dependencies with the Python environment used by ComfyUI:

cd /path/to/ComfyUI
python -m pip install -r custom_nodes/ComfyUI-PromptMaker-PNGTuber/requirements.txt

Restart ComfyUI after installing the node or dependencies.

Detection Pipeline

Default mode is anime_first.

1. Detect anime faces with models/lbpcascade_animeface.xml.
2. Refine the mouth region with local OpenCV edge/dark/lip-color analysis.
3. Fall back to MediaPipe FaceMesh if anime detection fails.
4. Fall back to a local ComfyUI YOLO face model when available at models/ultralytics/bbox/face_yolov8m.pt.
5. Interpolate missing frames and smooth the mouth track.
6. Inpaint the mouth region from every video frame using OpenCV TELEA.
7. Score each frame for mouth signal quality and likely occlusion.
8. Measure mouth-open variation. Closed-only or near-closed-only videos are flagged for local model generation instead of trusting forced extraction.
9. Extract mouth-only transparent sprites from the best non-occluded candidates.
10. Build angle-aware mouth sprite sets. The default range is -45 through +45 degrees in 15 degree steps, configurable by angle_range_degrees and angle_step_degrees. Missing bins are generated from the nearest real sprite set and are marked with generated: true in mouth_sprite_atlas.json.
11. Write atlas-level QA metadata under quality, including real/generated angle sets, frame quality, occlusion ratio, and warnings.

For closed-mouth videos, run a local Qwen Image Edit workflow with mouth_generation_inputs/reference_face_mouth_crop.png as the reference image, then connect the generated shape images to PNGTuberGeneratedMouthSpriteApplier. The applier can replace all five shapes, including closed, so a poor forced extraction is not kept just because the video had no usable articulation.

Static Image Eye Builder

PNGTuberImageEyeBuilder is the static transparent PNG blink pipeline. It mirrors PNGTuberImageMouthBuilder for eyes:

1. Load a transparent character PNG.
2. Detect the visible eye band automatically with multi-scale anime-face cascade plus detail fallback.
3. Optionally accept manual_eye_bbox as x0,y0,x1,y1 when automatic detection needs art-direction correction.
4. Erase the original eyes into clean facial skin while preserving the input alpha channel.
5. Generate open, half, and closed eye states with the local eigen/Qwen Image Edit path.
6. Write a complete pngtuber.imageEyeBundle.v1 runtime bundle.

Outputs:

• eyeless.png
• blink_loop.webm
• blink_loop_h264.mp4
• blink_frames/frame_0001.png ...
• eye_track.json
• eye_sprite_atlas.json
• blink_sequence.json
• bundle_manifest.json
• eyes/open.png
• eyes/half.png
• eyes/closed.png

Use blink_loop.webm as the transparent blink animation. Use blink_loop_h264.mp4 as the video source for the PNGTuber mouth pipeline when you want mouth tracking/sprite extraction based on the blink-ready character. The still bundle files remain available for runtimes that swap eye sprites directly.

Workflow Examples

UI-loadable workflows are included at:

examples/workflows/pngtuber_video_upload_bundle.json
examples/workflows/pngtuber_quality_asset_builder.json
examples/workflows/pngtuber_generated_mouth_applier.json
examples/workflows/pngtuber_qwen_mouth_generation.json
examples/workflows/pngtuber_image_eye_builder_api.json

These files use ComfyUI's canvas/LiteGraph format and show nodes when loaded from the ComfyUI workflow menu or by drag-and-drop.

For a normal video-to-PNGTuber bundle run, open pngtuber_video_upload_bundle.json, press upload video on the single PNGTuber Video Upload to Mouth Bundle node, select a local video, and queue the prompt.

For the quality image-or-video path, open pngtuber_quality_asset_builder.json. Use it when quality matters more than speed, especially for static anime character images that need local Qwen/eigen mouth generation plus a THA3 mouthless loop.

For generated-mouth post-processing, open pngtuber_generated_mouth_applier.json. It is intentionally independent: provide an existing bundle_manifest.json path and optional generated mouth images, then queue only that applier workflow.

For closed-mouth videos, open pngtuber_qwen_mouth_generation.json after the first pass creates mouth_generation_inputs/*. It uses local ComfyUI Qwen Image Edit nodes when they are installed.

API /prompt examples are included at:

examples/workflows/pngtuber_video_upload_bundle_api.json
examples/workflows/pngtuber_quality_asset_builder_api.json
examples/workflows/pngtuber_generated_mouth_applier_api.json
examples/workflows/pngtuber_qwen_mouth_generation_api.json

pngtuber_qwen_mouth_generation_api.json is the local-only closed-mouth fallback pass: it loads reference_face_mouth_crop.png, generates closed/open/half/e/u with Qwen Image Edit, then feeds those decoded images directly into PNGTuberGeneratedMouthSpriteApplier.

Restart ComfyUI after updating the custom node so the applier class is visible in the node registry.

If a workflow shows red LoadImage nodes, upload or replace the placeholder input file names with the actual generated mouth images in ComfyUI's input folder. Red missing-input nodes still mean the workflow loaded; a blank canvas means an API JSON was loaded into the UI by mistake.

Use the upload button or drag a video onto the node. Absolute paths are still supported through the advanced node or the advanced_video_path override, but they are no longer the normal UI path.

Quality Builder

PNGTuberQualityAssetBuilder writes pngtuber.qualityAssetBundle.v1.

Important output files:

• bundle_manifest.json
• mouth_track.json
• mouth/closed.png
• mouth/half.png
• mouth/open.png
• mouth/e.png
• mouth/u.png
• mouth_raw/*.png for review-only extracted mouth components
• mouth_cover/*.png for cover-only review patches
• _contact.png for sprite-only review
• _qa_overlay_contact.png for the real quality gate: generated mouths composited back onto the mouthless face
• loop_mouthless.webm and loop_mouthless_h264.mp4 when THA3 rendering is enabled for image sources

Quality rules in the node:

• Qwen/eigen receives the full source image, not a detached tiny mouth crop.
• Only the mouth area is masked for editing.
• Mouth localization must pass size sanity checks before a bundle is accepted.
• Runtime sprites are tight full-source mouth patches. Their mouth_track.json quad points to the exact patch rectangle, so mouth size and skin cover stay aligned with the character face instead of drifting from a centered icon.
• The returned preview_contact is the overlay QA contact, not just raw sprites.

PromptMaker Compatibility

PromptMaker pngtuber_mode=video_mouth accepts both the flat compatibility files and the angle-aware atlas bundle:

• loop_mouthless_h264.mp4
• mouth_track.json
• mouth_sprite_atlas.json
• mouth/closed.png
• mouth/half.png
• mouth/open.png
• optional mouth/e.png
• optional mouth/u.png
• optional mouth/angles/*/{closed,half,open,e,u}.png

PromptMaker uses frames[].spriteSet first and falls back to frames[].mouthAngleDegrees when selecting an angle sprite.

License Notes

lbpcascade_animeface.xml is included from nagadomi/lbpcascade_animeface under the MIT license. Its license header is preserved inside the XML file.