ComfyUI Node
FFMPEG Agent
AI-powered video editor: describe edits in natural language and the agent generates and runs the ffmpeg pipeline automatically.
FFMPEG Agent
- images_a
- image_a
- audio_a
- images
- audio
- video_path
- command_log
- analysis
- mask_overlay_path
- mask_points
◄prompt►
◄video_path►
◄llm_modelnone►
◄no_llm_modemanual►
◄quality_presetstandard►
◄seed0►
◄video_a—►
◄image_path_a—►
◄text_a—►
◄pipeline_json—►
◄mask_points—►
◄crop_data—►
◄save_outputfalse►
◄output_path►
◄ollama_urlhttp://localhost:11434►
◄api_key►
◄custom_model►
◄use_visionfalse►
◄verify_outputfalse►
◄advanced_optionsfalse►
◄preview_modefalse►
◄subtitle_path►
◄crf-1►
◄encoding_presetauto►
◄whisper_devicecpu►
◄whisper_modellarge-v3►
◄sam3_max_objects2►
◄sam3_det_threshold0.70►
◄mask_output_typeblack_white►
◄use_flux_kleinfalse►
◄flux_smoothingnone►
◄use_minimax_removerfalse►
◄marigold_output_typedepth►
◄video_depth_encodervits►
◄video_depth_colormapgray►
◄upscale_modelrealesrgan_x4plus►
◄upscale_scale4►
◄rembg_modelbria-rmbg►
◄rembg_backgroundtransparent►
◄mmaudio_modereplace►
◄batch_modefalse►
◄video_folder►
◄file_pattern*.mp4►
◄max_concurrent4►
◄track_tokenstrue►
◄log_usagefalse►
◄allow_model_downloadstrue►
CategoryFFMPEGA
Inputs (50)
| Name | Type | Default | Description |
|---|---|---|---|
| prompt | STRING | Natural language instruction describing the desired edit. Examples: 'Add a cinematic letterbox', 'Speed up 2x', 'Apply a vintage VHS look'. | |
| video_path | STRING | Absolute path to the source video file. Used as the ffmpeg input unless images are connected. | |
| llm_model | COMBO | none | AI model for interpreting your prompt. CLI models (gemini-cli, claude-cli, etc.) use locally installed CLI tools — no API key needed. Ollama models run locally via the Ollama server. Cloud API models (GPT, Claude, Gemini, Qwen) require an api_key. Select 'custom' to type any model name manually. Select 'none' to skip the LLM entirely and use no_llm_mode instead (manual pipeline, SAM3, Whisper, or MMAudio). |
| no_llm_mode | COMBO | manual | What to do when llm_model is 'none'. 'manual' runs the Effects Builder pipeline directly (no AI). 'sam3_masking' uses the prompt as a SAM3 text target. 'transcribe' runs Whisper speech-to-text and burns SRT subtitles. 'karaoke_subtitles' runs Whisper and burns word-by-word karaoke subtitles. 'generate_audio' uses MMAudio to synthesize audio from video/prompt. 'lip_sync' uses MuseTalk to sync lip movements to connected audio_a. 'animate_portrait' uses LivePortrait to animate a face — connect driving video to video_a. 'marigold' runs Marigold dense vision analysis (depth/normals/intrinsics) — choose output via marigold_output_type. 'video_depth' runs Video Depth Anything for temporally-consistent depth — choose encoder via video_depth_encoder. 'flux_klein' runs FLUX Klein editing directly — prompt is the edit instruction, works on images and videos (full-frame, no mask needed). 'rembg' removes the video background using AI segmentation — choose model via rembg_model and background via rembg_background. |
| quality_preset | COMBO | standard | Output quality level. 'draft' is fast/low quality, 'standard' is balanced, 'high' is slow/best quality, 'lossless' preserves full quality. |
| seed | INT | 00–18446744073709550000 | Change this value to force re-execution with the same prompt. Use the randomize control to auto-increment between runs. |
| images_aopt | IMAGE | Video input as image frames (e.g. from Load Video Upload). Connect additional video inputs and more slots appear automatically (images_b, images_c, ...). Used for concat, split screen, and multi-video workflows. | |
| image_aopt | IMAGE | Extra image/video input. Connect additional inputs and more slots appear automatically (image_b, image_c, ...). Used for multi-input skills like grid, slideshow, overlay, concat, and split screen. | |
| audio_aopt | AUDIO | Audio input. Connect additional audio and more slots appear automatically (audio_b, audio_c, ...). Used for muxing audio into video, lip sync, or for multi-audio skills like concat. | |
| video_aopt | STRING | File path to an extra video for concat, split screen, grid, or xfade. Uses zero extra memory vs tensor inputs. Connect and more slots appear (video_b, video_c, ...). Connect a primitive STRING node or any node that outputs a file path. | |
| image_path_aopt | STRING | File path to an image for overlay, grid, slideshow, or multi-image skills. Uses zero memory vs IMAGE tensor. Connect and more slots appear (image_path_b, image_path_c, ...). Use Load Image Path (FFMPEGA). | |
| text_aopt | STRING | Text input for subtitles, overlays, watermarks, or title cards. Connect an FFMPEGA Text node or any STRING source. More slots appear automatically (text_b, text_c, ...). | |
| pipeline_jsonopt | STRING | Connect the output from the FFMPEGA Effects Builder node here. The agent will inject the selected effects as hints into your prompt. | |
| mask_pointsopt | STRING | JSON-encoded point selection data from the Load Image/Video Path node's Point Selector. Guides SAM3 masking with click-to-select points instead of relying on text prompts alone. | |
| crop_dataopt | STRING | JSON-encoded crop rectangle from the Load Video Path or Frame Extract node's Crop Selector. Format: {"x":N, "y":N, "w":N, "h":N}. Crops the input video before processing. | |
| save_outputopt | BOOLEAN | false | When On, saves video and a workflow PNG to the output folder. Turn Off when a downstream Save node handles output to avoid double saves. Note: downstream nodes may re-encode with their own settings (format, quality, resolution), so the final saved file may differ from FFMPEGA's output. |
| output_pathopt | STRING | Custom output file or folder path. Leave empty to save to ComfyUI's default output directory. | |
| ollama_urlopt | STRING | http://localhost:11434 | URL of the Ollama server for local LLM inference. Default: http://localhost:11434. |
| api_keyopt | STRING | API key required when using cloud models (GPT, Claude, Gemini). Not needed for local Ollama models. | |
| custom_modelopt | STRING | When 'custom' is selected in llm_model, type the exact model name here. Use provider prefixes: gpt-* for OpenAI, claude-* for Anthropic, gemini-* for Google, anything else for Ollama. | |
| use_visionopt | BOOLEAN | false | When On, embeds video frames as images for vision-capable models (uses more tokens). When Off, uses numeric color analysis instead (cheaper, works with all models). |
| verify_outputopt | BOOLEAN | false | When On, the agent inspects the output video after rendering and auto-corrects if it doesn't match intent. Adds one extra LLM call (more tokens/time). Best for complex edits like overlays, color grading, or animations. |
| advanced_optionsopt | BOOLEAN | false | Show advanced options: preview, encoding, SAM3/Whisper tuning, FLUX smoothing, MMAudio mode, batch processing, and usage tracking. |
| preview_modeopt | BOOLEAN | false | When enabled, generates a quick low-res preview (480p, first 10 seconds) instead of a full render. |
| subtitle_pathopt | STRING | Direct path to a subtitle file (.srt or .ass). Alternative to using text_a with subtitle mode. | |
| crfopt | INT | -1-1–51 | Override CRF (Constant Rate Factor) for output quality. 0 = lossless, 23 = default, 51 = worst. Set to -1 to use quality_preset value. |
| encoding_presetopt | COMBO | auto | Override x264/x265 encoding speed preset. Slower = better compression. 'auto' uses the quality_preset value. |
| whisper_deviceopt | COMBO | cpu | Device for Whisper transcription model. 'gpu' is faster but uses ~3 GB VRAM (frees ComfyUI models first). 'cpu' is slower but avoids VRAM pressure — best for low-VRAM GPUs or intensive workflows. |
| whisper_modelopt | COMBO | large-v3 | Whisper model size for transcription. 'large-v3' is most accurate (~3 GB VRAM). Smaller models use less memory: medium (~1.5 GB), small (~1 GB), base (~150 MB), tiny (~75 MB). Models auto-download on first use. |
| sam3_max_objectsopt | INT | 21–20 | Maximum number of objects SAM3 will track per frame. Lower values reduce VRAM usage. Objects are ranked by detection confidence — lowest-confidence detections are dropped first. |
| sam3_det_thresholdopt | FLOAT | 0.700–1 | Minimum detection confidence for SAM3 to track a new object (0.0–1.0). Higher values = fewer objects tracked = less VRAM. Default 0.7 filters out low-confidence detections. |
| mask_output_typeopt | COMBO | black_white | Mask preview output format. 'black_white' outputs a raw B&W mask video (white = detected object) for use in external compositing. 'colored_overlay' composites colored SAM3-style regions + contours onto the video. |
| use_flux_kleinopt | BOOLEAN | false | Enable FLUX Klein 4B for AI-powered object removal (auto_mask:effect=remove) and text-guided editing (auto_mask:effect=edit). OFF by default to avoid high VRAM usage (~8–15 GB). When OFF, removal falls back to MiniMax-Remover (if enabled) or LaMa (~200 MB) and editing uses lightweight FFmpeg filter approximations. |
| flux_smoothingopt | COMBO | none | Temporal smoothing for FLUX Klein effects (remove/edit). 'none' = no smoothing (fastest, least VRAM). 'gaussian' = Gaussian blur across time (reduces flicker, +700 MiB RAM). 'adaptive' = per-pixel deviation check, only smooths outlier frames (+700 MiB RAM). |
| use_minimax_removeropt | BOOLEAN | false | Enable MiniMax-Remover for high-quality video object removal (auto_mask:effect=remove). Uses a purpose-built DiT model (~2.5 GB, ~5–8 GB VRAM). Takes priority over FLUX Klein for removal when both are enabled. When OFF, removal falls back to FLUX Klein (if enabled) or LaMa (~200 MB). |
| marigold_output_typeopt | COMBO | depth | Marigold output type (used in 'marigold' no_llm_mode or agentic mode). 'depth' = monocular depth map. 'normals' = surface normals. 'appearance' = albedo + roughness + metallicity. 'lighting' = albedo + shading + residual. |
| video_depth_encoderopt | COMBO | vits | Video Depth Anything model size (used in 'video_depth' no_llm_mode or agentic mode). 'vits' = Small (~7 GB, fastest). 'vitb' = Base (~12 GB). 'vitl' = Large (~24 GB, best quality). |
| video_depth_colormapopt | COMBO | gray | Depth map colormap (used in 'video_depth' no_llm_mode). 'gray' = standard B&W depth (for ControlNet/compositing). Others are artistic colormaps for creative visualization. |
| upscale_modelopt | COMBO | realesrgan_x4plus | AI upscaler model (used in 'ai_upscale' no_llm_mode). 'realesrgan_x4plus' = fast general-purpose. 'realesrgan_x4_anime' = anime/cartoon. 'hat_x4' = SOTA quality (Real-HAT-GAN). 'dat_x4' = balanced (DAT-2). 'swinir_x4' = classical SR. |
| upscale_scaleopt | COMBO | 4 | AI upscale factor (used in 'ai_upscale' no_llm_mode). '4' = 4× resolution. '2' = 2× resolution. |
| rembg_modelopt | COMBO | bria-rmbg | Rembg model (used in 'rembg' no_llm_mode). 'bria-rmbg' = BRIA RMBG (SotA quality, recommended). 'birefnet-general' = BiRefNet high quality. 'birefnet-general-lite' = BiRefNet fast. 'isnet-general-use' = ISNet general. 'u2net' = U²-Net classic. 'silueta' = Silueta (fastest, lightweight). |
| rembg_backgroundopt | COMBO | transparent | Background replacement (used in 'rembg' no_llm_mode). 'transparent' = alpha channel (outputs VP9/WebM). 'green' = green screen for compositing. Other colors fill the background with a solid color. |
| mmaudio_modeopt | COMBO | replace | How to combine AI-generated audio with existing audio (used in 'generate_audio' no_llm_mode). 'replace' replaces existing audio entirely. 'mix' blends generated audio with the original track. |
| batch_modeopt | BOOLEAN | false | When enabled, processes all matching videos in video_folder with the same prompt. Uses a single LLM call and applies the pipeline to every file. |
| video_folderopt | STRING | Path to a folder containing videos to batch process. Only used when batch_mode is on. | |
| file_patternopt | COMBO | *.mp4 | File pattern to match videos in the folder. '*.mp4 *.mov *.avi' matches multiple formats. '*.*' matches all files. Only used when batch_mode is on. |
| max_concurrentopt | INT | 41–16 | Maximum number of videos to process simultaneously in batch mode. Higher values use more CPU/GPU. |
| track_tokensopt | BOOLEAN | true | When On, prints token usage summary (prompt tokens, completion tokens, LLM calls) to the console after each run. Useful for monitoring costs with paid APIs. |
| log_usageopt | BOOLEAN | false | When On, appends a JSON entry to usage_log.jsonl for each run. Useful for tracking cumulative token spend over time. |
| allow_model_downloadsopt | BOOLEAN | true | When On (default), AI models (SAM3, LaMa, Whisper) auto-download on first use. Turn Off to prevent any automatic downloads — runs requiring a missing model will fail with a clear message and a link to download manually. |
Outputs (7)
| Name | Type | Description |
|---|---|---|
| images | IMAGE | Image frames from the output video. Returns ALL frames automatically when connected to a downstream node (e.g. VHS Video Combine). Returns only a thumbnail when unconnected (zero-memory preview). |
| audio | AUDIO | Audio extracted from the output video (or passed through from audio_a) in ComfyUI AUDIO format. |
| video_path | STRING | Absolute path to the rendered output video file. |
| command_log | STRING | The ffmpeg command that was executed. |
| analysis | STRING | LLM interpretation, estimated changes, pipeline steps, and any warnings. |
| mask_overlay_path | STRING | Path to a mask overlay preview video with SAM3-style colored contours. Connect to Save Video (FFMPEGA) to view. |
| mask_points | STRING | Pass-through of upstream mask_points data for downstream nodes. |