ComfyUI-Falcon-Perception

ComfyUI Falcon Perception

An elegant, robust, and highly-optimized ComfyUI custom node package for the tiiuae/Falcon-Perception vision-language model.

Falcon Perception is a 0.6B parameter early-fusion model designed for open-vocabulary grounding and instance segmentation. Given an image and a natural language query (e.g., "a red car" or "girl"), it returns precise pixel-accurate masks.

This ComfyUI port has been completely refactored from the official HuggingFace implementation to provide a seamless, bug-free experience on Windows and all PyTorch versions (>=2.5.0).

---

🌟 Key Features & Improvements

• No HuggingFace Cache Hell: Bypasses trust_remote_code=True completely. The architecture is natively bundled in falcon_core/, ensuring you never run into remote code injection bugs or version mismatches.
• Single File Weight Loading: No need to download a messy repository of .py and .json files. You only need one .safetensors weight file.
• OOM & VRAM Optimized: Fixed severe OOM issues present in the official upsampler mask generation ($O(N^2)$ footprint). It gracefully falls back to PyTorch's native scaled_dot_product_attention (FlashAttention2) for massive memory savings.
• Windows & Triton Friendly: Automatically detects your environment. If Triton is missing or PyTorch compilation fails (common on Windows), it gracefully falls back to Eager Mode without crashing.
• PyTorch 2.7.0+ Compatibility: Includes custom patches for the newly introduced flex_attention API changes and the removal of AuxRequest/BlockMask.

🛠️ Installation

1. Clone or Download this repository into your ComfyUI's custom_nodes/ directory:

   cd ComfyUI/custom_nodes/
   git clone https://github.com/your-username/ComfyUI-Falcon-Perception.git
   

2. Install Dependencies: Navigate into the folder and install the requirements using your ComfyUI's Python environment. (Note: This model strictly requires PyTorch 2.5.0 or newer for the flex_attention module)

   cd ComfyUI-Falcon-Perception
   # If using ComfyUI portable:
   ..\..\..\python_embeded\python.exe -m pip install -r requirements.txt
   # Or standard venv:
   pip install -r requirements.txt
   

📦 Model Weights Setup

Unlike the official HuggingFace pipeline, this node only cares about the safetensors weight file.

1. Download the model.safetensors file from the official HuggingFace Repo.
2. Create a folder named falcon_perception inside your ComfyUI models/ directory.
3. Place the downloaded .safetensors file inside.

Your directory structure should look like this:

ComfyUI/
└── models/
    └── falcon_perception/
        └── model.safetensors

🖥️ Usage in ComfyUI

After restarting ComfyUI, you will find two new nodes under the FalconPerception category:

1. Load Falcon Perception Model

• model_name_or_path: The path to your .safetensors file relative to the models/ folder. (Default: falcon_perception/model.safetensors)
• device: Select auto, cuda, or cpu.

2. Falcon Perception Generate

Connect your image and the loaded model here.

• query: Natural language text describing what you want to segment (e.g., "cat", "car window", "girl").
• max_new_tokens: Decoding steps limit.
• min_dimension / max_dimension: Image resizing bounds before passing to the model.
• compile_model: Enable torch.compile (Not recommended for Windows users without Triton).

Outputs:

• combined_mask: A single flat mask containing all found instances merged together.
• individual_masks: A batch of masks, each isolating a single found instance.
• info: Textual data detailing the coordinates xy and size hw of every detected instance.

---

⚠️ Requirements

• PyTorch >= 2.5.0: The architecture uses the new torch.nn.attention.flex_attention API. If you encounter an ImportError, you MUST upgrade your PyTorch environment.
• einops, pycocotools, transformers, pillow (Included in requirements.txt).

📜 Acknowledgements

Based on the original tiiuae/Falcon-Perception model by the Technology Innovation Institute. Custom refactoring, PyTorch API patching, and VRAM optimizations by the ComfyUI community.