Falcon Perception Generate

Somewhere in your graphs there's a step that needs a mask and you'd rather not draw it by hand. This node is that step, and instead of a lasso you get to use a sentence.

Falcon Perception Generate is the working end of the ComfyUI-Falcon-Perception pack: a 0.6B open-vocabulary segmentation model from the Technology Innovation Institute (the tiiuae people behind the Falcon LLMs). Feed it an image and a plain-English query like red car or girl, and it returns pixel-accurate masks for every matching instance. No click prompts (SAM's job), no whole-subject cutout (BiRefNet's). If you can say it, it gets masked - a niche that usually means a grounding model plus a detector plus a segmenter wired together, compressed into one node.

Why you'd actually reach for it

Masks are the currency of half of ComfyUI - inpainting, regional conditioning, compositing. The usual ways to get one are clicking (SAM via the Impact Pack), a whole-subject cutout (BiRefNet), or a ControlNet preprocessor guessing structure. None of those answer "everything that matches this phrase." That's this node's niche: auto-mask a scene for inpaint-by-category, isolate one object to composite, or feed a region mask into a ControlNet. It's small enough to be light on VRAM, unlike the 2B+ VLM edit models.

How it works

Falcon Perception is an early-fusion vision-language model: image patches go straight into the transformer decoder alongside the text, not bolted on via a separate encoder like CLIP-style late fusion. Internally the query is wrapped into a prompt shaped roughly like <|image|>Segment these expressions in the image:<|start_of_query|>{query}<|REF_SEG|>. The decoder emits special tokens for object coordinates and sizes, and each segmentation token gets dotted against high-resolution image features from a lightweight upsampler to produce a full-res mask. The decode loop runs with a KV cache, dedupes repeats, and applies mask NMS so you get clean per-instance masks instead of a pile of overlapping ones.

Where this pack's author earned the install: the official upsampler had a quadratic-memory footprint that OOM'd; this port swaps in PyTorch's flex_attention / scaled dot product attention and falls back to eager mode when Triton isn't available - read: on Windows.

The inputs that matter

• query (default cat) - the whole product. Natural language; plurals and modifiers help, so red cars generally behaves better than car.
• min_dimension / max_dimension - the image is resized so the short side is at least min and the long side at most max before inference. Leave the defaults unless you're fighting tiny-object misses.
• max_new_tokens (default 2048) - the decoding budget. Low numbers can cut detections off.
• compile_model - off by default, and leave it off unless you're on Linux with Triton. On Windows it can stall or fail on first call.

The model input takes whatever Load Falcon Perception Model hands you; image is any ComfyUI IMAGE (batches work too, with one query shared across the batch).

The outputs

• combined_mask - a single MASK with every found instance OR'd together; wire it into MaskPreview, an inpaint node, or a ControlNet mask input.
• individual_masks - a batch of masks, one per instance, shaped [N, H, W]. Slice it for per-object work.
• info - a STRING like Image 0: Found 3 instances … Instance 0: xy=…, hw=…. Pipe it into ShowText; the normalized xy/hw coordinates help with compositing math and sanity-checking that it found anything.

If nothing matches, you get a zero mask and No instances found. in info - check your wording before you blame the model.

Installing it

cd ComfyUI/custom_nodes
git clone https://github.com/222222222l/ComfyUI-Falcon-Perception
cd ComfyUI-Falcon-Perception
pip install -r requirements.txt

Or search "Falcon Perception" in ComfyUI Manager and restart. Note that the README's own clone URL is literally a your-username placeholder - use the real repo above. You also need weights: grab model.safetensors from the tiiuae/Falcon-Perception repo and drop it at ComfyUI/models/falcon_perception/model.safetensors. Requirements: torch>=2.5.0, transformers, einops, pycocotools, pillow - the torch floor matters most (below).

Where people get burned

• PyTorch must be >= 2.5. The architecture imports torch.nn.attention.flex_attention; on anything older you'll hit an ImportError before the node even shows up. Upgrade the whole torch install - you can't quietly skip this one.
• Query phrasing is segmentation language, not chat language. Concise noun phrases beat sentences; if you get zero detections, simplify before you tune tokens.
• First call is slow - loading the safetensors and warming up attention. Subsequent runs are fine.
• Expect rough edges. This pack is brand new - one commit, essentially zero community usage as of writing, so don't treat the defaults as battle-tested.

The name is a small lie in the best way: no API, no key, nothing remote - it all runs locally, and that's the point.