Web Image Scraper v0.82

Most ComfyUI nodes make images. This one goes and gets them. The Web Image Scraper (class EricWebFileScraper_v082, part of EricRollei's Download Tools pack) points a real headless browser at a URL, walks the page - pagination, infinite scroll, the whole thing - and dumps the full-resolution images and videos into a folder on your disk. If your workflow is missing a dataset, reference set, or just a pile of references for style, this is the node that fills the gap.

ComfyUI's whole pitch is that everything is a node and every step is visible. That works great for sampling and miserable for "I need a hundred reference images." The normal answer is a browser, a download-all extension, and a lot of clicking. This node collapses all of that into a single node you drop at the end of the graph and run once.

How it actually works

Under the hood it's Playwright driving headless Chromium, with the Scrapling library for the anti-bot and page-parse work. The pack ships 25+ site-specific handlers - Instagram, Reddit, Bluesky, Pinterest, DeviantArt, ArtStation, Flickr, Unsplash, plus a purpose-built handler for IPS/Invision forums that even opens spoiler blocks and walks every page of a topic. Anything without a handler falls through to a generic extractor that grabs every image meeting your size thresholds. Every image gets hashed (perceptual hashing by default) so duplicates are caught even when the same photo appears at different URLs, keeping the largest version. It's genuinely clever for a scraper, and it's the kind of node where "the author clearly scraped a lot of sites" is a compliment.

The inputs that matter

You will set maybe five of these. The rest have sane defaults:

• url - where to start. Multiline, so you can drop several pages.
• output_dir - where files land; relative paths resolve against ComfyUI's output folder.
• min_width / min_height - skip anything smaller than this (512px default). This is how you avoid thumbnail spam.
• download_images / download_videos / download_audio - pick your media types.
• max_files - a hard cap (1000 default) so you don't accidentally download a whole forum.

Two worth knowing in the optional tab: use_stealth_mode for sites that fight back, and interaction_sequence, which takes a small JSON list of click / fill / wait_for_selector steps for automating galleries that need a button press. hash_algorithm lets you swap between average_hash, phash, dhash, and whash if dedupe is too aggressive or too loose.

Wiring the outputs

This is an output node - it ends the graph, it doesn't feed a sampler. Its output_path (STRING) and metadata_json_path are the two you'd wire into a text/display node if you want the result logged; the rest are count outputs (files_found, files_downloaded, videos_downloaded, duplicates_removed, skipped_small…) plus a stats_summary string. In practice you read the node's own report and go open the folder.

Installing it

Via ComfyUI Manager, search "Download Tools" and hit install. Manually:

cd ComfyUI/custom_nodes
git clone https://github.com/EricRollei/Download_Tools
cd Download_Tools && pip install -r requirements.txt

Here's the gotcha Manager won't save you from: Playwright needs its browser binaries, which pip does not install. Run this once against ComfyUI's Python:

python -m playwright install

Skip that and the node imports fine but can't launch a browser. FFmpeg is optional and only matters if you let it pull video.

Where it bites

• Browser won't launch - you skipped playwright install. It's the #1 failure mode.
• Sites blocking you - turn on use_stealth_mode, or set auth_config_path and save_cookies for login-walled content.
• Only thumbnails saved - raise min_width/min_height.
• Duplicates piling up - that's perceptual hashing off; try a different hash_algorithm.

One more thing to know before you build a commercial pipeline on it: the whole pack is dual-licensed, CC BY-NC 4.0 for non-commercial use, and a paid license for commercial use. Check the LICENSE before you ship anything. Scraping is also legally murky territory in general, and site terms vary - use it on content you're allowed to have.