Text Scraper

Want today's headlines, a product page's title, or any page's main headings feeding into an LLM prompt live, without copy-pasting? Text Scraper fetches a public URL, pulls out the headline-like text (h1–h3, headline-classed elements, and matching links), and hands it to your workflow as a string. The default URL is Hacker News, so you can see the idea instantly: a prompt that knows what's trending, because the node just fetched it.

The genuinely thoughtful part is how safe it is. ComfyUI nodes that make network requests are a security review nightmare - and this one was clearly written with that in mind. It blocks local/private network addresses (localhost, 10.x, 192.168.x, etc.), only allows HTTP/HTTPS targets, and has a 10-second timeout so a dead site can't hang your queue forever. That's the SSRF hardening you'd hope every scraper node has, and most don't.

How it works

You give it a url and it does a requests.get with a browser-ish User-Agent, then runs heuristics over the HTML to find headline-like text - h1 through h3, class names that look headline-y, and matching anchor text. It returns the extracted text as the text output, or a clear error/validation message when something's wrong (blocked address, no headlines, timeout). The seed input is a small cleverness: it forces the fetch to run each execution even if the URL hasn't changed, so you can't accidentally cache your way into stale news.

There's no parsing library dependency beyond BeautifulSoup, which the pack installs - beautifulsoup4 is right there in requirements.

Install

Part of ComfyUI Text Processor:

cd ComfyUI/custom_nodes
git clone https://github.com/rookiestar28/ComfyUI_Text_Processor.git
pip install -r requirements.txt

Restart, or ComfyUI Manager → "ComfyUI Text Processor" → Install.

Gotchas

Three real limits, in order of how often they bite:

1. JavaScript-rendered sites return nothing. If a page builds its content client-side (most modern SPAs), the fetched HTML has no headlines and you get the "might use JavaScript rendering" message. Static pages - news sites, blogs, Hacker News - work; React apps don't.
2. Bot protection. Sites behind Cloudflare or similar may hand back a challenge page. There's nothing to do about it from here; pick a different source.
3. It's headline text, not full content. If you need an entire article body, this isn't the node - it's a headline extractor, and that's what it's good at.

Where this shines is live-context prompts: news-aware generation, "what's popular today" style runs, or any workflow that should reflect the world as it is right now. Just don't point it at anything you don't control unless you've thought about what a hostile page could put in its HTML.