⭐ Star Web Scraper (Headlines) 📰

Most prompt-building in ComfyUI is static - you type it once and it's frozen in the workflow. Star Web Scraper (Headlines) 📰 (StarNewsScraper) is the opposite: it fetches a web page, pulls the top headlines off it, and hands them to you as a text string you can drop into a prompt or feed to an LLM node. It's a niche, but it's a real one: news-aware image generation (generate a cover image "about today's top tech story"), or injecting current context into a local Ollama prompt so the model isn't stuck in its training cutoff.

How it works

Two inputs:

• url_choice - a dropdown of known sites, starting with NEW_URL plus a few presets (Hacker News, a horror-news site, Focus). The known list lives in sites.txt inside the pack's external folder, and it's editable - add your regular sources there and they show up in the dropdown.
• new_url - the one you type when url_choice is NEW_URL. Drop a new address in here and it's automatically appended to sites.txt so it persists for next time.

On run it fetches the page (with a browser User-Agent header, so it's less likely to be blocked as a bot), parses the HTML with BeautifulSoup, and collects headlines from common patterns - heading tags, title/headline classes, article links - deduped and capped at the top 10. Output is a single STRING: one headline per line, formatted with trailing periods, ready for a prompt field or an LLM node.

The inputs that matter

Honestly just two:

• url_choice - pick a known site or NEW_URL.
• new_url - the actual address when you're adding one.

That's the whole node. It's deliberately dumb, which is the point: no API keys, no RSS parsing config, no JS rendering.

Honest caveats - read this before relying on it

Three things will bite you:

• It's fragile by nature. Real scraping is: sites change their HTML, block scrapers, or require JS rendering to show headlines. The node returns "No headlines found. The website might be blocking scraping attempts." when it gets nothing - and that string is a normal output, not an error, so a silent workflow can hand you a "prompt" that's actually an error message. Check your output text before generating.
• It's a headline fetcher, not an article reader. You get titles, not content. The pack's README mentions newspaper3k for deeper extraction, but this node specifically does headlines - wire the output into an LLM if you want meaning extracted from the titles.
• It needs requests and beautifulsoup4. Both are in the pack's requirements.txt, so a Manager install gets them - but if you cloned manually and the node is missing from your menu, that's the import failing, and the fix is pip install requests beautifulsoup4 in your ComfyUI environment. The pack even prints a warning to that effect at startup.

When to reach for it (and when not to)

Reach for it when your workflow is about current events and you want the image or the LLM step to reflect "now" rather than a hardcoded example. Skip it if you need reliable, structured news data - an RSS feed or a real news API is the correct tool for that. This is a quick-context node: decent for a Hacker News headline collage or an "illustrate the top story" gag, not a production data pipeline.

Install

Part of the StarNodes pack:

cd ComfyUI/custom_nodes
git clone https://github.com/Starnodes2024/ComfyUI_StarNodes

or ComfyUI Manager → search Starnodes, restart, find it under ⭐StarNodes/Text And Data.