Griptape Tool: WebScraper

Every LLM's knowledge is a frozen snapshot, and the WebScraper tool is the antidote: it lets your agent actually load a live web page and read its contents. Wrap it around a task like "go read this article and summarize it" and the agent fetches the page, extracts the text, and works from what's actually there - current, real, not hallucinated. In the Griptape pack this is half of the classic "research agent" combo: WebSearch finds the links, WebScraper reads them. On their own, a search tool gives you URLs you can't do anything with; the scraper is what turns a URL into usable content.

It's worth being clear about the division of labor, because beginners routinely expect one tool to do both. WebSearch finds pages; WebScraper reads pages. You'll usually want both, or you'll find yourself with a great list of links and an agent that can't open any of them.

How it works

The node wraps Griptape's WebScraperTool and returns it as a TOOL_LIST. When your agent decides a page needs reading, it passes a URL to the tool, which fetches and extracts the page's text content and returns it for the agent to work with. Nothing happens at graph-build time - the tool is a capability, and the agent chooses when to invoke it, which means you need to prompt for it ("use your web scraper to read the page").

The input that matters

• off_prompt - default False. The default here matters more than it looks, because scraped pages are big. Off-prompt False means the full page text flows into the agent's context - useful (the agent has the raw material) but token-hungry (a single long article can eat thousands of tokens per read). Flip it to True (Keep output private) and the tool runs off-prompt, handing back only what you ask for, at the cost of the agent working with a processed result instead of the full page. On a multi-page research run, keeping it on is the difference between a manageable bill and a surprising one.

One output, TOOL (TOOL_LIST), feeds an agent.

Install

Standard pack install:

cd ComfyUI/custom_nodes
git clone https://github.com/griptape-ai/ComfyUI-Griptape

or ComfyUI Manager → search "Griptape" → install → restart.

Where people get burned

The "it found the link but won't read it" failure is almost always a missing WebScraper tool or a prompt that never mentions reading pages. Give the agent both search and scrape, and say what you want: "search for X, then open the top result and summarize it." Second, scraping is subject to the real world: sites that block bots, pages that need JavaScript rendering, or paywalled content may return thin or empty text. That's not a bug in the node - it's the nature of the web. Finally, remember the token math. If you're running a research workflow over many pages with off-prompt left on, watch your context and your bill; this is exactly the kind of workflow where the Prompt Summary tool earns its keep afterward.