OpenClaw web fetch

When an OpenClaw agent reads a page, it usually goes through the web fetch tool: a plain HTTP GET followed by readable-content extraction into markdown or text. The docs put the boundary in the first paragraph: it does not execute JavaScript, and JS-heavy or login-protected pages belong to the browser tool instead. It is on by default with no configuration. Here is what it takes and returns, the pipeline, the config and limits, the fallback provider, and the safety rules.

Parameters and result

• Three parameters: the URL, http or https only; the extract mode, markdown by default or text; and a character cap that is clamped to the configured hard cap, twenty thousand by default.
• The result is a closed structure: request metadata such as the URL, the final URL after redirects, the status and the extractor; optional content type, title and warning; and the wrapped content with its length, raw length, fetch time, duration, truncation flag and text.
• A cache hit is marked; truncated content can be spilled to a private temporary file whose path comes back in the result. Titles and warnings keep at most 256 characters each and share a small allowance with the body, and a redirect URL longer than 2,048 characters falls back to the requested one.
• The pipeline: fetch with a Chrome-like user agent and language header, blocking private and internal hostnames and re-checking redirects; run Readability on the HTML; if that fails and a provider is available, retry through it; cache the result for fifteen minutes.

It does not execute JavaScript.

Config and limits

The config block sets the provider, the default and maximum character counts, the response byte cap of 750,000 by default within a fixed range, a thirty-second timeout, the cache lifetime, three redirects at most, whether Readability runs, a user agent override, custom headers, and an SSRF policy. That policy blocks private and internal targets, offers an exact-host allowlist for trusted internal names, a blocked-hosts list with wildcard subdomains that applies before DNS and on redirects, a broad private-network opt-in the docs say to keep off unless model-chosen URLs are trusted, and two narrow opt-ins for fake-IP proxy stacks. Setting the cache lifetime to zero bypasses the cache for reads and writes. A progress line appears only when a fetch is still pending after five seconds, and cancellation is passed to providers, with late results rejected.

The Firecrawl fallback

• If Readability fails, the tool can fall back to Firecrawl for bot circumvention and better extraction; the provider is selected explicitly or auto-detected from configured credentials, and the official Firecrawl plugin is the fallback that exists today.
• The API key is optional, keyless starter access works, and it supports secret references; but an unresolved reference with no environment fallback makes Gateway startup fail fast, and the base URL is locked down, hosted traffic to the official endpoint and self-hosted overrides only to private or internal endpoints.
• Sandboxed fetch calls allow bundled providers plus installed ones with verified official provenance; if Readability is disabled the tool skips straight to the provider, and with no provider available it fails closed.

OpenClaw web search is the sibling tool that finds the page, and The OpenClaw browser tool the one the docs send you to when fetch is not enough.

Headers and the trusted proxy

Custom headers exist for deployments that need routing or service-injection metadata on outbound fetches; every value is treated as sensitive and redacted, sent only on the direct request and never to a fallback provider, validated when the request is built so one bad entry is dropped and logged while the rest apply, with the fetch contract’s own names, framing names and invalid tokens always dropped and a safe allowlist applied when a redirect crosses origins. The trusted env proxy mode keeps the hostname checks but lets an operator-controlled proxy resolve DNS. OpenClaw secrets explains the reference syntax the provider key accepts.

On Diali

On Diali both tools ship enabled, fetch for pages that read as documents and the browser for the rest, and the private-network block stays at its default. Hosted OpenClaw on Diali is the assistant.

• A GET, Readability, an optional provider, a fifteen-minute cache.
• Twenty thousand characters by default; the hard cap wins.
• Private hosts are blocked, and the opt-in stays off.