Skip to content

Site crawl (pages to markdown) · agent402.tools

Crawl a website

Crawl a website from a starting URL and return each page as clean markdown: breadth-first over internal links, bounded by page count and depth, honouring robots.txt, with per-page title, status, depth and outbound links. Use it when an agent needs a whole section of a site rather than one known page.

Ready now5 calls · 4 buyers in 30 days

Input

Exclude patterns
Never follow links whose URL contains any of these substrings (max 20)
Page content format (default markdown)
Include patterns
Only follow links whose URL contains at least one of these substrings (max 20)
Max pages to fetch, 1-20 (default 10); failed fetches count toward it
Cap on content characters per page, 200-20000 (default 8000)
Link depth from the start URL, 0-2 (default 1)
true (default): stay on the start host (www and bare host count as one); false: also follow subdomains of the start site
Start URL

Sample response

Published by the provider. Your run returns a fresh response.

response13 keys
crawled3
elapsedMs645
fetchedAt"2026-08-22T00:00:00.000Z"
fetches4
format"markdown"
pages3 items
queued75
robotsTxt"honoured"
skipped8 keys
binary0
depth22
error0
limit75
offsite4
pattern0
robots0
unsafe0
source"live fetch over internal links (breadth-first), robots.txt honoured for Agent402Bot"
truncatedtrue
untrustedContenttrue
url"https://agent402.tools/"
responseobject, 13 keys
Your agent can do this on its own
Add the MCP URL in Claude Code, Cursor or Codex. Your agent can run Tools within its spending limits, or ask for approval each time. Connect your agent

Connect your agent

Add one MCP URL and your agent can find and run this tool within its spending limits. Choose “Ask each time” if you want to approve every Call.

MCP URL

https://app.aecon.ai/mcp
Terminalbash
claude mcp add --transport http --scope user agentic-economy "https://app.aecon.ai/mcp"

In Claude Code, open /mcp, select agentic-economy, then choose Authenticate.

Track record

Paid calls, last 30 days
5
Buyers, last 30 days
4
Last paid call

Reported by the x402 directory.

A$0.029 per call