Skip to content

PennyAPI Robots Permission Check · pennyregwatch.com

Check robots.txt before crawling or scraping a page. Decide whether…

Check robots.txt before crawling or scraping a page. Decide whether GPTBot, ClaudeBot, Googlebot, or another crawler may fetch the exact URL and return ALLOW/DENY, winning rule, crawl-delay, sitemap URLs, and live provenance.

Ready now6 calls · 6 buyers in 30 days

Input

Exact public page URL or hostname to test
Crawler product token, e.g. GPTBot, ClaudeBot, Googlebot

Sample response

Published by the provider. Your run returns a fresh response.

response8 keys
allowedtrue
crawl_delaynull
decision"ALLOW"
matched_rulenull
oktrue
sitemaps0 items
target"https://example.com/"
user_agent"GPTBot"
responseobject, 8 keys
Your agent can do this on its own
Add the MCP URL in Claude Code, Cursor or Codex. Your agent can run Tools within its spending limits, or ask for approval each time. Connect your agent

Connect your agent

Add one MCP URL and your agent can find and run this tool within its spending limits. Choose “Ask each time” if you want to approve every Call.

MCP URL

https://app.aecon.ai/mcp
Terminalbash
claude mcp add --transport http --scope user agentic-economy "https://app.aecon.ai/mcp"

In Claude Code, open /mcp, select agentic-economy, then choose Authenticate.

Track record

Paid calls, last 30 days
6
Buyers, last 30 days
6
Last paid call

Reported by the x402 directory.

A$0.0015 per call