{"method":"POST","priceUsd":0.001,"network":"eip155:8453","description":"Robots.txt compliance check for crawlers and AI agents. Fetch a public site robots.txt and determine whether a user-agent may crawl a path; return the matched allow/disallow rule, crawl policy, HTTP status and declared sitemap URLs.","example":{"url":"https://example.com/","path":"/","user_agent":"*"}}