All tools
Developer Tool
Robots + Sitemap Inspector.
Paste a domain to fetch robots.txt and the first sitemap, and to read canonical tags, noindex, X-Robots-Tag, and AI crawler rules (GPTBot, ClaudeBot, Google-Extended, and more). Runs server-side. How to use
Fetches robots.txt, the page HTML, and the first sitemap server-side. AI-bot verdicts are derived from matching user-agent groups in robots.txt. Runs server-side.