Free Tool
AI Robots.txt Checker
Check if your robots.txt allows or blocks AI crawlers. Covers all major LLM bots — OpenAI, Anthropic, Google, Perplexity, and more. Instant results, no signup.
21 bots checked ~3 sec Free · No signup
0
Allowed
0
Blocked
0
Via Wildcard
0
Not Specified
View raw robots.txt
Want the full picture?
See exactly what ChatGPT and Gemini say about your business
Fixing your robots.txt is just the first step. A full AI audit shows what AI models actually say about you — what they get right, what they get wrong, and what to fix to get recommended more.
Get Free AI Audit →FAQ
Questions about robots.txt and AI crawlers
- What is robots.txt and why does it matter for AI?
- robots.txt is a plain-text file at the root of your website that tells crawlers what they're allowed to access. AI companies like OpenAI, Anthropic, and Google follow these rules when deciding whether to read your site for training data or real-time answers. If you block them, they can't learn about your business — which means they can't recommend you.
- What's the difference between "Blocked" and "Blocked (Wildcard)"?
- Blocked means there's a specific rule for that bot (e.g.
User-agent: GPTBot / Disallow: /). Blocked (Wildcard) means the bot isn't mentioned explicitly, but a catch-all rule (User-agent: *) blocks everything. Both have the same practical effect — the crawler won't access your site. - What does "Not Specified" mean?
- No explicit rule exists for that bot — either in its own block or via a wildcard
Disallow: /. In practice this means the crawler is allowed to access your site, but you haven't explicitly told it so. Most AI bots will crawl in this case, though being explicit is better practice. - Should I allow all AI bots?
- For most businesses, yes — especially bots that power search and recommendations (GPTBot, Perplexitybot, anthropic-ai). Blocking them means AI tools can't read your site, which makes it harder for them to represent you accurately in responses. The exception is if you have a specific reason to restrict training data use, in which case you can block training-focused bots while still allowing search and browsing bots.
- Do AI companies actually respect robots.txt?
- Major AI companies — OpenAI, Anthropic, Google, Perplexity — publicly commit to respecting robots.txt. Some smaller or less reputable crawlers may not. For the bots listed in this tool, compliance is generally expected and enforced. That said, robots.txt is advisory, not technically enforced — it relies on the crawler choosing to comply.
- How do I update my robots.txt to allow AI bots?
- Add explicit allow rules for the bots you want to grant access. For example, to allow OpenAI's crawlers add
User-agent: GPTBotfollowed byAllow: /on the next line. If you have a wildcardDisallow: /, add the specific bot rules before the wildcard block — more specific rules take precedence.