Your robots.txt Might Be Blocking AI Crawlers

Here’s a thing that sounds like it should be obvious but rarely gets checked: AI platforms send their own web crawlers to index content. And a lot of websites are blocking them.

Not intentionally. But robots.txt files written years ago — or generated by default CMS settings — often contain blanket disallow rules that catch crawlers nobody was thinking about when the file was written. OpenAI’s GPTBot, Anthropic’s ClaudeBot, Perplexity’s PerplexityBot. These didn’t exist when most robots.txt files were last reviewed.

The consequence is direct. If an AI platform’s crawler can’t access your content, your content can’t be cited in responses. You can publish excellent, authoritative material and still be invisible in AI-generated answers simply because of a configuration file you haven’t touched in three years.

This is worth five minutes of your time right now. Open your robots.txt file — it’s at yourdomain.com/robots.txt — and look for any User-agent rules with Disallow directives. If you see blanket rules like User-agent: * Disallow: /, you’re blocking everything, including the AI crawlers you want to let in.

The fix is explicit. You can add specific allow rules for AI crawlers, or you can review whether your general crawl restrictions are tighter than they need to be. If you’re intentionally blocking certain directories from search engines, that’s fine — but check that you’re not blocking pages you want AI to find.

There’s a second layer to this that’s more subtle. Even if your robots.txt is clean, some sites block crawlers at the CDN or server level, in configuration that never shows up in robots.txt at all. If your pages load fine in a browser but you’re not showing up in AI responses despite good content, that’s worth investigating.

The broader point is that AI discoverability has technical prerequisites that most teams haven’t audited. Getting indexed isn’t just about publishing good content — it’s about ensuring the pipeline from content to AI crawl to AI training or retrieval actually works.

For retrieval-augmented systems like Perplexity, which pull live web content at query time, this is especially important. Indexing delay means citation delay. The faster your content gets indexed, the faster it can influence what Perplexity tells your buyers.

A complete AI visibility audit covers the content layer (what you’re publishing and how it’s structured) and the technical layer (whether crawlers can actually access it). Most teams do the first and skip the second.

Beket.ai checks both. We identify technical barriers that might be preventing AI platforms from indexing your content, alongside auditing what the models actually say about you. Run a free check at beket.ai and see what’s blocking your AI visibility.