vibecheck0.3.0
AI readinessaeoai-crawlers-blockedwarning

robots.txt blocks AI crawlers

If your robots.txt disallows AI crawlers — GPTBot, ClaudeBot, PerplexityBot, Google-Extended and friends — then assistants can’t read your content and won’t cite you in their answers. This is often accidental: a blanket Disallow, or a default that shipped with a template, quietly cuts you out of the fastest-growing discovery channel. (If the block is deliberate, that’s a valid choice — just make sure it is.)

Framework fixes
GeneralNext.jsVanilla JS

Symptoms

  • robots.txt has a Disallow: / for * or for AI bot user-agents
  • Your content never appears in AI assistant answers
  • Assistants say they can’t access the page
  • VibeCheck lists the specific bots your robots.txt blocks

How VibeCheck catches it

In your widget · Problems

warningaeorobots.txt blocks AI crawlers

To your coding agent · MCP

agent › get_detected_issues
→ { detector: "aeo", issue: "robots.txt blocks AI crawlers", threshold: "robots.txt has Disallow: / for * or a known AI bot (GPTBot, ClaudeBot, PerplexityBot, …)" }

The same string in your widget and in your agent’s context — no screenshot, no copy-paste.

Root causes

  • A blanket Disallow: / left over from staging
  • A template default that blocks AI user-agents
  • Blocking AI bots specifically without intending to

The fix

Decide deliberately. If you want to be read and cited by assistants, ensure robots.txt allows the AI crawlers (or at least doesn’t Disallow: / for them). Scope any Disallow to genuinely private paths rather than blocking the whole site or all AI agents. If you intend to block them, keep it — just confirm it’s intentional.

  1. Open robots.txt and find any Disallow: / affecting * or AI bots
  2. Allow the AI crawlers you want to appear in answers
  3. Scope remaining Disallow rules to private paths only
robots.txt — allow AI crawlers, block only private pathshtml
User-agent: *
Allow: /
Disallow: /admin/

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

Sitemap: https://acme.com/sitemap.xml

FAQ

Which user-agents are the AI crawlers?
The common ones include GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot (Perplexity), Google-Extended (Google’s AI training), and CCBot (Common Crawl). VibeCheck names the specific ones your robots.txt blocks.
Should I allow AI crawlers?
It’s a business decision. If you want visibility and citations in AI answers, allow them. If you want to keep content out of AI training/answers, block them deliberately. The point is to choose, not to block by accident.
Does blocking Google-Extended affect normal Google Search?
No. Google-Extended controls AI/Gemini training and AI features; it’s separate from Googlebot’s normal indexing. You can allow Googlebot and still block Google-Extended if you prefer.